Pretraining data and evals // prev: founding team @inflectionai, PM @googledeepmind, VC at Invoke Capital and astrophysicist
Joined March 2009
- Tweets745
- Following1.7K
- Followers1.5K
- Likes1.5K
Pinned Tweet
“All that matters for anyone in life is their family, their health and that’s always the same for everyone” — Mike Lynch
bbc.co.uk/sounds/play/p0jkc9…
…or just talk with your kids like a human being
Business idea:
> Free holiday in Maldives for mathematicians
> In exchange for 10 math problem per day
> Sell the problems to AI labs
> Profit
Who's in? First 50 people only have to produce 8 problems/day.
Inspired by Benchmarks in Leipzig arxiv.org/abs/2606.05818
Positive reaction to the mai-thinking-1 tech report is more than I imagined. Some nice write-ups from the open research community
MAI-Thinking-1 by Microsoft looks to be approaching sonnet level model, the 109 page tech report is gold
they got 29T unique tokens without any synthetic tokens for pretraining which is exact opposite of what they were doing with phi models !!
so many counter intuitive decisions but the best part is they talk a lot about data.. this is a must must read
"MAI-Thinking-1: Building a Hill-Climbing Machine"
Microsoft just did something almost no frontier AI lab has done before
They shared how they engineered the data behind a frontier-scale model in unusual depth.
From data collection and eval decontamination, to data mix scaling, this paper lays out how they managed 30T pretraining tokens plus 3.55T midtraining tokens
Surprisingly, they also used no third-party distillation and no open-source training datasets
The model itself is not a jaw-dropping release, but the paper might be the best open look yet at a frontier-scale data factory and hill-climbing loop.
> buy truckloads of good books
> remove unspeakable amounts of slop from web data
> build a shedload of held-out evals
that was my work on mai-thinking-1
the model gets 97% AIME
and I can speak for hours about ISBNs
read the tech report: microsoft.ai/wp-content/uplo…
Qualitatively observed the same among AI researchers. The most successful are often exceptionally strong in seemingly orthogonal areas.
Stay general kids…
Build Jarvis with Claude Code, 100 lines of python and iMessage:
* Script watches Messages DB for texts from your number
* Forwards to Claude API with tools (shell, chrome, email)
* Claude executes + replies via iMessage
* Run as Launch Agent on always-on Mac