
MTS @ Datology | @ICMEStanford PhD | @dartmouth
San Francisco, CA
Joined April 2021
- Tweets76
- Following402
- Followers202
- Likes985
Kaleigh Mentzer retweeted
Yesterday at Backprop, @leavittron made the case that data quality is the compute multiplier: up to 150× less training compute, and a legal model experts preferred over GPT-5.6 Sol. Thanks @tokensandai!
See Matthew's latest research on backtest-safe data: datologyai.com/blog/beyondwe…
The boundary between midtraining and SFT is increasingly meaningless. High quality, diverse data is always your best bet
Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data.
Like everyone else since DeepSeek-R1 dropped, I’ve been treating "RL generalizes, SFT memorizes" as gospel.
Whenever SFT runs plateaued on out-of-domain reasoning, everyone just blame the algorithm.
Turns out, we might have been running an unfair comparison the entire time.
In standard setups, RL gets to explore a huge rollout budget per prompt while SFT gets trained on a tiny slice of data.
So the authors decided to actually level the playing field on Qwen2.5 and Qwen3 models by testing two things: mixing data sources instead of just distilling from one giant teacher, and scaling up the SFT data to match RL's rollout budget.
The results completely flip the usual narrative.
A carefully mixed dataset easily beats single-teacher distillation, and matching RL's rollout budget lets SFT generalize just as well as RL.
And when you combine both, SFT actually outperforms it.
To figure out why, they used LLM annotations to inspect the rollouts and proved (both empirically and theoretically on shallow transformers) that larger data budgets expose SFT to rare, "tail" solution methods.
Those uncommon reasoning paths are where the actual generalization signal lives.
Honestly, for everyone tired of debugging finicky online RL pipelines, this is a huge relief.
SFT wasn't broken, we were just starving it of diverse rollouts.
Full paper PDF: arxiv.org/pdf/2611.14912
Kaleigh Mentzer retweeted
Data quality is AI's most underinvested lever.
Tomorrow at Backprop Summit in SF, our co-founder Matthew Leavitt shows how it multiplies what each training FLOP buys, with up to 150× less compute.
Lightning talk ~3:30pm, hosted by @tokensandai
luma.com/backprop
The secrets of good data research are out!
The tech report behind our 6x pretraining compute multiplier is out today.
Training LLMs for data research takes more scientific control than training one great model does. Even small changes can confound results.
Some spoilers on how we did it:
datologyai.com/blog/text-cur…
Kaleigh Mentzer retweeted
More compute doesn't always mean it's better.
Catch @leavittron, co-founder of @datologyai, at Backprop Summit to hear why data quality could be the real multiplier behind better models.
luma.com/backprop
don't miss out!
Fellowship applications close in 10 days.
Datology Labs Fellows get funding, compute, and mentorship to spend a semester on the science of data. Paid, San Mateo/NYC, starting January.
Apply by Nov 15: labs.datologyai.com/fellowsh…
3
275
Kaleigh Mentzer retweeted
Fireworks serves >40T tokens a day of "specialized models." I'd define this as any post-trained model for a customer.
Kaleigh Mentzer retweeted
1/7 New Labs Research: Follow the Gradient
SafeTulu filters benign-looking data that weakens LLM safety, cutting attack success up to 22% relative while largely preserving capability.
🤗 Space: huggingface.co/spaces/Datolo…
Data: huggingface.co/datasets/Dato…
Paper: datologyai.com/blog/follow-t…
A backtest-able model with real modern synthetic data is huge alpha
To use LLMs for trading, you need to backtest them on time periods excluded from the training data.
But how do you do that with synthetic data?
Our latest blog shows how we can ensure near-zero knowledge cutoff contamination, even when using the most recent synthetic models.
11
378
Kaleigh Mentzer retweeted
When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees
Kaleigh Mentzer retweeted
Tomorrow I'll be on Datology's Curation Studio webinar discussing what went into measuring our 6x compute multiplier.
Come ask about scaling ladders, baselines, and quantifying data quality!
10am PT w/ @arimorcos @hurrycane @josh_wills: events.datologyai.com/curati…
guaranteed to be a good watch
Data quality is the ultimate compute multiplier.
Curated data matched our largest baseline with 6x less compute. A 12B-A1.4B model beat 30B-A3B on a fifth of the pretraining compute.
Live Tuesday 10am PT with @hurrycane @lukemerrick_ @josh_wills: events.datologyai.com/curati…
As a researcher, I cannot understate how much easier this determinism makes my life. No more mystery "maybe it's just distributed computing?? 🤷♀️" results from my dataloader
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Kaleigh Mentzer retweeted
Applications are open for the January cohort of the Datology Labs Fellows Program.
Funding, compute, and mentorship to work on the science of data.
Paid · One semester · San Mateo/NYC
Apply by Nov 15: labs.datologyai.com/fellowsh…
Fresh ideas, mentorship, compute, engineering support - what more could a researcher want!
Kaleigh Mentzer retweeted
defining the science of data. together.
10.14.26
defining the science of data. together.
Kaleigh Mentzer retweeted
"Not all data is equally valuable" - often yapped about, but Datology have done the work to quantify that and put it into in practice
Kaleigh Mentzer retweeted
Tbh I just found out that Thomson-1 beats GPT 5.6 Sol
That is insane!
That’s our curation is allowing a 400B model to beat the best model in the world at the time on it’s specialized domain: legal
Owning your own intelligence is more critical than ever, but harnessing your data to build a model that can beat the frontier has required deep data curation expertise.
We built @datologyai Curation Studio to change that. Curation Studio enables teams of all sophistication levels to customize curation to their use cases, delivering a massive compute multiplier that fundamentally changes the economics of model ownership.
We applied Curation Studio to the best publicly curated datasets from Nvidia, HuggingFace and others, and were able to deliver a 6x compute multiplier out of the box, all through better use of existing data.
This has allowed us to deliver huge wins for our customers. For example, we worked with @thomsonreuters to help them build Thomson-1 for only $450K in compute, which outperformed GPT 5.6 Sol in head-to-head comparisons and did so for 100x lower cost per task.
Curation Studio brings those capabilities to everyone.
This is a big moment for us at Datology, but it's just the beginning. We'll be expanding Curation Studio support to more modalities and verticals. And stay tuned for more research releases coming very soon!
Kaleigh Mentzer retweeted
We believe intelligence should be democratized, not just controlled by one or two labs on the planet, and we're getting there through data.




















