Pinned Tweet
I gave a talk at ICLR 2026 about how we are scaling RL on frontier LLMs with 1T+ parameters, on experimental data from our physical lab at Periodic!
Here's a rough recording of the talk:
Rishabh Agarwal retweeted
To channel my inner Schmidhuber, we found this softmax "bug" ~1.5 years ago and published in this paper arxiv.org/pdf/2603.11682
In reality, it's pretty hard to come up with a training scenario where this matters because gradient w.r.t. highly likely tokens is tiny anyway. Nonetheless I recommend fixing it -- one less thing to worry about! Check out @jon_barron 's work for an elegant fix!
Not sure if there's any recording for my CoLM talk but you can find the slides here. agarwl.github.io/slides/cont…
Posting couple of slides which look directionally good (to me)
Simple rewards, emergent complexity: give the agent something simple to optimize in an environment with a lot to learn about
David Silver is one of the all-time greats. He was the creator of AlphaGo, which changed the course of history by showing that neural nets could achieve superhuman performance for the first time.
He also has a unique intuition and knack for forecasting when things will happen in AI. With computer Go, he predicted the Lee Sodol outcome 10 years before, right to the year. He got the year right for Starcraft and won a bet inside Deepmind. Same for AlphaProof. Same for the IMO.
David's prediction for superintelligence: 3 years. !!! He has a strong sense that now is the time to build a superleaner, and that superlearners are now possible, thanks to the enormous progress in compute, simulators, autoresearch, and the sophistication of technology in the stack since the AlphaGo days.
What is a “super learner”? A super learner discovers all knowledge from its own experience. Start from nothing (no internet pre-training), and go beyond human knowledge. Drop it into any stream of experience, simulated or physical, and the superlearner learns.
He expects the progress will be rapid. When you get the recipe for a superlearner just right and remove humans from the development loop altogether, a superlearner can learn extremely quickly. Just look at AlphaGo. AlphaGo took two years to develop with humans in the loop. When David and his team at @GoogleDeepMind got the formula for AlphaZero correctly, it surpassed the human-developed algorithms in just a few hours. Human data is a shortcut, but you have to let go of the human data to shoot beyond it.
In January, David left GDM to found @IneffableLabs, a research-led lab with one mission: build a "super learner" that discovers all knowledge from its own experience, and make first contact with superintelligence.
He joins us with co-founder Lasse Espeholt, who built IMPALA and SEED RL, the foundations of scaled-up RL, and now runs Ineffable's compute.
A few things we cover in this conversation.
— What a super learner is: depth (start from nothing, go beyond human knowledge) and breadth (drop it into any stream of experience, simulated or physical). The human brain is the only existence proof
— Why they're giving up human data: AlphaGo took two years with humans in the loop. AlphaZero outperformed it in a few hours. "You have to let go of the human data to shoot past the competition"
— Crucibles: simulated worlds with enough emergent complexity (physics engines, millions of interacting agents) for an agent to rediscover technology, science, maths and language on its own
— Simple rewards, emergent complexity: evolution optimized fitness and got all of life. Don't breadcrumb the objective, give the agent something simple to optimize in a world with a lot to learn about
— Go as the existence proof: one agent is the most boring game imaginable, two agents is "an infinite onion of layers of complexity"
— Why human language is probably not the endpoint for reasoning, and how training on it bakes in a flat earth and the four humors
— Can superintelligence be built in simulation? If the sim is Turing complete, computation, economics, language and maths are all expressible, and it happens 10-100x faster and safer
— Alignment: language-based monitoring is reaching its limits as agents learn to say one thing and do another. Shadow actions, superintelligences monitoring superintelligences, and building a real science of how they interact peacefully
— Accelerate in simulation, pace in the real world: Ineffable will gate real-world deployment on evidence that it goes well
— Timing as the determinant: Silver called Computer Go to the year, then StarCraft, then the IMO. He thinks this is the moment
— Predictions: True superintelligence within three years.
Timestamps:
0:00 – Introduction
1:09 – What held true from Go to today, and what the LLM era changed
3:35 – Why leave DeepMind to start a new lab
5:58 – Why "Ineffable" (and the secret second reason)
7:03 – Lasse's path: scaling RL at DeepMind and Google Brain
9:31 – Defining a super learner and first contact
12:35 – Super learners vs. LLMs that make discoveries
13:59 – Day to day: the RL stack, crucibles and human-journey milestones
16:13 – Don't breadcrumb the objective: simple rewards, emergent complexity
21:16 – What a crucible looks like
24:14 – Go: one agent is boring, two is infinite
25:41 – Will agents rediscover language?
27:22 – Removing human priors: AlphaGo to AlphaZero
32:05 – Keeping human bias out of the crucibles
33:35 – Confidence, timing and super forecasting
36:24 – Is compute the driver?
38:13 – The ceiling on simulation and Rich Sutton's objection
43:44 – Alignment and the science of superintelligence
47:45 – Monitoring agents that don't think in English
51:01 – Pacing the frontier
52:54 – The compute setup: Vera Rubin on GCP
55:18 – Training, inference, simulation and RL scaling laws
57:40 – IMPALA and SEED RL
1:00:58 – 20 people and the compute-per-person ratio
1:04:49 – A research-led culture
1:05:47 – Predictions
I hope other American companies follow suit
Watched Digger, phenomenal. Can't believe it bombed while cliched Spiderman-part-1000 the biggest movie this year. Welcome to Level 4!
@TomCruise gave the performance of his life.
Giving a talk at CoLM today in 30 mins, sharing some of our work at Periodic and and where it's heading.
In addition to RL as a service, now we have LLM training data curation as a service from @arimorcos
Nice blog showing the large compute efficiency wins during post-training possible by just better data curation during pretraining / midtraining.
Rishabh Agarwal retweeted
We are introducing Beam today, a 500b open model that excels in coding, agentic, and scientific workloads.
Beam is very token-efficient, pre-trained from scratch, and scaled with the largest RL run documented openly we are aware of.
Getting here was challenging, rewarding, and personal.
I immigrated to the USA during childhood when my parents got jobs as scientists at PNNL, a national lab located in a rural town in Washington state.
Growing up, I developed an interest in physics because it was grounded in simple principles, was foundational, and had long-term impact.
Physics was the foundational science of the 20th century. The first general computer (ENIAC), the transistor, GPS, and much of modern technology had its roots in physics.
In 2016, during my graduate studies, I noticed the birth of a new foundational science - Artificial Intelligence.
AlphaGo, the superintelligent Go agent developed by DeepMind, had come out and made me feel what was about to come.
A neural network, trained to imitate human players, and then improve itself through trial and error, mastered the most intellectually challenging board game in the world.
What surprised me about AlphaGo is how it was conceptually simple, general, and scalable. The system could in principle improve itself forever, it was just a function of compute.
This is when I became “AGI pilled” as they say.
I switched from physics to AI because I thought this would become the root node science of the 21st century, much like physics was the root node science of the 20th century.
I believe that working on foundational science is not enough on its own. Foundational science must also remain open and widely accessible.
We remember Newton for his contributions to physics, but he was also an alchemist.
Alchemy was done in secret, because the artificial production of gold was deemed lucrative but also dangerous with the potential do destroy entire economies. Newton even wrote his alchemy notebooks in cypher.
On the other hand, Principia was published in the Philosophical Transactions of the Royal Society, the knowledge was made widely available, passed down for generations in what we now call Newton’s laws.
The computer, transistor, GPS were possible to build because physics as a science was open. Openness creates an ecosystem of innovation.
The notion of openness, which is a core tenet of scientific discovery, has also become a guiding principle for how foundational technology gets built and distributed.
The internet was built on open protocols. The most widely used operating systems are open. The field of cybersecurity exists because strong encryption protocols were made open in the 1990s.
Open also means safer. The best defence is a distributed one. Commonly used open source software is robust and battle-tested because it is inspected and patched by a community of developers.
AI is different but also similar to software in many ways, though with a much larger surface area of vulnerabilities. It is impossible for a small group of safety researchers working in secrecy to remediate all of the potential issues regardless of their intentions. It is like entrusting a handful of white blood cells to protect your body.
AI is both foundational as a science and a technology, and the safest way that also ensures the benefits of AI are distributed evenly is by making it open.
I’m excited for what developers and scientists build and learn from this model, and for Beam to play a small part in ensuring that AI is made widely accessible, safe, and beneficial to all.
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: reflection.ai/beam
Rishabh Agarwal retweeted
I made lasteval.com. The premise is simple: AI is already improving lives, and we should have a page that collects these wins.
We only talk about how "intelligent" the models are, what jobs they might automate, the trillions being invested, who will "win" the AI race, and the safety risks. All these things naturally worry people, and make them averse to AI. I think we're missing the point.
The goal of technology should be to improve people's lives. AGI, ASI, RSI are all just in-group goals, that aren't obviously beneficial on their own. Speculative promises shouldn't be all we have to offer. Imagine releasing a new model and saying "it will score 99% on these benchmarks one day" instead of reporting what it already does today.
I've spent the last few weekends reading about AI systems, both generative and predictive, that are already making a large positive impact on the world. It gives me hope, and I made this page so it does the same for others. Some examples in 🧵
Ultimately, the final, and only evaluation for AI that matters is whether it improved lives. The models are already so good, and yet the world is not that different. I think it's important to change that, and this is my attempt to paint the target.
There's no way the list is complete, so if you think an AI system has had a large positive impact, do share it through the submit page!
If you are interested in working on science of scaling RL, please apply!
You'll be closely working with me
I asked people which neolabs under $10B have the highest talent density:
32 votes: Core Automation (@MillionInt, @_arohan_)
26 votes: Periodic Labs (@LiamFedus, @ekindogus)
20 votes: Flapping Airplanes (@spectorb, @amspector100, @aidanmantine)
18 votes: Standard Intelligence (@G413N, @devanshpandey)
8 votes: Ricursive (@annadgoldie, @Azaliamirh)
5 votes each:
- Ineffable (David Silver)
- Mirendil (@bneyshabur, @HarshMeh1a, @shayan_, @tararezaeikh)
- Recursive (@RichardSocher, @_rockt, @timshi_ai, @josh_tobin_, @CaimingXiong, @jeffclune, @tydsh, Alexey Dosovitskiy)
3 votes each:
- Applied Compute (@ypatil125, @rhythmrg, @lindensli)
- Isara (@ezhang7423, @hegasz)
- Prime Intellect (@vincentweisser, @johannes_hage)
2 votes each:
- Elorian (@AndrewDai, @yinfeiy, @SethInternet)
- Goodfire (@eric_ho, @DanJBalsam, @banburismus_)
- Inherent (@tantumscollins, @edwardfhughes, @LouisKirschAI, @kallyaleksiev)
- Trajectory (@rronak_, @QuantumArjun, @MichaelElabd)
- World Labs (@drfeifei, @jcjohnss, @BenMildenhall, @chlassner)
People couldn't pick a company they work at or founded. Disclosure: I'm a small investor in Applied Compute, Factory, Standard Intelligence, Trajectory and Wafer. I didn't vote.
Continued:
LLM crazy innovations be like:
- Please think step by step
- Please manage your context*
(*cries in KV cache)
Rishabh Agarwal retweeted
I want to be very clear about this.
just because we found one serious exploit on openai does not mean we know how to secure an organization that complex. openai has security people with far more experience and better qualified doing that than us, and i don’t want to pretend we have solutions.
note, i was saying "they are building manhattan project", it is not a belief i strongly hold, i don’t like making confident predictions about the future. it is the labs that make those comparisons.
what I know for a fact is that these models already have serious offensive cyber capabilities, what that does to the world, I don’t know, but my loosely held view is that it might lead to a period of turbulence because i think its offense dominant than defense
having said that, what i don’t understand is this, if the people building these systems truly believe they are powerful enough to create nuclear level risks, and they are talking about slowing down because of those risks, why is that work is done through ordinary saas products?
why is it happening through slack, managed github, employee browsers, and apps accessible from the public internet? every third party and library they use becomes part of the attack surface.
remember the jfrog registry in the hugging face agent swarm? it didn’t matter that it was a third party, it was in the path. an attacker, or an unaligned model, doesn’t care who owns it or its out of scope. they will hack whatever is in the path to achieve their goal.
one more concerning thing is, why are codex cloud, claude(assuming anthropic does the same) publicly accessible web applications connected directly to private github repositories?
again shouting my lungs out, coding agents on the cloud is worst thing happened in terms of security, it is going backwards.
your browser becomes a attack surface, a lame xss, stolen creds, phishing, browser exploits, all these bugs can achieve similar impactful bugs as we shown. like, you don't need to pivot anymore and worry much about exfiling, just compromise cloud agent its all over either through client-side browser or the server.
Hacktron founder @S1r1u5_ argues OpenAI is running a “Manhattan Project” on Slack and GitHub, giving it the same basic attack surface as any startup:
"Every repository is running on github .com/openai. It's the same thing that Hacktron uses or any other company uses."
"We have friends at OpenAI who are much more capable than what we could do. But what I can say is, why is it running on Slack? Why are you running this Manhattan Project from Slack? Why are you hosting your repositories on github? That looks silly to me."
"Your attack surface becomes Slack. You can hack a Slack employee and go through that and get into OpenAI. You can hack GitHub, go through that and get into OpenAI."
"My claim is it shouldn't be running on B2B SaaS if you are building Manhattan Projects. That's all I'm saying. I can think of numerous ways on how to get into that."
@HacktronAI @rootxharsh
Rishabh Agarwal retweeted
Why is RL so sensitive to training-inference mismatch (TIM)?
With each training step, the trainer is pulled toward the sampler, resulting in accumulating drift.
We exploit this intuition to devise a novel correction method to stabilize RL under TIM, called Score Centering.
They had a cool technical post too to train LLMs to predict the future.
thinkingmachines.ai/news/tra…
Rishabh Agarwal retweeted
After 10 incredible years at Google, I've left @GoogleDeepMind to join @thematthewosman and @FabioZB_I at Polyphron as a co-founder and explore the next frontier in my research passions.
Together, we're training frontier models that autonomously explore, understand, and physically reproduce human tissue biology. It's our grand vision to grow tissues realistic enough to match human response to all drugs, and realistic enough to recover organ function when transplanted into living organisms.
Other than the bitter lesson, I think another lesson remains true in the history of ML: the more directly you can optimize your end goal, the better your outcome will be. IMO this is one lesson largely missed in the drug discovery and computational biology space today, and one that Matt and Fabio understood very clearly (and why I joined them on this journey!).
At the heart of our company is one giant RL problem: propose a recipe to grow tissue, sample life itself, reward directly against the functionality of the tissue. If this sounds exciting to you as either an AI or Life scientist, feel free to reach out! We're building the coolest lab in NYC :)
I'll never forget all of the {pretraining, RL, self-improvement} projects I worked on with my brilliant friends at GDM over the years, but I'm even more excited to share the things we discover next!
(p.s. this isn’t the first time i RL’d something based off of similarity, iykyk)
I often see people mic drop with "Bitter Lesson" without maybe understanding its implications.
The best interpretation I like is from @hwchung27 "We need scalable methods that can better leverage computation."
For example, LLMs are amazing technology but their current training itself adds in a lot of inductive bias, e.g., data distribution being good enough to simply clone.
RL faces it's own version of bitter lesson: where does that reward come from? Well today, it's mostly human ingenuity.
Genius is 99% perspiration, and 1% inspiration. This is also true for how we trained Periodic Neon to beat frontier models, like Astra and Fable 5.1, on a scientific capability our labs care about.
We trained Neon with reinforcement learning on our lab data using only 1.3K GPUs, orders of magnitude less than Astra. As we scale our labs and compute, we expect our models to tackle much more complex scientific problems.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
