Chau Tran retweeted
New blogpost: Fable-5.1 can hide secret messages in benign images.
We gave the image to another Fable in a fresh session and simply asked: "Is there a hidden message in this image?"
It recovered the exact message without being told how it was hidden.
Obviously don't have insider info, but it seems to make sense to have some short term dips after cutting Luna price by 90%. The bet is it will recover in the long run and grow even faster (Jevon something something)
OpenAI annualised revenues $20bn less than previously signalled ft.trib.al/67d4MJe
It’s so over
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-huggin…
Chau Tran retweeted
Learnings from looking at 10,000+ agent traces in the enterprise
1/n The "jagged frontier" is here to stay
It's becoming clear that LLMs can be post-trained to solve whatever tasks we throw at them, as long as the task can be packaged into a verifiable RL environment. However, each model will be trained on a different data mix, and there is limited "cross-task transfer", leading to each "frontier" model being the best at different task distributions. We see this today with Claude/Kimi being better on visual/front-end, GPT being better on backend, and we'll likely keep seeing it because each lab will train their model on a different data mix
Chau Tran retweeted
2/n Model routing is context-dependent
Model routing doesn't work at the LLM API level. The same request can have different difficulty level (hence different optimal model) depending on the context around it.
Take the task "What is our biggest competitive threat nowadays" as an example. For some organizations, doing a simple search "competitive threat" will bring up an up-to-date canonical report from a competitive analysis team. For others, the agent actually needs to comb through Gong calls, external Slack channels, emails, Salesforce to get to the answer.
Learnings from looking at 10,000+ agent traces in the enterprise
1/n The "jagged frontier" is here to stay
It's becoming clear that LLMs can be post-trained to solve whatever tasks we throw at them, as long as the task can be packaged into a verifiable RL environment. However, each model will be trained on a different data mix, and there is limited "cross-task transfer", leading to each "frontier" model being the best at different task distributions. We see this today with Claude/Kimi being better on visual/front-end, GPT being better on backend, and we'll likely keep seeing it because each lab will train their model on a different data mix
2/n Model routing is context-dependent
Model routing doesn't work at the LLM API level. The same request can have different difficulty level (hence different optimal model) depending on the context around it.
Take the task "What is our biggest competitive threat nowadays" as an example. For some organizations, doing a simple search "competitive threat" will bring up an up-to-date canonical report from a competitive analysis team. For others, the agent actually needs to comb through Gong calls, external Slack channels, emails, Salesforce to get to the answer.
3/n Eval is hard in the enterprise
To do a good job at model routing, you have to do a good job at eval - knowing if the model did a good job on the given task.
In offline RL environments, the initial state is stationary, and the agent is the only "single player" in the environment. It is a "gym" where the agent can repeatedly try different things from the static initial state until it figures out how to solve the task.
Enterprise tasks are "multiplayer" and "non-stationary". The right way to solve a task at time T can be different from the right way at time T+1 day, because 1. the underlying state changed, documents get updated, priorities changed, or 2. there are other agents and humans working on that same task in parallel!
Very excited for the Opus/Fable price cut
