🧠 + 🖥️ = 👁️ | Tweeting here when my ADHD meds have stopped working | Officially descended into madness
Ether
Joined April 2022
- Tweets7K
- Following369
- Followers163
- Likes3.5K
Pinned Tweet
I would like to kindly ask anyone or any journal that presents the results BEFORE the methods to just
Numbers are in, we're at 6.5x speed up. Didn't get to the 16x with the general approach since it's flexible, but the floor will be reached with custom, so technically 6.5x on explorations of new models and 16x on exploitation of specific models at scale. Amazing!
Funny story: I had a thing done by an agent manually and was redesigning the system structure and asked "could this be done programmatically?" and got "sure that's possible, easy" as an answer. The result? An engine for resolving a world's state that's 19k lines of code.
You don't. I didn't like fable 5.1 so I was still using fable 5, now I think opus 5.5 satisfies all I was looking for in fable 5. It really is a great model compared to the other options. Still same failure modes ALL models have, so no paradigm shift, but better flow 100%.
people be saying opus-5 has been quietly replaced by a soon to launch 5.2 and honestly I'm 100% sure that either this is true or they just idk retrained opus-5 or smth. MUCH better than what it was a week ago that's for sure.
Just started what my systems refer to as a "utility session" to investigate a quick data artifact's provenance and man it's crazy to see how the behaviors I saw with opus 4.5 that I built to avoid still show up with fable-5. 25 scripts with 293 reimplementations in 2h of work lol
I think all agents do that and I think it's because RL penalize going off-task, so they rely on "well technically" rationalization to turn every minute detail that you haven't specified ahead of time as "Your call". I do think I have found a (partial) fix for that now tho.
I spent the last several months (march) doing the same thing but with the exact opposite result: I've found that the failure modes that my system works to protect my stuff from to be persistent across models, with no sign of improvements. Better models just make it run smoothly.
OK. It's time to rethink this.
I've spend the last several weeks working on a harness that tightly constrains the agents to work the way that I want them to work. I set up all kinds of gates, and tests, and tools, and protocols, and ...
And while I was heads-down getting that to work, the agents got a LOT better. So much so that when I came up for air, the need for my harness was obviated. Indeed, the need for _any_ but the most liberal of harnesses may be obviated.
Just how good these things have gotten blows me away. I have had long debates with grok and codex about the structure of systems -- as if they were senior engineers. They often disagree with me and have their own perspectives. I have, more than once, found myself agreeing with their views.
I have not given up on constraints and tooling. Unit testing is still important. So is CRAP and Mutation testing. These tools still find bugs and offer useful constraints, though they can leave scars.
However, the agents have gotten so good that I can now give one a very significant task with a few guidelines and it will faithfully implement it. I can walk away for 40 minutes and when I return it will be done. CRAP will be satisfied, Coverage will be high, and Mutation testing complete. The architecture will be clean, and the code will be very good.
The end result may not behave perfectly, but it's so close that a couple of tweaks usually puts it into place.
What does this mean going forward? I'm not sure. But I'm beginning to think that harnesses should not treat agents as components within a software design.
Built a system for LLMs to write tests in a specific way. Built a query tool on these tests, but made it resolve empirically through the runtime (trade-off: it's slow AF). Best decision ever, now they have environmental signal when they break expectations/intents for tests lol
Fable 5 is MUCH better at understanding what you need from it and how it should communicate with you when orchestrating a task than Fable 5.1 is. Fable 5.1 feels more like opus 5 on that front. Relies on unverified assumptions, communicates with internal monologue and so on.
After interacting with Opus 5 and Fable 5.1 I think we need a new metric: assumption reliance. Because unverified assumptions that aren't reconsidered in face of evidence of wrongness are the new hallucination.
Say I have unobserved system states x with x_j=f(x_i) and observations y=g(x), if I derive a function h from features of y, T(y), such that T(y_j) = h(T(y_i)), if I show that using h(T(y_i)) I can predict system behavior z_j am I allowed to call h a mechanism of z?
Hypothesis confirmed: LLMs rely on conventions and broad knowledge of heuristic gotchas to compensate not understanding what the code does in depth. They just recursively apply superficial surface and fish for patterns.
One of my LLM systems has an explicit outlet for agents to report tool failures and Opus 5 is the only model that reports its assigned responsibilities as tool failures, like "I did X, so tool Y fails at doing X" when doing X is part of its assigned workload not a tool-side thing