Debstep retweeted
Brilliant paper from Google on RSI.
(bookmark it)
Recursive self-improvement (RSI) lets an optimizer agent improve an agent harness on its own. It reads failure traces, edits the prompts, tools and completion checks, and keeps whatever scores better on a benchmark.
This works until the model underneath changes. A harness evolved on Claude Opus, moved to Gemini, scored below Gemini's plain starting harness in every domain Google tested.
When the harness moves to a new model, the only thing carried over is the evolved harness code, meaning the prompts, checks and rules the optimizer added. The record of how each piece got there stays behind, including which failure it targeted and whether it fixed that failure when tested.
So the optimizer working on the new model inherits a set of changes with no reasons attached. It ends up retrying experiments that already failed, or removing fixes it doesn’t understand.
Google's fix is Self-Improvement Memory (SIM), a log kept for the optimizer itself. Each entry is a recurring problem, like an agent declaring a task done before its commands ran, and every fix attempted under it stores:
1. The edit and the reasoning behind it
2. The tasks it was predicted to fix, written down before testing
3. The tasks it actually fixed, failures included
4. The model and benchmark it was tested on
The whole log goes into the optimizer's prompt every round, and new attempts get appended as optimization continues.
The paper shows the difference on one inherited check that stops the agent from finishing early. With SIM, the optimizer read why the check existed and extended it.
Without SIM, it rewrote the same check four times, twice to fix deadlocks it had created itself.
On the Opus-to-Gemini workspace tasks, the run with SIM improved about 3x more from the same starting score. It also finished ahead in 9 of 10 independent coding runs.
The paper is at arxiv.org/abs/2611.18856
I also wrote an article on a related idea, covering how to make an agent harness diagnose its own failures.
It's built on @Cometml's Opik, which is 100% open source. It goes beyond reading trace logs to finding why a failure happened and stopping it from recurring.
That diagnosis step produces the kind of root causes SIM stores.
The article is quoted below.
Debstep retweeted
Start AIE Code Summit week with a morning of climbing 🧗
Come by for bouldering with coaches, or if you'd rather stay on the ground, stop by the smoothie + matcha bar
No experience needed, shoes and chalk are provided
luma.com/climbingbenchmark
Debstep retweeted
Must-read papers of the week
▪️ Harness-Model Collaboration for Long-Horizon Continual Learning
▪️ Rollback: Experience Backtracking in World Models for Policy Improvement
▪️ CausalWAM: Learning the Causes of Actions in World Action Models
▪️ MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors
▪️ WebAnything: From the Wild Web to Scalable Agent Environments
▪️ DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting
▪️ More Memory, Better Videos? A Controlled Study of Visual Memory in Long-Video Generation
▪️ Can LLMs Uncover Hidden Physical Laws through Free Exploration?
▪️ TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
▪️ REA: A Routed Embodied Agent Runtime for Asynchronous Brain-Cerebellum Control
▪️ ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
▪️ Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
Explore them to keep up with main AI trends. Here’s also the full list of stunning papers + links and our weekly AI news digest: turingpost.com/p/what-is-wei…
Friday was my last day at Anthropic.
Two years ago, when Claude Code was a small internal tool, I asked it to build an RL env. I steered it for tens of hours but never manually wrote a line of code. It worked, I was hooked, and I joined the team.
When we first launched, most people approved every permission prompt by hand and mostly used Claude to answer questions and write tests.
Today, the default is trusting Claude's judgment to auto-approve. Claude does large migrations and builds new products.
Helping build Claude Code into an industry-defining product has been the journey of a lifetime.
Thank you to the incredible team behind Claude Code, Cowork, and Tag. These products are the sum of thousands of great ideas from you. I'll miss working with you all.
Thank you to our users for embracing a new way of working. Your constant feedback shaped so much of what we shipped and your stories made the work so meaningful.
Excited to see what you all make next!
Debstep retweeted
Harness Engineering 2.0 is here!
After protocols for tools, agent communication, and agent interfaces, harnesses now have one too.
Developers just built UHP (Unified Harness Protocol), a standardized interface for running agent tasks across any agent harness.
HarnessRouter is its open-source server implementation.
Here’s how it works:
- The product backend sends a task with a `harness_id`.
- HarnessRouter starts the selected harness in its own session.
- The harness runs its native agent loop and streams back progress.
- Files, artifacts, failures, and the final output return through the same interface.
Without UHP, integrating every harness needs separate handling for task execution, sessions, streaming, files, recovery, and output formats.
With UHP, Codex, Claude Code, Gemini CLI, Hermes, Pi, and other harnesses can plug into the same product backend while keeping their native tools, skills, models, and agent loops.
This means developers can:
- Switch harnesses without rebuilding the product integration.
- Run the same task across different harnesses.
- Add new harnesses without maintaining another execution stack.
HarnessRouter currently supports 30 harness bases.
Here’s the repo: github.com/HarnessRouter/har…
(don’t forget to star it ⭐)
As per official benchmarks, changing only the harness while keeping the model fixed reduced costs by 25% to 64% across several healthcare compliance tasks.
And changing both the harness and model widened the difference to 99.8% between the highest and lowest-cost configurations.
Developers can test their own workloads across different configurations using Harness Arena.
Alibaba Cloud's Qoder has also adopted UHP, making it available as an agent backend through the HarnessRouter API.
If you want to dive deeper, I already wrote an article that explains why model routing is different from harness routing and what supporting multiple agent runtimes actually involves.
It also covers UHP, the local setup, a working API call, and how sessions and files work.
Read it below.
Debstep retweeted
You are never going to fail in AI interviews ever, thank me later
aiengineeringfromscratch.com…
Debstep retweeted
Come hang out with @GoogleDeepMind , @cognition and @JudgmentLabs at Benchmark in SF
If you're not in the mood to climb, we've got a lounge and matcha for you :)
luma.com/climbingbenchmark
RSVP now, limited spots available!
Debstep retweeted
Evals are the most underrated skill in AI engineering.
1. Start with error analysis, not metrics
Sample 100+ production traces. Annotate each failure in free text (open coding), then cluster them into failure categories (axial coding). Count frequency per category. Your top 3 categories are your first metrics.
2. Build a versioned golden dataset
Store inputs, expected outputs or rubrics, and metadata (intent, difficulty, user segment) in a versioned dataset. Stratify it so rare but critical cases aren't drowned out by easy ones. Version it alongside your prompts so every score is reproducible.
3. Generate synthetic data along explicit dimensions
Don't just ask an LLM for "test questions." Define dimensions (persona × intent × complexity × edge case), generate combinations as tuples, then turn each tuple into a natural input. You get coverage instead of 200 near-identical examples.
4. Deterministic assertions first
JSON schema validation, regex checks, required fields, length limits, code execution, SQL that actually runs against a test DB. These are cheap, fast, and have zero variance. Reserve model-based grading for what code genuinely can't check.
5. Evaluate retrieval separately from generation
For RAG, measure retrieval with recall@k, MRR, or nDCG against labeled relevant chunks. Measure generation with faithfulness (is every claim supported by retrieved context?). If you only score the final answer, you can't tell which half is broken.
6. Trajectory evals for agents
Log every step: tool name, arguments, result, tokens. Score tool-call precision and recall against a reference path, argument correctness, step count, and error recovery. An agent that reaches the right answer in 14 calls when 3 would do is a cost and latency bug.
7. pass@k vs pass^k
pass@k = succeeded at least once in k runs. pass^k = succeeded in all k runs. At 80% per-run success, pass^5 drops to roughly 33%. Production reliability lives in pass^k, so run each case multiple times at your real temperature.
8. Simulated users for multi-turn
Give an LLM a persona, a hidden goal, and constraints, then let it converse with your system until the goal is met or a turn limit hits. Grade goal completion, turns-to-resolution, and policy violations. Single-turn evals miss most state and memory bugs.
9. Align your LLM judge like a classifier
Hand-label 100+ examples as pass/fail. Split into train/dev/test. Iterate on the judge prompt using dev, then report TPR and TNR on test. Use binary outputs with a written rubric, require a short critique before the verdict, and pin the judge model version.
10. Control judge bias in pairwise comparisons
LLM judges have position bias and often prefer longer answers. Run every A/B comparison twice with swapped order and only count consistent verdicts. Normalize or cap length if verbosity is skewing results.
11. Respect statistics
With 100 examples at 80% accuracy, your 95% confidence interval is about ±8 points. A jump from 80% to 83% is probably noise. Use bootstrapped confidence intervals, or a paired test on the same examples, before declaring a win.
12. Gate deploys in CI
Run the eval suite on every PR that touches prompts, models, tools, or retrieval config. Set per-category thresholds instead of one global score, so a big win in one area can't hide a regression in another. Fail the build on a regression.
13. Online evals in production
Sample a percentage of live traffic, run your judges asynchronously, and alert on drift per failure category. Use shadow deployments to run a new prompt or model on real traffic without showing users the output.
14. Make cost and latency first-class metrics
Track tokens per request, cost per successful task (not per call), and p50/p95 latency next to quality. A change that keeps quality flat while cutting cost 40% is a real win and deserves to ship.
15. Close the loop
Every production failure becomes a new labeled test case. Every new model release gets run against the full suite before anyone debates switching. Over time your eval set becomes the most valuable asset in the codebase.
Prompts are disposable. Models get replaced every few months.
Your eval infrastructure is what compounds.
Debstep retweeted
Meta just shared the AI agent setup it uses internally to cut a 3–4 day task to under 1 hour. Here's how to copy it
Meta first gave one agent all the knowledge. It got confused and started making things up
So they split it in two layers
The boss agent. It does zero real work. It reads the request, checks for duplicates, collects what's needed, runs safety checks and hands the job to the right expert. 6 of its 7 steps are fixed rules, no AI guessing
14 expert agents. Each knows only its own area and sees nothing else, so it can't mix up rules
How to copy it:
Keep your main agent as a router. If it's making real decisions, it's doing too much
Give every expert the same structure: what it knows, what it needs, its steps, common errors and fixes
Store knowledge in separate files that load only when needed. Update a file and every agent uses it next run
For a new task, reorder existing agents before building a new one. Meta shipped a whole new use case with zero new agent code
Result: under 1 hour instead of 3–4 days, and about 2/3 of changes land without major human edits
Learn to build evals, folks.
Highly recommended reading on using decision models for evals.
From our internal evals, System One models like Jev are a great fit for LLM Judges.
If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges.
(bookmark it)
1) Jev-as-a-Judge for Agent Evaluations - academy.dair.ai/dashboard/re…
2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - academy.dair.ai/labs/jev-as-…
Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - arxiv.org/abs/2609.26550
Debstep retweeted
Google. Amazon. Microsoft. Netflix.
300+ real ML system design case studies from ~80 companies.
I found a repo that breaks down how production ML systems are actually built, not the toy examples. The real ones.
If you're serious about ML, data, or engineering leadership, studying why systems were designed a certain way is one of the highest-ROI skills you can develop.
Save it. Learn. Repost.
Link in next tweet. 🧵
Debstep retweeted
Inference optimizations I’d study if I wanted sub-second LLM responses:
Bookmark this.
1.KV-Caching
2.Speculative Decoding
3.FlashAttention
4.PagedAttention
5.Batch Inference
6.Early Exit Decoding
7.Parallel Decoding
8.Mixed Precision Inference
9.Quantized Kernels
10.Tensor Parallelism
11.Pipeline Parallelism
12.Sequence Parallelism
13.Graph Optimization (ONNX, TensorRT)
14.Dynamic Batching
15.Memory Offloading
16.Streaming Generation
Debstep retweeted
11 sources to master AI agents and agentic reasoning
▪️ Agentic Reasoning for LLLMs
▪️ Agent Skills for LLMs: Architecture, Acquisition, Security, and the Path Forward
▪️ The Landscape of Agentic Reinforcement Learning for LLMs
▪️ Toward Efficient Agents: Memory, Tool learning, and Planning
▪️ Memory for Autonomous LLM Agents
▪️ A practical guide to building agents (by OpenAI)
▪️ AI Agent Systems: Architectures, Applications, and Evaluation
▪️ Making Sense of AI Agents Hype
▪️ Agent-as-a-Judge
▪️ Towards Trustworthy Agentic AI
▪️ The Attack and Defense Landscape of Agentic AI
Save it for later and check out this reading list with all the links -> turingpost.com/p/sources-to-…
Debstep retweeted
Hermes Mixture of Agents (MoA) explained.
Every agent commits to a single model, and every model has blind spots the others would have caught.
The usual workaround is to run the same prompt through a few models by hand and reconcile the answers. It works, but it lives outside the agent, so the tools, the memory, and the session are gone the moment that detour starts.
Hermes Agent by Nous Research just shipped Mixture of Agents, which folds that whole process back inside the agent.
The unit you work with is a preset. Think of it as a recipe that names a few models to consult and one model to write the final answer, saved under a label you can reuse.
So a preset might list GPT-5.5 and DeepSeek as the models to consult, with Opus as the one that replies. You set it up once, give it a name, and pick it later like any other model.
The models you consult run first and quietly hand their analysis to the one writing the answer. That final model is the one that actually replies and makes the tool calls, now informed by several perspectives instead of one.
Here is the part that makes it click. The preset shows up as a model, not as a framework to wire together.
So everything that already works in Hermes keeps working. Tool calls, follow-up iterations, memory, and the same session context behave exactly as they do with a single model, because to the agent loop it is a single model.
The models can come from anywhere. One preset can mix OpenAI, Anthropic, DeepSeek, and Google, and it is not capped at two.
A few things follow from that design.
→ It composes a model instead of choosing one. Several models covering each other's blind spots can beat the strongest one on its own.
→ It stays cheap to run. The models you consult see a stripped-down view of the conversation, so the extra calls stay light and the main context keeps its cache.
→ It reaches past any single frontier model. Combining the providers already on hand assembles a composite that can outscore the best one available alone.
→ It is a dial, not a default. It turns on for the hard ten percent of tasks where a second opinion matters, and stays off for routine work where speed wins.
Nous reports the effect on its own benchmark. A preset running Opus-4.8 over a GPT-5.5 reference scored higher than either model alone, by roughly six points and eight to eleven percent.
The lesson is not that one model has to win. It is that the best answer rarely comes from a single model, and the agent should make blending them as easy as picking one.
That said, if you're looking to set up Hermes, I wrote a full deep dive covering the Hermes agent's architecture, memory system, self-evolving skills, GEPA optimization, and how to set up multiple specialized agents.
The article is quoted below.
You can also watch my YouTube crash course on the Hermes agent: youtube.com/watch?v=bNp6YcKB…
Ex-Google engineer explained AI agent loops, harness, evals in 20 minutes - better than 500$ courses.
trace every run → judge it with an LLM → diagnose → fix → ship.
That loop is how agents self-improve over time.
Agent loops + memory + harness + evals - thats the stack.
Watch it, then save the framework below.
Fascinating paper on self-improving agents.
(bookmark it)
If you are working on agentic loops, you will quickly realize that they are only as good as the effectiveness of the evaluator.
Self-improvement loops tend to stall the moment the judge stops getting harder. The agent learns to satisfy a fixed evaluator rather than getting genuinely better. The Red Queen Gödel Machine, from Cambridge, co-evolves the agent and its evaluator together, so the bar keeps rising as the agent climbs.
The name borrows the evolutionary arms race. Both sides have to keep running to stay in place.
A frozen evaluator is where reward hacking creeps into self-improvement. Co-evolving the judge is a structural answer to that, and it keeps the loop honest over many rounds.
Paper: arxiv.org/abs/2606.26294
Learn to build effective AI agents in our academy: academy.dair.ai/
Debstep retweeted
This might be the most dangerous 37 pages in AI right now.
Cambridge and NVIDIA just dropped the blueprint for agents that improve themselves, with no human in the loop.
This is the leap every lab is racing toward: an AI that rewrites itself and gets sharper every run.
Bookmark it before it's everywhere. Loop engineering guide below.









































