
I simplify complicated things, and complicate simple things. (Dev) Advocate as a Judge (at @AI21Labs)
Joined April 2022
- Tweets397
- Following260
- Followers358
- Likes6.3K
How do developers actually use coding agents? 7 weeks & 4k+ turns of real developer traffic, classified by category.
Held up against public coding benchmarks, my colleague Eli Lepkifker (from @AI21Labs) found a real difference across:
- Task type: Confirmations, redirections, git commands, and pasted context make up about 40% of real developer turns but appear in almost no public coding benchmarks
- Task difficulty: Not every question is a complex, multi-file architectural issue. Sometimes it's just, 'Why don't I see my changes in git status?' Public benchmarks over-index on the former.
- Task context: Public benchmarks tasks are clean, 1k characters messages. Real developer messages are...not. We found they avg 80 characters, are sent mid-session and 60% contain more than one distinct task.
More on Eli's work building a real-world coding benchmark here: ai21.com/blog/evaluating-cod…
1
42
There's one question you should ask before you build an agent. Most people skip it (and pay for it in compute).
Over a year of agent research at @AI21Labs across deep research, RAG indexing, agentic search and coding, and it comes down to this: can a candidate answer be checked against anything?
Property of the task, not of your model. The answer is yes? Verify, then commit to one. No? Run varied attempts and combine them.
>>
Coding: Yes. And also No.
In this task, the end is checkable: the tests pass or they don't. The middle isn't: you can read most of a repo and still miss a file the bug touches.
That's why coding looks like a recall problem. It is, in the middle. It isn't, at the end.
So we split the work along that line. Small open models explore the repo in parallel, and a mid-size model sums up what they found. Only then does a frontier model write the patch, once, where the check is.
SOTA on SWE-Bench Pro, at a third of the cost of a frontier model doing it all alone.
>>
1
2
13
It won't fix everything, and the answer isn't always clean (see: coding). But it's the first question I ask now, before reaching for a bigger model.
For the full details: ai21.com/blog/the-verifiabil…
12
My talk from @aiDotEngineer SF is up!
While everyone moved on to agents, somebody still has to take care of the chunking.
One window has to serve “what is X” and “why did X happen.” It can’t. Small cuts the reasoning, big buries the fact. So stop picking one.
youtu.be/r9OwPx_HoV0
1/ Your agent already found the right answer, but your aggregation threw it away. Across every generator we tested (weak and frontier, open and closed), pass@k surpasses pass@1. Correct answers are routinely in the pool and just not selected. The binding constraint isn't answer generation, it's selection. (Work by @AI21Labs)
5/ This is not benchmark specific: it transfers to BrowseComp-Plus, a benchmark and retrieval tool it never saw in training. Precision holds, recall takes a hit - but a ~15 step SFT on 85 examples closes it back up.
Check out the full blog with experiments, training details, and cost breakdowns: ai21.com/blog/you-need-a-ver…
Worth reading alongside @jackyk02's work on verification scaling for coding agents. Same shape of finding in a different domain: the candidates are good enough, selection is the bottleneck, and verification beats buying a bigger generator.
One difference worth noting: they rank candidates with the same model that produced them. We give the verifier no generator context at all and make it re-research the claim from scratch. Open question how much of the lift comes from different weights versus fresh evidence, and I suspect it's mostly the latter. xitter.gq/jackyk02/status/208942…
Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰
As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost.
For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench.
Try it out today: github.com/llm-as-a-verifier…
More on verification scaling in my previous post.
5
126
New episode drop: Thrilled to welcome Not Diamond co-founder & CEO @tomas_hk to YAAP, hosted by @YuvalinTheDeep, to unpack why the old model routing playbook breaks.
🎙️Listen here: ai21.com/yaap/the-model-rout…
RAG isn't dead, and there's still a lot to say about it :)
The best RAG systems don't retrieve what they embed.
When we talk about RAG, it's usually thought: index the doc → retrieve the same doc.
But indexing ≠ retrieval
So the data you index doesn't have to be the data you feed the LLM during generation.
Here are 4 smart ways to index data:
1) Chunk indexing
- Split the doc into chunks, embed, and store them in a vector DB.
- At query time, the closest chunks are retrieved directly.
This is simple and effective, but large or noisy chunks can reduce precision.
2) Sub-chunk indexing
- Take the original chunks and break them down further into sub-chunks.
- Index using these finer-grained pieces.
- Retrieval still gives you the larger chunk for context.
This helps when documents contain multiple concepts in one section, increasing the chances of matching queries accurately.
3) Query indexing
- Instead of indexing the raw text, generate hypothetical questions that an LLM thinks the chunk can answer.
- Embed those questions and store them.
- During retrieval, real user queries naturally align better with these generated questions.
- A similar idea is also used in HyDE, but there, we match a hypothetical answer to the actual chunks.
This is great for QA-style systems, since it narrows the semantic gap between user queries and stored data.
4) Summary indexing
- Use an LLM to summarize each chunk into a concise semantic representation.
- Index the summary instead of the raw text.
- Retrieval still returns the full chunk for context.
This is particularly effective for dense or structured data (like CSVs/tables) where embeddings of raw text aren’t meaningful.
Once you've decided what to index, the next question is how efficiently you can store it.
Every strategy above multiplies your vector count, sub-chunks, questions, and summaries all add embeddings on top of the originals.
Binary quantization is an effective solution to this.
It shrinks each vector by up to 32x while keeping retrieval quality intact, so smarter indexing doesn't blow up your memory budget.
We wrote a full article about it recently.
Read it below.
👉 Over to you: What are some strategies that you commonly use for RAG indexing?
108
Join @YuvalinTheDeep, Senior Developer Advocate at @AI21Labs, for a live webinar in partnership with @DataCamp: The Four Gaps Between Demo Agents and Production Systems.
If you're shipping agents to prod, this is where prototypes break and real systems begin.
🗓 May 6 · 11am ET
Register:datacamp.com/webinars/the-fo…
Day 1 of #AIDevSF is live. 🙌
📍 Find us at booth 121 🎤 Catch our Chief Product Officer Or Dagan's session on Day 2: Efficiently Navigating the Agentic Action Space: Meta-Model Orchestration using AI21 Maestro.
Thanks @DeepLearningAI for having us.










