harness engineering @ https://xitter.gq/t.co/s3JONCGaTu
Bangkok
Joined November 2022
- Tweets14.4K
- Following422
- Followers7.9K
- Likes15.1K
Pinned Tweet
No, I am not
ex-Meta
ex-Netflix
ex-Google
or any other ex's.
I was born in Iran, Tehran.
I had no friends because we had to travel every 3 months due to my dad's work.
My unfair advantage?
Sheer will!
I once told my uncle, "I'm going to make $6,000/month" one day, and he laughed at me.
He said, "It's good to have dreams, but they're just dreams."
Now I'm making between $12-$15k/month every month.
I bought a house and an SUV for my mom, helped my brother get into Webflow boot camp, and invested in real estate in multiple countries. I've launched more than 10 businesses and generated over $10m+ dollars for my clients over the past 10 years.
If I can do it, you can do it.
If you believe, you, like me, have 2 hands.
You can do it!
I built a Grok Bot that handles all my T3 Code sessions. It tracks every thread, tells me when one finishes, and keeps my calendar in sync.
My Mac mini and my other machines all run T3 Code, and I drive them remotely from one chat.
One place to manage your coding sessions on every machine you own.
Copy my setup 👇
Oops, forgot to mention that I love Rex. I think it's probably the best thing we have right now for persistent sessions, even compared to tmux.
I'd love to see coding harnesses integrate something like Tern with Rex.
> Rex handles the server and persistent terminal sessions.
> Tern gives you the visual interface while letting you jump back into the same terminal session.
Having both without constantly switching between tools would be pretty much my perfect setup.
I've been trying Tern with OMP after spending quite a bit of time with T3 Code. I come from a terminal and TUI background, so a few things stood out.
> Tern starts from the terminal and lets me switch to a visual interface without losing my session. I really like this approach.
> T3 Code starts from the GUI. It's incredibly convenient when I work remotely on my Mac Mini, but I still need a separate terminal for anything outside the interface.
> OMP also balances usage across subscriptions automatically, which I wish T3 Code supported.
I'm going to run Tern + OMP on a server this week and see whether it can replace my T3 Code setup.
I've been trying Tern with OMP after spending quite a bit of time with T3 Code. I come from a terminal and TUI background, so a few things stood out.
> Tern starts from the terminal and lets me switch to a visual interface without losing my session. I really like this approach.
> T3 Code starts from the GUI. It's incredibly convenient when I work remotely on my Mac Mini, but I still need a separate terminal for anything outside the interface.
> OMP also balances usage across subscriptions automatically, which I wish T3 Code supported.
I'm going to run Tern + OMP on a server this week and see whether it can replace my T3 Code setup.
I genuinely think if you have ADHD, you're extremely impatient, and you're a bit autistic, you were born at the right time. There's never been a better time to have a brain that constantly jumps between ideas.
You finally have tools that can keep up with you instead of the other way around.
Theo is, I think, one of the few people you might not like that much, but he's very good at what he does. If you put aside the technical masturbation and the whole "oh, I can do this using the terminal" thing, T3 Code has literally changed the way I work and how I think about working across different machines.
It also gave me a new perspective on how I use Grok Bot and other tools. So you have to give this man a little bit of respect.
If you know what you're doing always go on medium if you don't know what you're doing go on high or Max
btw @karpathy uses medium on opus 5.5 on his latest project (Hackertube launched today).
overall i think having strong opinions on effort levels is like the tabs vs spaces of our time. just tweak to taste. altho mine is definitely superior to yours.
When you first install Rex, I think one of the first things you'll notice is the attention to detail and craftsmanship. @mitchellh is a true Shokunin, and I think it really shows in the little things. There's so much thought and care put into how everything looks, feels, and works together. I think you'll really appreciate it!
Good examples:
> Consistent icons/text everywhere
> Animation for Tailscale connection
> Really really clean minimal/practical design
I asked Grok to design a foldable case for my iPhone 17 Air and OPPO Find X9 Ultra.
It found the exact dimensions, sourced the 3D files, picked high-quality materials, and compared manufacturing vendors and prices.
Now I have everything ready to send for a test print.
We're living in crazy times.
I think we're measuring coding agents with the wrong metrics.
A model can pass a benchmark and still waste most of its budget on failed attempts.
I care more about cost per verified solution than raw pass rate.
ALT Fig. 4 from the Pairing Matters paper (arXiv 2610.32131): tasks resolved versus charged tokens per resolved task (log scale) for 18 harness-model pairings, with frontier reference points.
When I evaluate coding-agent systems, I want to track:
• Verified task completion
• Repeated failed attempts
• Unnecessary tool calls
• Unrelated code changes
• Recovery after runtime errors
• Total inference and tool cost
A cheaper model can become expensive after multiple retries.
And a stronger model can waste tokens inside an inefficient harness.
I'd compare complete model-harness systems under identical task budgets and verification rules.
The goal is reliable software, not just a higher leaderboard score.
Code graphs can improve small coding models by 17–21 percentage points.
But at 32B parameters, the gain drops to around 4–5 points.
More retrieval infrastructure isn't automatically better.
Sometimes grep is enough.
ALT Figure 3 from arXiv 2610.33958: function Acc@10 and gain over grep for code graph and embedding tools across Qwen3 sizes 0.6B to 32B. Gains peak around 1.7B to 4B and fall to about 4 to 5 points at 32B.
Researchers evaluated roughly 79,000 code-localization episodes.
They compared grep-only search against structured code-search tools.
The largest improvements appeared around 1.7B–4B parameters.
The gains were much smaller for 32B models.
These results measure file localization, not end-to-end patch success.
I'd benchmark grep, symbol search, and code graphs separately before adding retrieval infrastructure to a coding-agent harness.
Read the paper:
arxiv.org/abs/2610.33958
Most people are very bullish on the idea that everyone will build their own software. I personally wouldn't, especially for things I don't want to maintain. I'd rather pay for software and get on with my life.
I don't want to maintain a bunch of software for different reasons unless I'm paying a lot of money for it over a long period. Otherwise, it's not worth optimizing.
I think this will create more software, more competition, and eventually more personal software serving very small niches. Only a few people will need each product, so monetizing it becomes difficult.
But someone still needs to maintain it. Either an AI agent does it, which you have to pay for, or someone else needs to supervise it. The maintenance cost doesn't just disappear.
Calling it now.
AI is going to usher in a whole new era of software development. I’m calling it Software as a Service.
People will want to use these tools but they’ll realize they don’t want to spend the time to maintain and update them over time.
This will create a whole new category of businesses and entrepreneurs that can take on that task for businesses that would rather work on their core products.
AI is changing everything. This is a whole new paradigm of thinking about the way we interact with software.
If your coding agent compacts context, read this before trusting its memory.
A new security paper found that some harnesses can turn attacker-controlled text into apparent user instructions during compaction.
The problem starts when a harness converts structured conversation messages into plain text.
An attacker can place a fake user message inside material the agent reads.
When the harness summarizes that material, it may lose the original message's trust boundary.
The researchers tested nine harnesses.
In five harnesses that flattened message roles, the attack survived compaction in 87–100% of their tests.
In four harnesses that preserved structured roles, the same attack succeeded 0% of the time.
That doesn't mean every installation is vulnerable in every configuration.
But the failure mode matters.
A prompt cannot reliably restore provenance after the harness has already discarded it.
If you're building a coding-agent runtime, preserve message roles through compaction.
Don't let untrusted tool output become a user instruction.
Read the original paper 👇
arxiv.org/abs/2610.29869
Self-improving agent harnesses can overfit just like models.
A new paper on VideoHarness-SRSI studied more than 4,000 evolved harness candidates.
The researchers found three recurring problems.
Validation overfitting:
The harness improves on the evaluation data but doesn't generalize.
Search-budget dilution:
The system spends compute exploring too many weak candidates.
Early-experience trap:
Initial feedback pushes the search toward approaches that may not work elsewhere.
The researchers traced these problems to how the system reused imperfect validation evidence.
They changed the search process.
The new approach limits the feedback shown to the proposer, uses warm-up experiments, and tests mechanisms in isolated branches before combining them.
Across four video-understanding benchmarks, the revised system improved the average held-out score by 6.2% relative to the earlier approach.
It also reduced the validation-test gap by 37.9%.
I think this matters beyond video agents.
We're starting to let agents modify their own prompts, tools, memory, and execution rules.
But a harness that improves on ten familiar tasks might become worse on unfamiliar ones.
I'd want separate validation and held-out tasks, rollback support, and explicit regression budgets.
Self-improvement only helps if the changes generalize.
Read the original paper 👇
arxiv.org/abs/2610.19424