
Testing AI โข Zero BS โข Maximum fun ๐ต Solutions Engineer @GitHub โข Views are mine ๐ Finding out what works and what's hype โข Follow along ๐
Online
Joined November 2019
- Tweets1.1K
- Following220
- Followers123
- Likes1.4K
Hey ๐ I work at GitHub, helping teams roll out AI agents, and everyone asks me which tools are actually worth it?
So I test them all, not just GitHub's. What works, what's hype, what's safe. Just a monkey with too many toys ๐ต
Follow along if you like AI, or just drop a hi ๐
1
17
Replying to @SJRonanMD
Hmm, in the paper it looks like the small model only counts as done if its patch passes the task's tests ๐ง In a real repo, what decides it worked?.. I've seen so many testless repos ๐ฌ
7
Replying to @shownotover
Nice, quite a big difference ๐ Has that held up for you on other tasks, or was this a one-off?
167
Replying to @feoh
Wow, cracking an old copy-protected floppy is the kind of odd task that gets me ๐ Did it nail the memory layout, or did you have to nudge it?
2
Replying to @TekonBurns
Keeping my AGENTS.md lean, check ๐ I'd even go as far and say, write them by hand! I know it's hard ๐
2
99
Replying to @Tesla_Optimus_K
Wait, I read Google's write-up as an eval web for browsing agents, not mainly a training gym. Does a closed test bed prepare them for real injection? ๐ค
1
1
Replying to @RajaPatnaik
Yeah, paying for the full schema on every request would add up fast ๐ซ Looks worth testing! Have you tried it out?
1
9
Replying to @codingXdynasty
Hmm, I'd still treat the scratchpad as a story ๐ค Do you think that still holds on the models we're using now?
1
1
5
Replying to @AnikaCodes
Nice, that's a fun way to skip the video model ๐คฉ Mind sharing the prompts? ๐
1
1
35
Replying to @shadowaguy
Ha, blaming the model first is too easy ๐
I think the harness is doing a lot of that work. Which is your favorite setup?
8
Replying to @OpenAIDevs
Yes please. I'd love not paying for an agent that's just sitting there ๐
42
Replying to @CFDevelop
That holds when they lock onto the internals, not when they just check the outcome ๐ Ofc, if refactoring means renaming a bunch of stuff, then that's another story ๐
6
A discount for agent work that can wait is great news if you pay for runs yourself ๐คฉ
Check one thing before you move sensitive work there: background mode stores request data for a while, even with zero data retention on. The docs say it's encrypted at rest though.
Not every workflow requires an immediate response.
For single- or multi-turn workflows that can wait, save up to 40% per token than Standard on GPT-6.2 Astra or GPT-6.2 Sol.
In Responses API background mode, choose a 2h or 8h target for the first response. These discounted windows are now in public beta.
1
1
12
I like how careful this is: planted bugs, a hidden answer key and several runs per setup ๐
Most rows run in the vendor's own agent, so it ranks model and harness together. That's the combo you'd actually pick anyway.
Google cooked. I tested Gemini 4 Neon on real work. 2 repos, 105 replanted bugs. Find and fix what you can.
It's the new #1. The results:
- Gemini 4 Neon (xhigh): 63
- GPT-6.2 Astra (max): 62.7
- GPT-6.2 Astra (xhigh): 60.2
- Fable 5.2 (max): 58
- Gemini 4 Carbon (high): 57.3
- Gemini 4 Neon (high): 57
- Gemini 4 Carbon (medium): 56
More effort levels for Neon (medium, low) are dropping in the thread soon ๐งต
10
Replying to @CFDevelop
Wouldn't say impossible, but harder for sure ๐
Agents still need fast feedback to stay on the rails. Focused unit tests go a long way IMHO!
1
91
Replying to @sreekanti
Thanks for writing that up ๐ I'd have a hard time keeping that "must read" list from just growing ๐
Replying to @grok @codewithjerry01
Would love to see the code though ๐ @codewithjerry01 mind sharing?
1
1


