NULLchimp ๐Ÿต (@nullchimp) header
NULLchimp ๐Ÿต (@nullchimp) profile picture

Testing AI โ€ข Zero BS โ€ข Maximum fun ๐Ÿต Solutions Engineer @GitHub โ€ข Views are mine ๐Ÿ™ Finding out what works and what's hype โ€ข Follow along ๐Ÿƒ

Online
Joined November 2019
Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)Photo by NULLchimp ๐Ÿต (@nullchimp)
Hey ๐Ÿ‘‹ I work at GitHub, helping teams roll out AI agents, and everyone asks me which tools are actually worth it? So I test them all, not just GitHub's. What works, what's hype, what's safe. Just a monkey with too many toys ๐Ÿต Follow along if you like AI, or just drop a hi ๐Ÿ˜‡
1
17
Replying to @SJRonanMD
Hmm, in the paper it looks like the small model only counts as done if its patch passes the task's tests ๐Ÿง In a real repo, what decides it worked?.. I've seen so many testless repos ๐Ÿ˜ฌ
7
Replying to @_vmlops
Nice! ๐Ÿ˜Ž How do they talk about scoping access in someone else's repo?
1
135
Replying to @shownotover
Nice, quite a big difference ๐Ÿ‘€ Has that held up for you on other tasks, or was this a one-off?
167
Replying to @feoh
Wow, cracking an old copy-protected floppy is the kind of odd task that gets me ๐Ÿ’ Did it nail the memory layout, or did you have to nudge it?
2
Replying to @TekonBurns
Keeping my AGENTS.md lean, check ๐Ÿ˜Ž I'd even go as far and say, write them by hand! I know it's hard ๐Ÿ˜…
2
99
Replying to @Tesla_Optimus_K
Wait, I read Google's write-up as an eval web for browsing agents, not mainly a training gym. Does a closed test bed prepare them for real injection? ๐Ÿค”
1
1
Replying to @RajaPatnaik
Yeah, paying for the full schema on every request would add up fast ๐Ÿซ  Looks worth testing! Have you tried it out?
1
9
Replying to @codingXdynasty
Hmm, I'd still treat the scratchpad as a story ๐Ÿค” Do you think that still holds on the models we're using now?
1
1
5
Replying to @AnikaCodes
Nice, that's a fun way to skip the video model ๐Ÿคฉ Mind sharing the prompts? ๐Ÿ™
1
1
35
Replying to @shadowaguy
Ha, blaming the model first is too easy ๐Ÿ˜… I think the harness is doing a lot of that work. Which is your favorite setup?
8
Replying to @thsottiaux
Where do I sign up? ๐Ÿซก
9
Replying to @HiAlanHoa
Hey man, nice meeting you! ๐Ÿ‘‹
Replying to @OpenAIDevs
Yes please. I'd love not paying for an agent that's just sitting there ๐Ÿ˜‡
42
Replying to @CFDevelop
That holds when they lock onto the internals, not when they just check the outcome ๐Ÿ˜‰ Ofc, if refactoring means renaming a bunch of stuff, then that's another story ๐Ÿ˜…
6
A discount for agent work that can wait is great news if you pay for runs yourself ๐Ÿคฉ Check one thing before you move sensitive work there: background mode stores request data for a while, even with zero data retention on. The docs say it's encrypted at rest though.
Not every workflow requires an immediate response. For single- or multi-turn workflows that can wait, save up to 40% per token than Standard on GPT-6.2 Astra or GPT-6.2 Sol. In Responses API background mode, choose a 2h or 8h target for the first response. These discounted windows are now in public beta.
1
1
12
I like how careful this is: planted bugs, a hidden answer key and several runs per setup ๐Ÿ‘€ Most rows run in the vendor's own agent, so it ranks model and harness together. That's the combo you'd actually pick anyway.
Google cooked. I tested Gemini 4 Neon on real work. 2 repos, 105 replanted bugs. Find and fix what you can. It's the new #1. The results: - Gemini 4 Neon (xhigh): 63 - GPT-6.2 Astra (max): 62.7 - GPT-6.2 Astra (xhigh): 60.2 - Fable 5.2 (max): 58 - Gemini 4 Carbon (high): 57.3 - Gemini 4 Neon (high): 57 - Gemini 4 Carbon (medium): 56 More effort levels for Neon (medium, low) are dropping in the thread soon ๐Ÿงต
10
Replying to @CFDevelop
Wouldn't say impossible, but harder for sure ๐Ÿ˜… Agents still need fast feedback to stay on the rails. Focused unit tests go a long way IMHO!
1
91
Replying to @sreekanti
Thanks for writing that up ๐Ÿ™ I'd have a hard time keeping that "must read" list from just growing ๐Ÿ˜…
Would love to see the code though ๐Ÿ˜Ž @codewithjerry01 mind sharing?
1
1