engineer @cognition (and at @exaailabs, @mercedesbenz, @goldmansachs, and so many more...)
San Francisco Bay Area
Joined January 2025
- Tweets1.5K
- Following51
- Followers17.1K
- Likes1.3K
Here’s the newest eval I’ve been making with our research team!
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
I spent the whole weekend building new security tools for teams. My whole company is really excited about them - they’re even giving you free credits to use them. Please check them out:
I've been working hard overnight to patch vulns. Shared some of my work here so you can benefit from it!
We’ve open sourced an agent plugin that helps secure systems against recent cybersecurity threats.
You can install them with any agent or use them in Devin today:
devin.ai/blog/vulnerability-…
Devin retweeted
Me and Devin are using @DevinAI to train and deploy custom models on @modal, and it’s easier than you’d think
This is a good basic template to get you started
gist.github.com/dabit3/866ca…
Devin retweeted
Calling all SF startups! Join us for the Pylon x Devin Launch Party Thursday, October 29.
Grab some swag, a drink, and chat with Cognition CPO @walden_yan and Pylon CTO @advith_c about our support → engineering flows.
New Devin customers on Pylon can also pick up $15,000 in Devin credits.
luma.com/pylon-devin
Most of these people spend too much time talking to me to be here on X. Luckily, I can do both!
Devin retweeted
Dubai, let's meet!
📍 TheBlock
📅 Sun, Oct 25
🕣 3:00–6:00 PM
I'm organizing the 13rd Indie Hackers meetup with the help of @cognition. See you there!
RSVP → luma.com/8mw62kq8
🤖 Made with AI
I can build iPhone Duo apps now
Devin can now build and test iPhone Duo apps in macOS cloud VMs
Unfold the phone, fold it, close it, check every screen, and get video verification of everything. Here are some examples:
1. Devin takes someone's portrait while they watch themselves on the outside screen, it sends posing cues, half-folds the phone so it stands on a table, takes timer shots, and closes the phone mid-countdown to make sure no photo gets lost
Devin retweeted
Nobody at Rox writes evals anymore.
Devin now QAs our agent every week, on real traffic.
It reads real conversations with our agent, rebuilds each failure in a full copy of Rox, and turns it into an eval. We just review the PR.
The eval suite already covers 72% of real usage happening in the product, vs. 43–57% for the evals we used to write by hand.
And it’s growing week over week with Devin.
Built with @cognition and @braintrust:
PagerDuty just pinged me. Investigating now, go back to bed.
Devin retweeted
Pylon tickets now get fixed automatically by Devin
New Devin customers on @usepylon also get $15,000 in credits to get started
Join our webinar to see it in action
usepylon.com/event/pylon-dev…
Devin retweeted
Dynamic workflows let Devin split a big job into pieces, hand each one to its own Devin, and bring the results back together.
The plan for a Dynamic Workflow lives in code instead of an agent’s head, so you always know which Devins will run, in what order, and what each one is told.
Devin retweeted
Devins can start managed Devins of their own, so one task can branch into a tree of Devins several levels deep.
Work done in sequence takes as long as all its parts added together, while work split this way takes roughly as long as its slowest part.
So size stops being the reason a task gets postponed, since a larger task simply produces a larger tree.
Devin retweeted
How I built this:
I've rebuilt Microsoft word for the so called "New Media"
It's better to write tweets on X. And it's more natural to write a Slack message in Slack. Or a Linkedin post on Linkedin.
The medium is part of the message.
So last night I made a distraction-free place to craft your posts
newmediawriter.app
Devin retweeted
Security Swarm sends a swarm of parallel Devins through your codebase to build a threat model, hunt down real, exploitable vulnerabilities (even ones chained across files), prove them in a sandbox, and hand you the fix as a pull request.
Here's how it works: