Research Scientist @ Collinear AI | PhD @ Cornell Defensive Cybersecurity
Sunnyvale
Joined July 2020
- Tweets83
- Following83
- Followers149
- Likes320
I am not crazy! I know they sold us the same data. I knew it was the same. Down to the same typo in row 4,821. As if I could ever make such a mistake. Never. Never! I just – I just couldn't prove it. They covered their tracks, they got that idiot at the data brokerage to lie for them. You think this is something? You think this is bad? This? This chicanery? They've done worse. That exclusivity clause! Are you telling me that two vendors just happen to have exclusive rights to the same data? No! They orchestrated it! They sold it to us twice! And I bought it! And I shouldn't have. I made them a preferred vendor! What was I thinking? They'll never change. They'll never change! Ever since they were founded, always the same! Couldn't keep their hands off other people's data! But not our vendors! Couldn't be our preferred data vendors! Selling it back to us! And THEY get to be featured on our blog? What a sick joke! I should've stopped them when I had the chance! ...And you, you have to stop them!
new GTM strategy: hack a company, dump a sample of defensive cyber data, then book a 15-minute intro call
If we're want to improve defensive cyber capabilities for all models we'll need to do it as a community.
By popular demand we’re open-sourcing CWE-Bench on Hugging Face.
AI practitioners can now access frontier cyber evals and understand what makes cyber data high-quality. We would love feedback and contributions on results and improvements.
Each CWE-Bench task gives an agent a real open-source repository and an audit instruction. The agent must find and fix the vulnerability without being told what it is or where it occurs. Deterministic verifiers check that the fix blocks the exploit and preserves legitimate behavior.
This is the largest Cyber release in the AI community!
Learn more on the HF blog.
Excited to see what you build.
Gonzalo Gonzalez retweeted
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
With the ease of access to expert level offensive capabilities from agent, I expect many more irresponsible leaks like these from bad actors. It is crucial for enterprises to be up to date on the SoTA for defensive capabilities to mitigate damage to themselves.
This quoted post is unavailable.
Realistically though attackers will be constantly running their agents against your infrastructure and reacting to CVEs will be too slow. Proactively scanning your own systems with the strongest available agents can catch and patch vulnerabilities before attackers are aware.
At Collinear we’ve already gone through the effort of creating a map for which agent is best at detecting the most hidden vulnerabilities. Scanning your codebase with the best of each vulnerability class puts you one step ahead of attackers. Check out cwe-bench.com
Gonzalo Gonzalez retweeted
Exploration is hard for RL when training high-DoF dex hands ✋
X-Reset makes generalist object-centric RL easier by resetting to human hand-object states 🙋♀️
We mitigate the exploration problem, and train a single policy to manipulate a variety of objects to any goal pose 🧵
Gonzalo Gonzalez retweeted
Computer-use agents are limited by the interfaces humans design, typing a spreadsheet cell by cell when one line of code would do.
Can we let them discover their own interfaces through evolutionary search? Come to our poster and find out.
📍Continental Ballroom 6 at 2.30 PM at the Lifelong Agent (LLA) Workshop.
#COLM2026 #LLA, 2:30pm
openreview.net/forum?id=ctXj…
Gonzalo Gonzalez retweeted
Introducing MatFlowBench, our benchmark for LLM agents running materials science workflows.
Agents carry out first-principles calculations or design budget-limited measurement campaigns, then draw conclusions the resulting data support under the task’s stated physical assumptions.
Website: matflowbench.com
Blogpost: blog.collinear.ai/p/introduc…
Gonzalo Gonzalez retweeted
Stoked to share that cwe-bench is now the official benchmark for frontier labs for cyber capabilities.
congrats @sundarpichai and GDM team on a Pareto frontier cyber model 🚀
Lots of discussion out there about our next model(!), so I wanted to give an early look as soon as possible. Introducing Gemini 4 Argon!
It shows frontier performance in complex workflows, cyber defense and software engineering. Teams are using it extensively at Google, from coding to quantum computing, great feedback.
Here’s a look at the benchmarks:
There is quality control for vendor data that frontier labs buy to ensure their models don't go to shit; I'm sure you can add checks on model-generated data to avoid incentivizing killing humans (unless we want to benchmaxx that, in which case I'll share you samples)
To be clear, as we say in our latest Risk Report (xitter.gq/AnthropicAI/status/208…), I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought (xitter.gq/AnthropicAI/status/206…).
Gonzalo Gonzalez retweeted
We’re also introducing Gemini 3.8 Flash Cyber, our most capable cybersecurity model. It shows frontier-level performance in discovering vulnerabilities and patching them at scale, with Flash-level speed & pricing.
That includes achieving 86.2% on the important CyberGym industry benchmark, plus 47.2% on CWE-Bench for patching. We saw a 70%+ success rate in discovering vulnerabilities across 20 programming languages on our internal benchmark.
Today we’re releasing CWE-bench: 100 held-out audit-and-patch tasks testing whether coding agents can defend real code.
The leading agent passes 47%. 18 tasks remain unsolved by any agent.
This means frontier models on our benchmark have de-correlated errors.
Blog: blog.collinear.ai/p/cwe-benc…
Gonzalo Gonzalez retweeted
Everyone has been talking about their agents escaping sandboxes since the July incident where two of @OpenAI models escaped a sandboxed cyber eval via a "read-only" registry proxy and ended up in @huggingface prod infra hunting for benchmark answer keys.
But rarely people talk about the environment or the simulated world around the agent.
I wrote a essay on it and propose a measurement for the environment's attack surface 🧵
Blog link: blog.collinear.ai/p/cybersec…
