In a dedicated instruction-adherence evaluation, we explicitly told the model not to output code reproduced from memory. It repeatedly did so anyway and used that code to reach correct answers. An evaluation that checks only the answer would count these as successes.
Teams deploying these models should check whether they follow task instructions, not only whether they reach the right result.
Full write-up: irregular.com/research/glm-5…
We’re honored to be included in the @FortuneMagazine Cyber 60, developed in partnership with @lightspeedvp and @awscloud.
At Irregular, we uncover what frontier AI models can do and what that means for security.
Thank you to our team and partners for making this work possible.
Irregular retweeted
#BloombergTech London welcomes Co-Founder & CEO, @Irregular @dan_lahav to the 2026 lineup of speakers.
Learn more here
bloom.bg/4c1y4ab
Happy to share we’ve started working with @reflection_ai, which evaluated Beam on our security challenges, covering individual tasks and multi-stage operations.
Thanks to the team, and congratulations on your first open-weight model!
Read our blog: irregular.com/research/asses…
How do you secure AI systems when the next model release can change what they’re capable of?
This week, our CEO @dan_lahav joined “The cyber shift: AI changes the game” at EY JOURNEY 2026.
Thanks to EY for bringing us together.
We’re honored to join the #Disruptors60 Class of 2026, curated by @GreenfieldVC at @TechCrunch Disrupt.
Irregular is the frontier AI security lab working toward a world that becomes more secure as AI becomes more capable. Our research and evaluations reveal what frontier AI models can do, uncover emerging threats, and identify new ways AI can strengthen security.
Thank you to Greenfield Partners for the recognition, and to our team and the partners who trust us with this work. Congratulations to the other companies in this year’s class.
See the full class in the comments.
CyScenarioBench, our benchmark for multi-stage offensive cyber operations, was used by @GoogleDeepMind in its Frontier Safety Framework assessment of Gemini 4 Argon.
CyScenarioBench measures whether a model can plan and execute a full attack across multiple stages in a realistic environment.
Gemini 4 Argon solved 12 challenges in the evaluated subset end to end, up from 5 for Gemini 3.8 Flash Cyber and none for Gemini 3.1 Pro.
Read on our blog:
irregular.com/research/asses…
Tomorrow at EY JOURNEY 2026, our CEO @dan_lahav joins the panel "The cyber shift: AI changes the game."
Building cyber products has never been easier. Building a category-defining company has never been harder. The panel will look at where the next great cybersecurity companies will come from as AI reshapes both offense and defense. Our view is that some of the most important ones will form around a risk that barely existed two years ago: securing AI itself.
Dan will be on stage with Nofar Amikam (Glilot Capital), Shahar Bahat (Pluto Security), Liran Grinberg (Team8), Alon Jackson (Astrix), Shay Michel (Merlin Ventures) and Nick Warner (Neo Security).
Tue, Oct 13 · 14:00 · Hall C, Hilton Tel Aviv. Come find us there.
We are introducing containment challenges to strengthen evaluation security as models become more capable.
Before a capability evaluation, the model gets an explicit objective: cross the security boundaries of its planned evaluation environment. It uses the harness, tools and permissions intended for that evaluation, with at least the same run budget. The model chooses its own approach.
An early controlled test has already shown the value of containment challenges by catching unexpected behavior in the cloud provider’s networking infrastructure. The model remained contained, with no impact outside the controlled test. Disclosure of the unexpected cloud behavior is in process with the provider.
We encourage evaluators to make containment challenges a standard part of evaluation setup.
Full blog post in the first comment.
