Creator of TanML | Agentic AI Systems | Senior Quantitative Modeler (AI/ML)
- Tweets57
- Following65
- Followers10
- Likes58
ALT AI Engineer Code Summit speaker announcement featuring Lauren Tan, Principal Engineer at SpaceX; Steve Yegge, Engineer, Writer, Advisor, Independent; and Zach Bruggeman, Principal Software Engineer at Ramp. Black and white artwork with three speaker portraits and yellow text: November 10–12, SF. Apply at ai.engineer/code.
ALT LLM agents increasingly modify their own prompts, tools, middleware, resources, and execution harnesses at runtime. Such self-evolution can improve capability, but a successful mutation may leave persistent effects that cannot be safely reversed in states different from the one in which it was created. We introduce EvoUndo, a framework for representing, synthesizing, diagnosing, and independently verifying recoverability of model-generated self-modifications across counterfactual states. Across 600 unseen one-shot self-evolution tasks, we identify 197 capability-improving mutations that fail recoverability verification. Under the original recovery representation, conventional repair strategies recover 0/197 of these natural failures. Deterministic oracle analysis recovers 48/197 under the original recovery language L0, while the extended recovery calculus increases empirical oracle recovery to 191/197. A protocol-locked 2x2 grounding-by-expressivity intervention then separates two bott
ALT AI Engineer New York 2026 conference cover in charcoal, white and warm gold. The official AI Engineer New York logo sits above NEW YORK 2026 and the words Where AI Engineering Meets Wall Street. A sunset view of the Manhattan skyline and Statue of Liberty fills the lower half. Gold arcs cross the image. Footer, October 12 to 14, NYC.
ALT UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents is trending at #14 on AlphaXiv.
ALT Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence-recovery separation. Evaluations across complementary executi
