Pinned Tweet
Introducing Continual-ARC, a benchmark for agents that learn from experience.
If you think your harness gets better with experience, we'll rerun it on 50 new ARC-style puzzles that nobody outside unify has seen and post its share learned.
continual-arc.unify.ai/submi…
OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.
On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.
Highly recommended read.
This is a topic I have been researching and writing about recently.
The way we use agent memory in harnesses today works, but mostly for simple, one-off, and general tasks.
For more novel, complex, and domain-specific tasks, you will often find that current harness and memory systems don't generalize so well.
So the big question is: how can we encode continual learning capabilities in models and harnesses?
This is where the idea of co-evolving harnesses and models begins to make sense.
This work, along with the benchmark (Continual-ARC), provides an interesting analysis of this.
The harness is not just for steering the model and providing tools and context; it is increasingly becoming a key part of training pipelines to unlock continual learning in models. The harness continues to be important, but it can remain minimal and efficient.
OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.
On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.
unify retweeted
OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.
On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.



