Towards Intelligence in Action
I keep wondering what the agents of the future will look like. I am convinced they will not be systems that sit and wait for a user to speak and then answer. They will act in the world on their own and keep responding to what happens around them.
I think behavior trees might be the last step toward making agents truly autonomous.
New blog: How to Control an Agent with a Growing Behavior Tree
Two ideas I explore in this post:
1. behavior trees may be the best harness design for an agent
2. an agent can keep growing its own tree by learning in tasks
blog: sihanxu.me/harnesstree/
code: github.com/EvolvingLMMs-Lab/…
lmms-lab retweeted
I keep wondering what the agents of the future will look like. I am convinced they will not be systems that sit and wait for a user to speak and then answer. They will act in the world on their own and keep responding to what happens around them.
I think behavior trees might be the last step toward making agents truly autonomous.
New blog: How to Control an Agent with a Growing Behavior Tree
Two ideas I explore in this post:
1. behavior trees may be the best harness design for an agent
2. an agent can keep growing its own tree by learning in tasks
blog: sihanxu.me/harnesstree/
code: github.com/EvolvingLMMs-Lab/…
For the first time, we see clear signals that codec beats uniform frame sampling on general video tasks.
Today we're releasing LLaVA-OneVision-2.5, with major efforts from @GlintLabAI.
In OV-2, we aim to prove that codec can work for video understanding. We trained on video using both ways, frame sampling and codec, and each did better on different kinds of tasks. We observed codec can work but mostly good at temporal grounding tasks, not on all video tasks.
But we think it can work in a more unified way. Even a single image can be stored in a compressed view, like JPEG. Video is even less a pile of independent images.
In OV-2.5, we show that codec input comes out ahead of frame sampling on most of the general video benchmarks.
More deltas on OV-2.5:
1/ One model, two modes. The same model and the same chat template serve both offline and streaming video. In offline mode,
2/ In offline mode, the model is good at understanding extremely long video. 9× spatial compression and a 262k context let it read 1,000+ frames in a single pass. And it beats Qwen3.5-4B on 16 of 17 video benchmarks.
3/ In streaming mode, the model is proactive and knows when to speak. It watches video streams at 24 fps, and the 4B model ties an 11B streaming model on OVO-Bench and more than doubles the next best score on StreamArena proactive interaction.
Weights and training code are open today, and the technical report is coming soon.
Github: github.com/EvolvingLMMs-Lab/…
HF: huggingface.co/lmms-lab-enco…
shout out to @ziyongfeng @yiye24549678 and more contributors.
keep pushing!
5
150
New LMMs-Lab column is live---Slack of Shining
First post:
The Hard Road to Real Evaluation
Harder benches. Longer leaderboards.
But is a +2 score really better ability or just better exam fit?
We break it down:
• period × lifecycle as the map
• runtime leakage & grader taste & scoring policy
• what dynamic & environment eval actually buy you
If interested, please check our main content:
lmms-lab.com/posts/hard_road…
🤖 Made with AI
early this year, when people say "multimodality is useless or harmful" we say, "it's good."
then we see frontier models like Astra/Fable start to have impressive capabilities to understand and interact with environments.
this is just the start. we will then see models interact, gather feedback, and improve themselves efficiently.
multimodality is good, and it is a precondition for next-level intelligence.
we dont claim anything bold, we just build, to benefit people and the world.
check out our thoughts at lmms-lab.com
Try it! And we will continue to update.
More and more designs, interesting results and fancy demos will open for community.
🔥 Open Sourcing 𝐓𝐡𝐞 𝐇𝐚𝐫𝐧𝐞𝐬𝐬
🐎 One harness to rule them all
A unified harness for building, testing, and advancing general-purpose agents across diverse environments
🖥️ Computer-use agents
🤖 Robot-use agents
🎮 Game-play agents
- Project: harness.lmms-lab.com/
We believe future agents should be autonomous.
Today we’re open-sourcing The Harness, our first step in that direction: Automatic Control for intelligence in action. It runs on its own, thinks while it acts, and learns as it works.
Released under the MIT license: free to use, modify and build on, including commercially.
It’s early and there’s a lot we still need to get right, and we’d love your feedback.
GitHub: github.com/EvolvingLMMs-Lab/…
Docs: harness.lmms-lab.com/
OpenAI's new CoT monitorability post is not "smarter models are harder to monitor".
It is that CoT habits survive distillation.
In plain words: they took two teachers with similar downstream capability but different CoT controllability, and SFT'd the same base on each teacher's rollouts. The students kept the gap.
This matters for anyone training on model-generated data. When you pick a teacher, you are not only picking its answers. You are also picking how it writes its reasoning.
There are also some details:
1/ no-CoT capability tracks base model perplexity smoothly.
2/ GPT-6 Astra looks like a jump on the timeline.
3/ controllability here is measured with toy instructions (insert "meow" between words, all lowercase), and the authors call it a noisy proxy for monitor evasion.
xitter.gq/jeffreygwang/status/21…
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle?
We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
2
6
487
Always-On!
We have built this systemic analysis for personal multimodal agent.
Action-Sensing-Memory can explain all!
Follow us to find more interesting works about multimodality.
1/15 Personal agents are the hottest topic in AI right now. Frontier AI labs have shipped products such as Muse, dots, Claude Duet, Grok Bot and Gemini Daily Brief, and wearables like Omi and Bee capture conversations through the day. Robots are moving the same way. Frontier models are already becoming the robot's high-level brain, and the next step is a life companion that knows you better. What would make an assistant truly always-on, and how far are we from it?
We wrote a survey to answer this: "Always-On Personal Intelligence: A Roadmap Towards Continuing, Self-Evolving Loops". It is joint work from NTU, UC Berkeley, Princeton, Stanford, NUS, KAUST, Dartmouth, UNC Chapel Hill and Northwestern.
We describe an always-on assistant as Sensing, Memory and Action linked by six channels. We review each module, place 119 systems on six levels of loop closure, and find two channels with the thinnest support: Memory guiding what to observe, and action outcomes returning to Memory. We also release HALO, an open-source tool and MCP plugin for checking these channels in real systems.
📃 Paper: preprints.org/manuscript/202…
🌎 Website and demo: always-on-intelligence.githu…
📚 Reading list: github.com/Always-On-Intelli…
🔧 HALO: github.com/Always-On-Intelli…
🧵👇

ALT Cover of the survey Always-On Personal Intelligence: A Roadmap Towards Continuing, Self-Evolving Loops. On the left are the title, links to the paper (doi.org/10.20944/preprints202611.0793.v1), the website and the reading list, and the logos of the nine institutions: NTU, UC Berkeley, Princeton, Stanford, NUS, KAUST, Dartmouth, UNC Chapel Hill and Northwestern. A line below reads: 632 works cited, 119 systems placed on six levels, 40 works audited with HALO. On the right is the paper's first page.
The sim problem is often argued to be solvable via parameter passing and coordinate systems, without the need for visual modalities.
But what about real-world settings?
Many may marvel that Fable 5.2 appears to have cracked robotics, yet when confronted with the dynamics and variations of real scenes, it still demands faster, more flexible visual modules and preprocessing pipelines.
We remain convinced that real-world deployment ultimately comes down to a multimodal problem!










