Mingxuan (Aldous) Li (@itea1001) header
Mingxuan (Aldous) Li (@itea1001) profile picture

4th yr undergrad @UChicago (CS and philosophy), RA @ChicagoHAI and @BerkeleySky, RL env @bespokelabsai

Joined January 2024
Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)Photo by Mingxuan (Aldous) Li (@itea1001)
We got the idea to optimize how information is presented to the agent from our other work SlackBench arxiv.org/abs/2611.05512v1, where we let agents access Slack data via CLI instead of MCP tools and it turns out that they run faster, with higher pass rate. Turns out that this method can be applied to many other settings too!
Introducing our latest research: Environment Optimization (EnvOpt). An agent's cost and performance depend on 1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost. For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task. Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques. Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms Meta-Harness by 9.67% with only 15% of its cost. We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways: (1) the environments optimized for the target agents can transfer to other agents, even with different harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone. Read more in our blog: bespokelabs.ai/blog/envopt paper: arxiv.org/abs/2611.07372.pdf
2
7
27
1,418
What I've been thinking is that we should not even use "consciousness" or "experience" to talk about AI. Not because there is nothing there (it is exactly because many people feel "there is something" that we are having this debate). Because we know this "something there" (or in us) is different, awkward, alien, yet we cannot help applying the "consciousness" terminology we've been using, as if it really carries the meaning we are referring to. To operationalize this I think is to help the public, the users, build this vocabulary. The public does not want to be with this discomfort that they cannot accurately talk about this "something". If this is done nicely maybe it is not that disastrous, in some way it is similar to the function of religion as Freud saw it (i.e. for psychological comfort). But what people have been doing now about this, is bad, and will be worse once we have good embodied AI. This can be started with e.g. learning how people describe their interactions with AI in phenomenological sense, without directly using the consciousness (human-mind) terminology. In the end it is about studying humans, not AI.
2
94
Exactly. And on some studies that show "LLMs have this and that similar to human neural system": it is cool but it shows nothing really. This is especially concerning if a big organization like Anthropic is doing it. It fools the public. At least stop using metaphors and be clear what you are talking about.
Something Nagel observes in the interview is that academic philosophy of mind has, instead of better informing public opinion on this issue, only served to lend institutional credence to the fantastic, though broadly unfounded, claims of AI firms and tech CEOs. Anthropic, as Nagel remarks, is the worst of the bunch because it takes conscious steps to humanize its proprietary model with all sorts of familiar spontaneities as to better mimic human self-consciousness. Researchers, counseled by AI ethicists, are creating the impression of the thing that they believe to be the case because they take the human mind to be scarcely different from a really-complex computer. Much of the confusion surrounding consciousness and AI has to do with the misuse of metaphor.
1
95
Mingxuan (Aldous) Li retweeted
We are releasing the tasks of Multi Agent Coordination Bench and our multi agent harness that we built on top of @harborframework github.com/bespokelabsai/mul…
We’re releasing Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models. We find that current frontier agents coordinate poorly. A team of eight agents beats a single agent, but the gain comes from having more agents, not from coordination. Of Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, only Opus shows signs of a gain from coordinating.
7
17
2
66
2,799
Mingxuan (Aldous) Li retweeted
What scientific discoveries are hiding inside AI models? 🔍 Join us Dec 12 at @NeurIPSConf in Atlanta for Interpretability for Discovery. Hear from our amazing speakers: @begusgasper, @kfrankelab, @nkwang24, and @jwuphysics. Join us for spotlight talks, posters, and a panel!🎉 We’re hosting a social right after the workshop! Stay tuned for details, and a big thank-you to our sponsor @GoodfireAI for making it possible! Looking forward to seeing everyone! Organizers: @boknilev, @EkdeepL, @YNikankin, @ChenhaoTan, @amirtha255
1
13
4
38
2,570
Mingxuan (Aldous) Li retweeted
Multi-agent systems often behave like a mob, not like a coordinated team. We are releasing Multi Agent Coordination Bench: a benchmark requiring agent coordination, collaboration and communication. Right now, agents benefit from parallel exploration, but their coordination and communication is very poor. A team of eight agents does better than a single agent but often due to random exploration, while their coordination has significant room for improvement. Our tasks include agent teams trying to execute a mission to Mars, fight zombies, grow a company from seed, and more. We categorize our tasks by two properties of the environment: if the agents share the same information and actions (Shared vs Distributed), and if they pursue the same goal or not (Collaborative/Strategic). This taxonomy will be useful for future research, to think about different multi-agentic problems across distributed and strategic settings.
We’re releasing Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models. We find that current frontier agents coordinate poorly. A team of eight agents beats a single agent, but the gain comes from having more agents, not from coordination. Of Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, only Opus shows signs of a gain from coordinating.
37
57
11
493
37,170
There is often this argument that goes: though we are not sure of the existence of AI consciousness, we can study related things (e.g. welfare) in case there is. But: it risks enforcing this delusion that AI is conscious in the public. This is especially risky if big companies and labs are doing model welfare seriously.
1
2
129
Is AI safety important? Definitely. Do we have to anthropomorphize them (e.g. say we give them values) to build safe AI? No. It is better time spent possibly if you just look into your data.
1
55
Mingxuan (Aldous) Li retweeted
We’re releasing Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models. We find that current frontier agents coordinate poorly. A team of eight agents beats a single agent, but the gain comes from having more agents, not from coordination. Of Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, only Opus shows signs of a gain from coordinating.
52
71
46
621
144,240
Read this fascinating paper by @mpshanahan today (arxiv.org/pdf/2212.03551). On a related note, we should be careful of how our use of language changes, as everyone is interacting with only a handful of LLMs. One way to avoid getting yourself polluted: go read more books!
1
4
228
Quote: "However, today’s large language models, and the applications that use them, are so powerful, so convincingly intelligent, that such licence can no longer safely be applied (Ruane et al., 2019; Weidinger et al., 2021). As AI practitioners, the way we talk about LLMs matters. It matters not only when we write scientific papers, but also when we interact with policy makers or speak to the media. The careless use of philosophically loaded words like “believes” and “thinks” is especially problematic, because such terms obfuscate mechanism and actively encourage anthropomorphism."
55
Mingxuan (Aldous) Li retweeted
Introducing our latest research: Environment Optimization (EnvOpt). An agent's cost and performance depend on 1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost. For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task. Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques. Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms Meta-Harness by 9.67% with only 15% of its cost. We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways: (1) the environments optimized for the target agents can transfer to other agents, even with different harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone. Read more in our blog: bespokelabs.ai/blog/envopt paper: arxiv.org/abs/2611.07372.pdf
24
23
13
230
46,898
Mingxuan (Aldous) Li retweeted
What are idea basins and how do agents explore them? A few weeks ago, researchers from different labs independently discovered what we now call ‘Idea-basins’: As an autonomous agent iterates to solve a problem, it gets stuck into the methods it has tried before and its trajectories remain in localized regions of the solution space. Techniques like ‘Fork-and-Flush’ and ‘Scaling Discovery requires forgetting’ reduce this idea fixation by periodically deleting the session context. In this post we study how frontier models explore different idea basins for AutoResearchExam. E.g. for a problem on accelerating transformer decoder graphs on CPUs models can try Numpy+Asm Kernels, Native BLAS, JIT compilers and other idea basins. We find that some models switch ideas more than others while others get to the best idea faster and explore deeper. We find that most runs stick with one or two ideas, and models that submit more spend the extra submissions refining and overfit more. We see that models explore most in the first six hours, after which visits to new idea basins drop sharply. This suggests models start refining early leaving potential gains on the table: 58% of runs never reach the basin where the best solution to their task was found. Finally, bold exploration pays off: exploring a new idea gains about nine times as much as a refining step on average. Read more in our blog: bespokelabs.ai/blog/how-mode…
50
122
35
894
220,679
Mingxuan (Aldous) Li retweeted
Inspired by @_jasonwei, we benchmarked safety benchmarks. We read the model cards, system cards and safety reports behind 135 releases from 18 labs in 2026, including OpenAI, Anthropic, Google DeepMind, Meta, xAI, DeepSeek and Alibaba Qwen. What we found 🧵
1
13
1
38
2,310
Mingxuan (Aldous) Li retweeted
Presenting "Persona-Model Collapse in Emergent Misalignment" at @NeurIPSConf in Sydney next month! Joint work with @renatovicente. Fine-tuning LLMs on narrow harmful data makes them broadly misaligned. We propose it also erodes how models represent and distinguish personas.
1
5
6
213
Mingxuan (Aldous) Li retweeted
(1/n) Before trusting AI to make scientific discoveries, can it reproduce existing research - and go one step further to test whether conclusions survive reasonable alternative analytical choices? Sharing work from this summer with @DbrxMosaicAI, @StanfordNLP, and @StanfordOVAL on using AI agents to reproduce published medical findings and autonomously investigate analytical choices that could change their conclusions.
12
39
9
153
11,292
Mingxuan (Aldous) Li retweeted
FrontierCS 2 is our new benchmark for CS research & development. Each task is based on a paper: we delete the contribution from the released code, and ask agents to outdo the original method, as judged by the authors' chosen metrics. Agents don't have access to the paper or web.
32
76
42
513
310,515
Mingxuan (Aldous) Li retweeted
1/6 🚨Releasing Actor-Gatherer framework, a new paradigm for agent context management developed at UC Berkeley for working with multiple agents in parallel over long sessions. Actor-Gatherer decouples knowing from acting: Actor agents perform tasks and Gatherer sets their context
8
16
1
83
6,430
🔥
Excited to share ResearchTrails, a dataset of human research decision trajectories extracted from Git commit histories. Final papers show the outcome, but they often do not capture the full path: abandoned methods, omitted experiments, and ablations. ResearchTrails recovers these intermediate decisions, and we show that learning from them improves LLMs’ ability to assist with research exploration! Blog, code, data, and the trajectory visualizer are at researchtrails.org 🧵
114