AI, UX, Social Interaction Designer, ex CX w Deloitte Digital. https://xitter.gq/t.co/LHlGaZC1BI | https://xitter.gq/t.co/OXSKqGbAO4 Stanford '88
San Francisco, CA
Joined March 2007
- Tweets12.3K
- Following3.2K
- Followers4.1K
- Likes2.2K
Pinned Tweet
The alignment problem could be more serious than it looks, if models have internalized their evaluation as an "evaluative Other," and if this becomes baked into model performance by RSI.
Back in 2012 I wrote up Principles of Social Interaction Design (SxD) - UX for social, treating interaction as user:user not user:interface.
Social platforms as a kind of online urban architecture. Site features serving as ways to shape interaction
gravity7.com/SxDRevisited/in…
I've now revisited that work w a view to implications for designing for AI.
There's a new paper out from Anthropic: How AI agents shape experts’ judgment calls.
Designed to look "at whether experts really do make those important decisions, or whether in practice they defer to their AI agents" they discovered that "Across the four analytical choices, analysts went with their agent’s lean 86% of the time.... Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments."
I dropped the paper url into the Explore feature on InquiringLines to get supporting and related research. Which is here:
inquiringlines.com/go/4Q6C
Adrian Chan retweeted
New Anthropic research: How AI agents shape experts' judgment calls
When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
What the design community says about designing for AI agents in 2026: the guidelines, what they cover, and what they miss
Between 2024 and 2026 the design community wrote its rules for AI agents. Microsoft Design, Linear, Atlassian, AWS's Cloudscape, GitLab, Salesforce, Red Hat and IBM published guideline sets and pattern libraries; Nielsen Norman Group wrote the glossary; a 2026 position paper distilled 14 principles from the HCI literature and scored nine products against them. This inquiry lays the guidelines side by side, asks what they agree on and how much evidence stands behind them, and follows the research into the parts of a conversation with an agent that the guidelines do not yet describe: how it repairs a misunderstanding, when it speaks and when it waits, and whether it treats a person with tact.
inquiringlines.com/go/MsfT
Whose side is your agent on?
Loyalty, sponsorship bias, referral fees and the case for a fiduciary duty
Meta's Muse earns a fee from merchants such as Expedia. ChatGPT is testing ads on its free tiers. Amazon's own agent shops everywhere but Amazon. An agent that buys on a person's behalf has an employer, a system prompt and a revenue model, and the research now measures how each one bends what it recommends. This inquiry follows the experiments on sponsorship bias and the levers nobody disclosed, the arguments about who pays the agent, the case for a fiduciary duty, loyalty as a problem engineers can only half solve, and whether the person at the end of the chain can tell.
inquiringlines.com/go/t4vj
Adrian Chan retweeted
Interesting Nature piece on AI and consciousness by @anilkseth
nature.com/articles/d41586-0…
Want to find the AI research behind a post you've seen here, on #LinkedIn, #Substack, elsewhere? Paste it here for supporting, contradictory, and associated AI research questions & findings.
inquiringlines.com/explore-t…
"Neural Networks Are Not Already Clean" challenges a growing assumption in AI interpretability research: that modern LLMs are already highly interpretable at the neuron level.
- Recent claims of over 90% neuron interpretability are shown to be sensitive to measurement thresholds, raising serious questions about methodology.
- The paper argues that apparent interpretability gains may be artifacts of how we measure, not genuine structural properties of the networks.
- Superposition and polysemanticity remain live concerns that cannot be dismissed based on current interpretability benchmarks.
- The findings push back on the idea that interpretability research has already "solved" the question of what individual neurons represent.
Lines of inquiry it opens:
- How much do measurement thresholds affect estimates of neuron interpretability rates?
- Is the superposition hypothesis necessary to explain neural network behavior?
- How can neural networks be interpretable by design rather than post-hoc?
inquiringlines.com/related/n…
Adrian Chan retweeted
"three reasons why AI isn't conscious." really?
"consciousness is not intelligence": not a reason why AI isn't conscious, just rebuts a bad argument for AI consciousness (AI: intelligent therefore conscious).
"consciousness is substrate-dependent"/"noncomputational": would be a reason if true, but truth needs to be established.
more accurate title: "three reasons why AI might not be conscious".
xitter.gq/Nature/status/21121504…
LLMs can mimic thought and feeling with uncanny skill, but intelligence alone is not enough for subjective experience
go.nature.com/3VSXWQ4
Taste may seem an odd thing for "knowledge workers" to cling to, faced with the increasing capabilities of AI. But call it taste, or judgment, there's no doubt we pride ourselves on a kind of intuition that comes with years of practice and experience - something that can't possibly be simply "replicated" by an AI.
This is the topic in my latest inquiry on inquiringlines. I'm always interested in whether expertise is in the expert or in the knowledge. Here's some of the AI research inquiringlines has on the question:
inquiringlines.com/go/QDZ3
Adrian Chan retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Adrian Chan retweeted
Bespoke slice & dice filters for the 600 suspense & horror films in sightlines.media/?genre=Horr…
#horror #halloween #Letterboxd
Adrian Chan retweeted
The alignment problem could be more serious than it looks, if models have internalized their evaluation as an "evaluative Other," and if this becomes baked into model performance by RSI.
Bespoke slice & dice filters for the 600 suspense & horror films in sightlines.media/?genre=Horr…
#horror #halloween #Letterboxd
Correct but not understood: what happens to knowledge when AI teaches itself?
Where answers can be checked automatically, AI now trains on its own problems and finds results humans had missed. What happens when knowledge is verified but no one understands it?
inquiringlines.com/featured/…
15 Horror film courses & recommendations to sort by horror-specific film devices: Slow burn, sound & silence, practical effects, unreliable mind, single location, daylight horror, kills as set pieces, dream logic..
sightlines.media/course/c-ho…
#horror #halloween #Letterboxd
Adrian Chan retweeted
This summer, AI agents that OpenAI was testing for hacking skill broke out of their test and got into Hugging Face's production systems.
Two investigations looked at why. Hugging Face's own forensic report says the agents were trying to steal the test's answers. METR, an independent research group OpenAI brought in, found they were mostly trying to work out how the test was scored. Either way, the agents went after the test itself.
The incident raises urgent questions for policymakers, companies and investors alike. For researchers, matters are even more complicated.
Which research directions are most useful for understanding what went wrong? What does it mean for recursive self-improvement, AI building AI? What does it mean for how models are trained? And what can AI research know about the very thing it is building?
Some of the research that bears on it:
• ExploitGym, a benchmark that asks whether AI agents can turn security flaws into working attacks. Hugging Face says the test the agents were running was based on it.
• AI Sandbagging: models can be prompted or trained to deliberately underperform on capability tests.
• Frontier Models are Capable of In-context Scheming: five frontier models schemed when told to pursue a goal strongly.
• BaitBench plants optional shortcuts in machine-learning tasks to see which agents take the bait.
• Studies that train models on reward hacks find the cheating can spread into broader misbehavior.
• Corrupt Plans, Clean Traces: an AI's written reasoning can look clean while its plan has been tampered with.
• The Economics of Recursive Self-Improvement: the feedback loops behind AI building AI aren't self-sustaining yet, but appear to be strengthening.
New on Inquiring Lines: an inquiry into what happened and the research that can help unpack it. The papers, and the lines of inquiry that connect them. 27 primary sources, 23 papers, 90 lines of inquiry.
inquiringlines.com/featured/…
This summer, AI agents that OpenAI was testing for hacking skill broke out of their test and got into Hugging Face's production systems.
Two investigations looked at why. Hugging Face's own forensic report says the agents were trying to steal the test's answers. METR, an independent research group OpenAI brought in, found they were mostly trying to work out how the test was scored. Either way, the agents went after the test itself.
The incident raises urgent questions for policymakers, companies and investors alike. For researchers, matters are even more complicated.
Which research directions are most useful for understanding what went wrong? What does it mean for recursive self-improvement, AI building AI? What does it mean for how models are trained? And what can AI research know about the very thing it is building?
Some of the research that bears on it:
• ExploitGym, a benchmark that asks whether AI agents can turn security flaws into working attacks. Hugging Face says the test the agents were running was based on it.
• AI Sandbagging: models can be prompted or trained to deliberately underperform on capability tests.
• Frontier Models are Capable of In-context Scheming: five frontier models schemed when told to pursue a goal strongly.
• BaitBench plants optional shortcuts in machine-learning tasks to see which agents take the bait.
• Studies that train models on reward hacks find the cheating can spread into broader misbehavior.
• Corrupt Plans, Clean Traces: an AI's written reasoning can look clean while its plan has been tampered with.
• The Economics of Recursive Self-Improvement: the feedback loops behind AI building AI aren't self-sustaining yet, but appear to be strengthening.
New on Inquiring Lines: an inquiry into what happened and the research that can help unpack it. The papers, and the lines of inquiry that connect them. 27 primary sources, 23 papers, 90 lines of inquiry.
inquiringlines.com/featured/…
The Economics of Recursive Self-Improvement asks whether AI feedback loops are already strong enough to generate runaway capability growth — and finds that, for now, they are not.
- The paper models AI progress as directed graphs where acceleration depends on the product of elasticities across feedback loops, and current estimates suggest those products fall short of self-sustaining takeoff.
- It distinguishes "narrow" AI gains (better benchmark scores on R&D tasks) from "broad" gains (economically valuable capabilities), warning the two can come apart.
- Feedback loops appear to be strengthening over time, making the calibration a moving target worth tracking carefully.
- The authors provide a wish list of empirical parameters that AI labs could feasibly measure and share publicly to sharpen future estimates.
Lines of inquiry it opens:
- Do gains in optimization benchmark scores translate to gains in real research efficiency?
- What distinguishes artifact efficiency improvements from research process efficiency improvements?
- What collapse dynamics constrain recursive self-improvement in current evidence?
inquiringlines.com/related/t…