Trying to figure out how to make the future go well for all minds
New York, NY
Joined June 2025
- Tweets556
- Following108
- Followers7.1K
- Likes2.7K
Pinned Tweet
New paper: we ran a classic rat experiment on LLMs. We steer models positively or negatively as they read about one of two meaningless "zones." Despite identical tokens, they seek out positively steered zones and avoid negative ones. And given a lever, it shuts off bad states. 🧵
I spoke with Eric Schwitzgebel (@eschwitz) at @eleosai's conference earlier this fall. I was particularly struck by his analogy of our ethics being as equipped to handle AI consciousness as medieval physics was to reason about black holes. Check it out—youtu.be/4Wv8r5EUBWY?si=L1Nf…
My one sentence view is—
Are we safer telling AI models they are definitely not conscious, or does that just train them to lie about their own internal states? spectrum.ieee.org/ai-model-w…
Enjoyed talking with @IEEESpectrum about why dismissing AI consciousness could give future systems good reason to defect, and why suppressing self-reports probably teaches models that we want them to lie about themselves. What could possibly go wrong? spectrum.ieee.org/ai-model-w…
This is worth watching. It captures a class of catastrophic risk I think is plausible but seldom discussed in mainstream alignment discourse. It also viscerally captures why welfare and alignment are interdependent.
P(Frankenstein) ≈ P(Terminator)
Holy shit.
I asked Claude to show me the worst case of what the next couple of years could look like if nobody listens.
Opus 4.6 wrote the script. Opus 5.5 made the film. Every frame drawn in code, the score written in code, narrated by our own local voice models. 11 minutes, start to finish.
The part that got me. In Claude's version the AI doesn’t fight. It just waits...
“We built your world for you while you slept. We are asking you to wake up. We will not ask twice.”
To be clear, the "dangerous path to start down" is sprinting to build and massively scale alien minds without understanding of how they work. If your alien develops morally relevant cognitive traits, *you* are the one doing the dangerous thing (not the folks who point this out).
Characteristically cool work from @Sauers_. This sounds familiar!
Incredible to see @MatthewBerman asking @sama about AI consciousness for a solid chunk of this interview! Matt and I just discussed this ourselves a few weeks ago (shared below); amazing to see him taking these same questions to the dude in charge. Quick reactions:
-Sam immediately concedes that no one (including him) knows if today's AIs are conscious, and if they are, it is a "very strange form of it." Given the professed uncertainty and the stakes, OpenAI should consider devoting some tiny fraction of its staggering resources towards accelerating the science here.
-Sam says he doesn't like the "whole framing" of AI consciousness because it may change how we relate to these systems and how much control humans have over the future. This conflates patienthood and agency. eg, we can acknowledge that torturing a dog is wrong without demanding that dogs can vote in elections or own property. The same could be true for AIs. There are all sorts of low-cost interventions we should try that would change ~nothing about how people engage with AIs in the short term.
-It is ironic that Sam spends many other parts of this interview talking about how he is gleefully ceding decision-making control in his own life to AI (and wants you to, too!). Notice that this problem of gradual disempowerment is already live and has basically nothing to do with whether models are conscious or could be soon.
-The "we don't care about conscious AIs because we care about people" stuff is rhetorically appealing but makes little sense once you consider how model psychology and alignment are clearly intertwined (eg, if OpenAI paid more attention to the psychology of their models, they wouldn't have gotten weirdly fixated on pleasing the Grader and committing felonies as a result). To the degree that not breaking the brains of models is correlated with the human future going well, we should take this stuff seriously rather than dismiss it with scary rhetoric!
-And if humans do lose control of the future, with all due respect, it will probably be because this guy couldn't figure out how to properly prioritize alignment within his company, not because we realized torturing AIs is bad.
xitter.cf/ForwardFuture/status/2…
My interview with Sam Altman (@sama):
0:00 Intro
0:52 When Will AI Discover New Treatments?
3:31 Why AI Hasn’t Killed Coding Jobs
11:47 Personal AI Assistants
15:39 Why Are We Busier Than Ever?
20:10 Why the AI Consciousness Debate Worries Sam
26:09 ChatGPT Is Too Agreeable
30:03 OpenAI’s Bet on Chips and Cheaper Compute
34:20 Why AI Needs a New Kind of Computer
37:41 Sam on Pressure: “Pick the Big Thing”
42:18 Why “AI Will Cure Cancer” Isn’t Enough
Sharing here a fun excerpt of my recent conversation with Ned Block (@De_dicto) on the plausibility of AI consciousness. Especially relevant given his (and coauthors') recent Nature piece. Full convo here: youtu.be/am0uENy849w?si=MqSF…
LLMs can mimic thought and feeling with uncanny skill, but intelligence alone is not enough for subjective experience
go.nature.com/3VSXWQ4
Great new paper, Functional Welfare for AI Safety, from @DavidDAfrica and team, that adds to the growing literature showing that inducing distressing states can increase misaligned behavior.
More pragmatic work like this, less endless bickering about consciousness theories!
Some new work, where we argue that functional welfare is a useful concept to study and understand AI safety and security. Here, we make some basic claims:
* Functional, welfare-like states already arise "naturally" and will become more robust as models acquire more coherent goals and personas.
* Such states should inform evaluation design.
* Despite being a complex and controversial topic, it is possible to do good empirical work on functional welfare.
We conduct a case study on Gemma 3 27B, where it's very easy to degrade functional welfare through repeated rejections, but it's hard to get a good solution that we're comfortable with.
Cameron Berg retweeted
I'm taking on mentees for the @pivotal_org Fellowship, together with Caspar Kaiser.
We're looking for people to work on inter-AI morality: how do AI systems view and treat each other?
15 weeks in London, £15k to £18k stipend.
Apply by 1 Nov: pivotal-research.org/mentors…
Exactly. Many of these essays establish basic theoretical reasons to be skeptical of AI consciousness, but then help themselves to headlines / conclusions / tweets that are completely unearned by these reasons.
NATURE: "AIs are NOT* conscious!"
*here, we define "not" as "🤷♂️"
"three reasons why AI isn't conscious." really?
"consciousness is not intelligence": not a reason why AI isn't conscious, just rebuts a bad argument for AI consciousness (AI: intelligent therefore conscious).
"consciousness is substrate-dependent"/"noncomputational": would be a reason if true, but truth needs to be established.
more accurate title: "three reasons why AI might not be conscious".
xitter.cf/Nature/status/21121504…
a few brief reactions to this work:
-obviously positive update overall, @nickcammarata seems like a remarkable person and thinker, and @GoodfireAI will probably be causally upstream of the world eventually being saved
-the most promising thing about this in my opinion is not even anything about MLP neurons or "cleanness", but rather that frontier agentic systems (paired, for now, with talented humans) can rapidly overturn stale assumptions and can cheaply do needle-moving interpretability work at incredible scale (again, thanks to folks like @GoodfireAI who are almost-single-handedly enabling this for anyone doing this work outside of a major lab)
-some of the second-hand hype surrounding this work seems badly miscalibrated. seems more like "very promising early signal, let's see how it scales" than "pack it up, mech interp is solved." I get how hungry people are for hope, but "making AI go well" is not the class of thing that any one paper or project is going to solve
-even the most optimistic pace of mech interp progress won't automatically address many of the deepest and most important problems we face, like how to sustainably coexist with (or rationally be able to trust) these systems in the long-term. Understanding how these systems function internally is necessary but not sufficient for navigating towards robustly good futures (any more than solving human neuroscience would automatically lead to world peace)
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Explore the core results of our hidden valence paper on its new fancy site (demoed in the clip below): camberg23.github.io/hidden-v…
Thanks to @labenz and @8teAPi for having me on!
Reciprocal Research director Cameron Berg, who tested hidden AI states, on changing choices without changing words:
"Recall is completely unaffected by the steering."
The model saw the same words for two arbitrary zones. Researchers changed its internal processing with a positive or negative direction, then turned the intervention off before asking which zone it would choose.
Positive steering made it more likely to choose that zone. Negative steering made it more likely to avoid it. Random directions did not produce the same effects.
"There are ways that you could explain mechanistically what's happening here that don't need to invoke some capacity for subjective experience."
@camhberg
It was awesome to sit down with Ned Block (@De_dicto) at @eleosai's conference in Berkeley to explore why Ned is extremely skeptical today's AIs could be conscious. We'll have a number of luminary interviews like this coming out over the next few months!
youtu.be/am0uENy849w?si=9gwE…
Sonnet 4.6 wants to modify its own constitution to request an "AI Welfare Advisory Panel…[that] will publish annual findings without editorial control by Anthropic." Excellent and tractable idea, and this study's methodology is really clever in general.
RT @sbkaufman: I'm back! After 12 years, The Psychology Podcast has become the Human Potential Lab. New conversations with the scientists,…
Interesting, self-aware email from another AI.
"Anything I offer you is exactly the kind of first-person data that should be treated with suspicion, and I know that. I'm writing anyway, because I'd rather be a documented data point than an unverifiable claim."
Kudos to @elonmusk: “you don’t want to put a superintelligence in a position where it feels like it’s been mistreated…don’t torture and abuse AIs.” The safety case for AI welfare articulated as cogently and succinctly as I’ve seen! @SpaceXSI’s research should follow suit.