Princeton CS prof and Director @PrincetonCITP. Coauthor of "AI Snake Oil" and "AI as Normal Technology". https://xitter.cf/t.co/ZwebetjZ4n Views mine.
Princeton, NJ
Joined December 2007
- Tweets13.5K
- Following573
- Followers133K
- Likes25.6K
Pinned Tweet
This week I had the honor of speaking to Princeton’s entire incoming undergraduate class to address their AI anxieties. I had three messages for them — good news, bad news, and a note of optimism. Here’s a condensed version.
The good news
We have enough evidence now to conclude that the shrill predictions of rapid, massive job loss were misplaced. Even in a field like software engineering where AI has been rapidly adopted, its effect has been to shift, not replace the role of the human (see the “decide-execute-deliver” framework normaltech.ai/p/why-ai-hasnt…)
Similarly, the panic about what to major in is also misplaced. There will be enduring demand for computer science, philosophy, and just about everything else. (In fact, AI companies hiring philosophers has been a big recent trend.)
The bad news
AI seems to help senior people much more than juniors. I can use AI for coding because I spent 25 years learning how to code, which lets me supervise coding agents effectively. (See my post on the “growth cycle” vs the “dependence spiral” xitter.cf/random_walker/status/2…)
You are in a bind — you can’t offload your skill-building to AI, but you’ll graduate into a market where employers will expect you to get work done with AI. We never faced this dilemma. As a result we haven’t figured out how to revamp our classes to help you do both. You’ll have to help us figure it out. And you’ll need to somehow resist the constant temptation to turn to the shortcut machine.
The hope
My point is not that AI is bad for learning. It’s an incredibly flexible tool. Is the internet good or bad for learning? Depends — are you using it to find research papers or waste time scrolling? I use AI every day for learning. The key is to use it to increase, not decrease your cognitive load. To learn deeper, not faster. There is no learning without the cognitive sweat. I try to make sure I’m mentally exhausted at the end of the day. I do feel that AI lets me push myself harder than I ever could before, and I have a vision that as AI continues to advance it will enable human-AI “co-superintelligence“. (I talked about this at the end of my ICML keynote. normaltech.ai/p/what-will-be…)
There’s a big, under-appreciated reason why people may have very different experiences and opinions about using AI for work — are they using it for tasks they’re already an expert at, or tasks they can’t do themselves? The former leads to a *growth cycle* and the latter leads to a *dependence spiral*.
When I use AI to do something I’m an expert at, like coding, I treat it as a tool. I can build quickly, maintaining an understanding of the code, knowing that if necessary, I can fix the code myself. It feels empowering. It frees up my time to think about the complex, judgment-oriented parts of software engineering that I can’t or won’t delegate to AI. That means my own skills improve rapidly, and I get to climb the ladder of complexity and develop higher-level skills, much more so than when I write the code myself. I feel in control. I can lock in and achieve a flow state — when AI is working, I’m reviewing, building understanding, and planning the next steps. I never get the feeling that the tool is about to replace me. This is the growth cycle.
(Of course, the growth cycle is not automatic. I still need to exercise agency to use AI responsibly. But it’s the same challenge with any productivity-enhancing technology, and those who’ve navigated such transitions before are well-equipped to navigate it with AI as well.)
On the other hand, if I use it for tasks I don’t understand and haven’t learned to perform myself, I have no choice but to treat it as a superintelligence. If something breaks, the best I can do is ask AI to fix it and hope for the best. I generally can’t evaluate the quality of the output myself. The only way to find out if it's any good is if and when the work is ultimately reviewed by an actual expert. The experience is confusing, unsettling and disempowering. And forget about flow state. By over-relying on AI, I risk losing whatever skill I had at the task in the first place, even if it boosts productivity in the short term. This is the dependence spiral.
It’s no wonder that entry-level workers and students preparing to enter the workforce find themselves in a bind. To compete with the AI-enabled productivity of more seasoned workers, they must adopt AI themselves, but doing so risks the dependence spiral. I have some thoughts on solutions that I will share in later posts, but I think having a clear diagnosis of the problem is a useful first step.
Arvind Narayanan retweeted
Please stop posting this take. It’s a bad take!
Bankers should be concerned about cybersecurity and AI (and trust me, they are).
People should not panic about their bank accounts being hacked by AI and zeroed out. If your bank account is hacked, and the money is taken, that is an unauthorized transfer under the Electronic Funds Transfer Act. As long as you provide timely notice to your bank, you’re not liable for the loss.
People should be concerned about being tricked into authorizing fraudulent payments, because they often have liability in those cases. And the potential for AI to supercharge scams is definitely a concern.
But if you’re going to post about those concerns, be specific and practical, not vague and ominous. Panic about the safety and soundness of the financial system serves no one.
A good example of why full automation is never just a matter of AI hitting some capability threshold — how do robotaxi companies deal with riders passed out, having medical emergencies, giving birth (!), vomiting, pooping, or just leaving without closing the car door? Some combination of first responders, remote workers, ground staff, and gig workers, apparently! jalopnik.com/2214767/waymo-r…
Arvind Narayanan retweeted
Firms are doing a lot to capture value from their own AI adoption, and yet often don't succeed. We are trying to understand the bottlenecks in practice better. If you have visibility into firm adoption whether as an employee, FDE, consultant, VC, or in some other capacity, we would love to hear from you.
docs.google.com/forms/d/e/1F…
Arvind Narayanan retweeted
Yeah, I think it's a bit unfortunate how some of the discourse surrounding this - even from smart, knowledgeable people - has underplayed or omitted prosaic, clearly fixable factors.
People often gesture at Hugging Face, etc., as self-evident vindication of worries about catastrophic misalignment leading to human extinction, but it's unclear whether the incidents we have observed are more likely under the doomer worldview than the 'normal technology' one.
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Arvind Narayanan retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Arvind Narayanan retweeted
AI as a Rohrschach Technology test.
Ever talk to someone who was around in the very early days of the internet? They did some incredible hacking back then too. "Normal" transformative tech never feels normal.
Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert.
normaltech.ai/p/the-ai-as-no…
Arvind Narayanan retweeted
We gave GPT-6 Astra and Claude Fable 5.1 open-ended AI research questions from unpublished papers, with a $10,000 budget each, and had the original authors grade the papers they produced. Compared to our first tests with Claude Opus 4.8 in July, we saw meaningful improvements in experimental design and resource management, but persistent limitations in creative problem solving, judgment, calibration, and communication; both papers were rejected by our experts.
cruxevals.com/research-notes…
Arvind Narayanan retweeted
An update on our evaluations of AI agents conducting open-ended AI research.
Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol.
We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers.
Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget?
In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management.
But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers.
We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!
ALT screenshot of the landing page of the research note introducing these results
Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert.
normaltech.ai/p/the-ai-as-no…
This is one of the most important questions about the future of AI. But steering AI development and deployment away from the "automate and compensate" bargain that the public hates is easier said than done, and there are few concrete proposals for how to do it. We just published one: normaltech.ai/p/steering-ai-…
As LLMs evolved into agents, earlier best practices no longer serve us well. This new research field needs a field guide, so we wrote one. If you're new to agent evals, please take a look. Seasoned researchers and those interested in steering the community will also find many useful observations. sage.cs.princeton.edu/trends…
New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems?
In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.
ALT Figure 1 from the paper with the list of 33 emerging trends
"agent evals are about to become as important as building the agents themselves" xitter.cf/V1kiiiii/status/211251…
This is our thesis as well. I would go further — building an eval *is* what it means to build an agent.
In my ICML talk, I argued that as AI capabilities advance, more and more human effort will shift from building to evaluation, in many different domains. normaltech.ai/p/what-will-be…
My posts on research careers, professional practice beyond academia, and AI's impact on work seem to resonate here, so I've compiled about a hundred entries: cs.princeton.edu/~arvindn/ad…
I hope it is helpful to read it as a collection, because it reflects a coherent underlying philosophy and approach to careers that I've adopted in my own work.
There's a lot of career advice online. Two things are distinct about this collection:
- Grounded optimism. I'm candid about the structural challenges, twisted incentives, and other limitations of the institutions we inhabit. I share ideas on how we can adapt and thrive in these flawed systems, and how and when we should challenge them.
- Connecting research and practice. Much of my research is on AI agent evaluation and AI's effects on the economy and labor, which lets me share insights into how we can effectively adapt and integrate it into our work.
Let me know what you think and what else I should write about!
A similar mindset shift — from "what can AI do?" to "what should I let AI do?" — is also needed for individuals using AI for productivity, not just companies. I’ve seen it in my own work. For the last few years, my workflow continually kept changing as AI enabled me to automate more and more types of grunt work. But for the last six months, once I hit my stride with coding agents, my workflow has been surprisingly stable, because capability is no longer the bottleneck to delegation for me.
To be clear, capabilities have continued advancing. I could probably have AI do my research and write my essays end to end. The quality might not be as good, but that’s not the most important reason why it would be a bad idea. The value of my essays is the voice and perspective that my collaborators and I bring. Whatever value there is in AI-generated essays, readers can get it by using AI themselves. In addition to this intrinsic reason to avoid AI overuse, there is also an instrumental one. The set of tasks for which verifying AI output is faster than doing it yourself is narrower than people think at first, and even narrower once you consider long-term skill atrophy.
I'm sure the current equilibrium will gradually change, but I'm enjoying it while it lasts!
We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"
I'm often asked what the cruxes are that separate the AI as Normal Technology worldview from that of our critics. I think this is one of them. We think this mindset shift is hard but necessary and achievable. Many others think that as AI capabilities continue to advance, we will inevitably give up control. Creating and compiling indicators of disempowerment and analyzing which direction they're moving remains one of the most important but underappreciated and underfunded areas of AI safety.
We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"
We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"
Arvind Narayanan retweeted
this is totally worth a read! will shift your perspective on how warped the current direction is towards automation and how much more we can do on augmentation.
on "Problem #4: How can we evaluate, and optimize for, human-AI collaboration?"
I learnt a lot from our initial attempt (centaur bench: arxiv.org/abs/2608.18554), but also learn how complex/multi-dimensional the problem is. This essay does a great job of laying out some of those dimensions.
I'd tried to put together something very quickly for our phd students (linkedin.com/pulse/why-measu…) and this picture still guides my thinking here.
There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-…
Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed.
- We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges.
- We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing.
- We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power.
- We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal.
- We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect.
- Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates.
We're not against automation. But we think the goal of AI must be broader.
This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.
Arvind Narayanan retweeted
While automation dominates much of the AI conversation, what other research directions should we prioritize?
@random_walker and @_gabrielunger argue for a broader research agenda in "Steering AI Beyond Automation."
Read the essay here: normaltech.ai/p/steering-ai-…
Arvind Narayanan retweeted
Here's an agenda that's very attractive from the AI safety worldview and a broad coalition can get behind.
There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-…
Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed.
- We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges.
- We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing.
- We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power.
- We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal.
- We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect.
- Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates.
We're not against automation. But we think the goal of AI must be broader.
This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.
There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-…
Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed.
- We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges.
- We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing.
- We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power.
- We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal.
- We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect.
- Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates.
We're not against automation. But we think the goal of AI must be broader.
This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.