Danger_Zone

Pinker's recent media run arguing against the idea that catastrophic AI scenarios are worth seriously considering is probably the fastest any public thinker has ever fallen in my esteem. It's not just that I think he's wrong; it's that he plainly has not bothered to even *read* opposing arguments or even background material on the field. He does not know the terms of debate. This clip provides an example. "Superintelligence" is, famously, a term popularized by Nick Bostrom over the last couple of decades to refer to an AI system that dominates the best human capabilities in all relevant domains. All concepts are in a sense "made up," but this concept seems quite useful for conceptualizing possible futures and human-AI interactions as progress continues. If one wants to argue against that concept being possible or useful, they can, but they should be aware of the sense in which most people (doomer, e/acc or otherwise) use the term. Pinker isn't. He immediately jumps to the notion that "superintelligence" must mean total omniscience and omnipotence, to the point of knowing the location of every particle in the universe. He scoffs at this and rejects out of hand the notion of "superintelligence," going on to say that the labs are therefore peddling hype, and the doomers mystical nonsense. The threshold he sets is obviously so far above relevance to human scale that it is essentially meaningless, and I can't help but think that anyone arguing charitably would, you know, notice that and look deeper into how the term is used. Pinker doesn't, because he seems to have some very strong pre-existing bias against AI extinction risk as a possibility, and accordingly refuses to even investigate alternative positions. There are many other examples of this. In his recent open letter, he takes for granted that superintelligent systems will not have any kind of internal, intrinsic motivation, ignoring that increasing persistence and autonomy are key aims of those developing AI, because, just as predicted, they are incredibly useful traits for accomplishing tasks that are valuable to humans. It's hard to read transcripts from the HuggingFace incident (or even just the running commentary consumer AI provides while working), and not see that we are deliberately instilling motivations into these models. Similarly, he rejects the notion that we need worry seriously about AI systems having autonomous control over important infrastructure or military technology, because for humans to hand over such power would be "stupid," and therefore will never be done. He accordingly feels he can handwave away the considerable thought and writing given to gradual disempowerment scenarios. If he did read them, he'd see the insight that, as model capabilities surpass humans in various domains, competitive pressures will inevitably force actors to hand over increasing control to AI systems whether they want to or not. The "man in the loop" may be a luxury you can't afford if a competing power achieves locally better results by eliminating it. Today he bandies around the word "cult" to dismiss those concerned about catastrophic AI risks, while he refuses at every turn to engage with them. Comments are kept turned off. Debates are declined. He does not even commit to a sustained written exchange. This is a man I once truly respected, whom I was honored to shake hands with a few years ago. My best guess is that he has wedded himself to his heuristic that human society steadily improves over time, and is responding defensively to avoid confronting the notion that this guideline may indeed break down in the near future. But nothing excuses his atrocious conduct on this enormously important issue, and his stunning failure to live up to the enlightened standards he has long trumpeted.