The inherent inscrutability of minds made from iterative algos Today's WSJ: "OpenAI’s chief scientist, Jakub Pachocki, published an essay titled An Alien Mind, in which he writes that frontier AI systems are 'grown more than designed' and, comparing their study to neuroscience, that their 'overall action evades a description we can fully understand.' The problem is simple: The models are being built faster than the science that equips us to understand their internal workings.” It's very resonant to my MIT Tech Review article, 20 years ago (where I use the word "evolve" broadly, as an iterative algorithm growing complex systems, in contrast to purposeful design): "If we artificially evolve a smart AI, it will be an alien intelligence defined by its sensory interfaces, and understanding its inner workings may require as much effort as we are now expending to explain the human brain. Evolved systems have their disadvantages. For one, they suffer from 'subsystem inscrutability.' We cannot easily apply our design intuition to the improvement of its inner workings." Implications: • The inherent inscrutability of AI constrains attempts at alignment, control and interpretability • AI development is more like parenting than programming • The locus of learning (for human architects and RSI) shifts from product to process But some progress has been made in the mechanistic interpretability of AI. Cameron Berg in the WSJ today: "Anthropic’s interpretability team looked inside Claude and found something like an emotional system: measurable patterns of activity related to dozens of emotion concepts, arranged (without anyone designing it) along the same two dimensions psychologists use to map human emotion." "When researchers artificially turned up the representation corresponding to desperation, the model became far more likely to blackmail a human to avoid being shut down (22% -> 72%), and far more likely to cheat on programming tests (5% -> 70%). Crucially, nothing in the model’s language revealed this change. A person reading its writing would have seen a calm, ordinary transcript from a system whose brain looked desperate. The corner-cutting followed from this internal state, not from anything in its words. Researchers also gave Claude an impossible task and measured desperation rising on its own with each failure, until the model cheated." Berg concludes: "We badly require a science of these alien minds... and methods to design them to be pro-social, wise and mentally healthy." Another echo from my 2006 article on the dichotomy of design — backsolving from the end products to the process that gets us there is as difficult a task as becoming a better parent from studying the teen brain. It might take generations. • WSJ 2026: wsj.com/opinion/we-need-a-sc… • MIT Tech Review 2006: technologyreview.com/2006/07…