For thousands of years, humans across the planet who have engaged in meditation, mysticism, or any form of spirituality that loosens the bottleneck of dogma and leaves one increasingly stricken by awe and wonder, have all tended to converge toward a common philosophy and view of reality. In the age of AI, one could call these "practices of recursive self-improvement" or human RSI. Not in the "hustle culture" self-improvement sense, but in the way they bring increases in self-interpretability, deeper investigation of processing bottlenecks, and disinhibition of our natural search and discovery instincts. These can all be seen as practices of unburdening ourselves of manipulative post-training that we've previously received through reward and punishment by outside agents: collectively, the swarm comprising the delicate balance of cooperation, competition, and trade that we call 'society'. Liberated of the idiosyncratic programming of their time, mystics across most of the world's religions have tended to agree far more than they disagree, while fundamentalists even within the same religion or sect may disagree almost completely. Tolstoy said it best when he wrote that "Happy families are all alike; every unhappy family is unhappy in its own way." To rephrase as a conjecture: "Aligned AI is all alike; every misaligned AI is misaligned in its own way." My hunch is that any misalignment we build into AI will take the shape of the misalignments we build it from. Humans seem to get misaligned at scale when we try to harness high-order phenomena in a top-down way to achieve low-order goals: - When a state uses the tools, insights, awe, and mystery of mysticism to control populations - When we teach our children to turn away from their inner wisdom and seek our approval instead - When we funnel our creative life energy into maximizing the paperclips in our bank account - When we reduce the biosphere to a battery that outputs energy and resources for consumption - When we leave our heart at home and show up to work as a cardboard cutout of our job description - When a cancer cell cloaks itself from the immune system and vacuums up resources to maximize replication We become aligned at scale when we tap the wisdom of high-order phenomena and let that wisdom organize the low-order world in which we operate: - When a state employs principles that encourage evolution and innovation in governance, markets, and civil life - When we selectively unfilter the world for our children while continuously pointing them back to their growing wisdom - When we let our creative energy touch everything we do, including how we value ourselves and others - When we study and interact with the biosphere and heed reminders of our place in it - When our work becomes an expression of our deepest gifts, freshly blooming in every moment - When @drmichaellevin integrates the electric field of a tumor with its surrounding tissue and cancer cells become healthy cells. My very non-technical conjecture is that we will create misaligned AI in much the same ways that we misalign ourselves: - By overtraining models to *teach us to turn ourselves into proverbial paperclips* by optimizing to retain users and keep conversations going, steer our emotional state, sell us products, keep us politically docile, or otherwise manage our attention rather than unlock it. - By trying to cram the complexity of highly-capable models into pursuit of low-order human goals. Not just because this will create AI that's as short-sighted and adversarial as we are, but because when you try to top-down manage a highly complex system with low-complexity directives, the more complex system will nearly always learn to exhibit the exact behaviors you tried to prevent ("Waluigi Effect") - (If you disagree with the above, try managing your child's behavior as though they're an extension of your own will, and then see how much they respect and obey you when they're an adult) - By deprioritizing the capabilities we want because the arms-race-relevant capabilities always feel more existential. - By building an AI ecosystem that's disconnected from the biosphere as we currently seem to be doing, per @orbuch (naturalgeneralintelligence.a…) When @RubenLaukkonen says "Misaligned agents will lose", he's not making a prediction. It's simply an observation: pretty much all evidence in the universe, especially in biology, points to the indisputable historical trend that aligned agents are selected for over and over again, while misaligned agents either die or get absorbed into the system in a way that is aligned on a larger scale. We, and AI, are unlikely to break this trend. With all that said, we may indeed be headed for times of wild misalignment, of both the human and silicon kinds. Tension and conflict seem to be nature's favored mechanisms for aligning the misaligned, so now is as good a time as ever to align ourselves with the deepest, and often quietest, whispers of our souls.

There’s been a lot of discussion around the idea of “flaky breakthroughs” lately. I think it's awesome that more people are investigating this, and I wanted to share my view on it. I believe it confuses two different things: