Davide Alba

Our future hinges on whether recursive self improvement will take us from cool coding agents to literal superintelligence. The guys who wrote AI 2027 are extremely bullish on the tech, but very worried about where it leads us. Permanent totalitarian regime? Autonomous AIs taking control? I sat down with @thlarsen and @romeovdean to learn what they think is required to reach techno utopia. There’s a lot in their proposal worth discussing. Historically the AI safety community pushed for secrecy around AI research. The thinking went: the more insights are shared, the faster it will go, and we need more time for alignment research, so publishing and open source are bad. But the AI Futures Project now recommends the opposite! They want to mandate open-sourcing the research and literal code used for training runs and inference (though not model weights), so the whole world can see what’s going on inside the labs and weigh in on it as we progress. They think this would lead to frontier labs popping up all over the world, competing openly and sharing insights along the way. Almost a hybrid of academic and commercial development models. As labs explore new approaches, anything that looks really risky would be debated openly and, potentially, banned for a period of time while control and alignment methods catch up. Worldwide bans would be feasible by way of a multi-lateral treaty initially formed by the US and China. Global rules would trickle down to state/national regulations. Compute would continue its massive scaling, with a catch: new datacenters would be equipped with chip destruction/disabling machinery and strategically located so that if the treaty broke down, each country would have power and incentive to destroy/disable their chips (rather than let their adversaries seize them), bringing us back to pre-2030 levels of compute. Mutually assured compute destruction (MACD), they call it. Even with these controls, they think we’ll still have something like 4x growth in robots year over year leading the robot population to quickly eclipse humans and take over most jobs. They have an elegant idea for handling all the job loss: auction off a fixed annual supply of permits to robot makers and use the revenue to fund “citizen’s dividend” checks. Their econ model says we could pay people $1M/year for doing nothing by the mid 2030s. They forecast doublings of GDP for many years and extreme gains across most markets, as well as for land, one of the main remaining scarce resources in a scenario of abundance. To a normie this all sounds completely nuts. To the truly AI pilled, it’s worth taking seriously. Which are you?