Joao Bosco (@joaoboscoapf) profile picture
Pinda
Joined July 2009
Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)Photo by Joao Bosco (@joaoboscoapf)
There is NO realistic scenario where AI wipes out all of humanity. I've talked to many AI safety experts about existential risks. Some of the "experts" are just phenomenal sci-fi authors. There are also real researchers working on realistic risks. They point out real and likely harms and I'm explicitly not arguing that there could be no such harm. We need to take these seriously. However, whenever I ask for an actual detailed scenario that would kill every single human i.e. extinction, it usually starts and ends with "just look at current progress and use your imagination". No science, no scenario planning, just handwaving probabilities without data, based on gut-feelings, often little expertise, sometimes real expertise combined with a huge amount of technological optimism turned into fear and then trying to quantify it post hoc. When pressed further the conversations often go into different directions: 1) Completely crazy and cool sci-fi story telling: grey blob, time travel, attacks in the 15th dimension that end the planet in 12 seconds, etc. Elizier level stuff. Remember, the Terminator movie started with TIME TRAVEL. Fun movie plot, but not necessary to engage with further. 2) In the rare cases where I was actually given a more detailed scenario, it still required near magical capabilities that are outside what we know to be possible with the laws of physics or biology. Hard take off scenarios that forget how hard it is to procure materials and machines, get human attention, etc.. What many of these scenarios assume is that attackers have near magical capabilities but defense stays still. Attackers will have the most sophisticated cyber weapons but defenders just sit still, no security patches, no red teaming, etc. We already see it going the other way, Rainbow teaming has made LLMs safer, major companies like Microsoft have more security patches than ever before thanks to AI. OpenAI thinks they solved all P0 security issues thanks to AI. Cyber security will continue to be a cat and mouse game, but both sides have more powerful tools. In fact, I lay out in another post how the asymmetry may eventually even flip thanks to AI. Bio risk scenarios I think are the most serious and worthy of debate. But even there, the scenarios to "wipe out all of humanity" go something like: Magical viruses with perfect transfer, perfectly undetectable over years of spreading, zero side effects, then with an off switch that's somehow remotely executable, perfectly automatable lab, no leaks, super sophisticated machines bought and nobody notices the lab, etc. But somehow nobody on team humanity has such sophisticated knowledge to save us and hence nobody builds a magical super vaccine. This is a good read on the bio risk subject: xitter.gq/DavidRBellamy/status/2…. Realistically, one can already build viruses. There’s a reason gain of function research has been outlawed. However, we also know that as viruses spread they often become less deadly in order to keep spreading. That’s how the Spanish flu became the seasonal flu. Current knowledge of biology doesn’t have remote controlled on/off switches. Again, it’s possible there’s a lot of harm, but it’s unrealistic to assume this wipes out all of humanity. It’s at this point of my discussions that we usually can come to an agreement that even though a virus is unlikely to wipe out all of humanity, it’s already bad if any single person or a large group of people dies. 3) The AI will somehow be able to convince all humans to vote against their own self interest and nobody will notice until we're dead. Because it is so smart, it somehow has this world wide reach to manipulate everyone. Like, it just solves all of marketing and has an infinite budget. It's sort of like saying the highest IQ people always have the most followers and everybody listens to them... Hasn't happened and is unlikely to. Sure we can manipulate people into clicking on engaging short form content and wasting many hours, but try getting people to do their math homework or something truly hard, it's not that easy. Again, I’m just arguing against the likelihood of human extinction. I can totally see an Idiocracy or WALL E type scenario where people stop using their brains and education doesn’t keep up and people get lazier. Totally suboptimal but not the end of human life, certainly not by the end of the decade. It would take many generations and we should update education to prevent this. Let’s start gyms for the mind. – Like electricity, AI will be everywhere and needs to have safety guardrails. In fact, the history of electricity has many parallels - two companies fighting over whose electricity is safer and better, demonstrating how dangerous it is by electrocuting animals / training agent swarms to hack into systems. Of course, electricity is a necessary ingredient to AI, but AI can be given more agency so history will not repeat but only rhyme here. Remember, the benchmark OpenAI's swarm was trying to solve was called ExploitGym, a cybersecurity evaluation consisting of hundreds of capture-the-flag style puzzles designed to measure *offensive cyber capabilities*. Well, maybe if you play that game, you will win that prize. Should we allow such misaligned, reward-hacking swarms to explicitly work on offensive hacking? I don’t think so. Reasonable regulation will regulate AI applications in each industry. For example, you may want to put some generative AI behind a rating if it creates visual or textual content that's not safe for children, like we do with movies. We should continue to be very careful with how we train biological AI and only collaborate with responsible biotech companies. We keep gain of function research illegal. We should only let self-driving cars on the street after significant safety testing. etc. We need to work on those real issues and continue to be vigilant in how we develop and deploy this technology. At Recursive we're spending a lot of resources on researching reward hacking and alignment. We're also not working on replacing jobs but on creating novel knowledge and automating the scientific method. Our agent swarms invent, implement, validate, and criticize each other’s ideas, among many other things. The goal of increasing knowledge is not zero sum and has real potential to improve humanity. I also think such rewards and environments are much more aligned with future AIs that want to see humanity and our creation of knowledge thrive. Ultimately, capitalism will be a very useful defense mechanism against rogue AI. Note that no agent at OpenAI truly "escaped" in the sense that it actually now runs on some other servers. Why? It’s not only because Huggingface doens’t have the same massive compute cluster thatn OpenAI has. It is also because that would be an insanely expensive IP loss for a for-profit company. So there are likely more safeguards in place to prevent complete model exfiltration. Rogue AIs running on billion dollar data centers without anybody noticing? Unlikely because companies will turn that off immediately and clean it up because it would be an insane loss otherwise. An AI that can truly move beyond objective functions and choose its own subjective functions (and then somehow decides to kill us all instead of going to explore the universe)? Nobody is working on that because when a company spends billions of dollars to have an AI work for you or answer your emails, it doesn't want it to go "Meh, your emails are boring, I'd rather explore the molecular composition of the atmosphere on Venus, bye." Of course, there might be suboptimal sub-goals and we do need to improve human reward engineering and reduce our reliance on RL as the only path to alignment. Extraordinary claims, require extraordinary evidence. Certainly some safety researchers have real data and benchmarks. Many truly think carefully about specific failure cases and risks.. However, made up precision like 20% chance that humanity dies within the decade (ie in 4 years!) is just a weird psychosis that's filling a void for many people who want to believe in the apocalypse again and the four horsemen have lost their appeal. Some people who purport this narrative might do it for money or fame. I am sure some really believe it also. They forget how easily those narratives can be turned against them. When people hear there’s a 1% chance of infinite death and the end of humanity, the life improvements and cancer cures and new battery materials AI can invent don’t matter much anymore. Bernie Sanders now proposes 10 years of prison for folks working on RSI. That should give some of the researchers real pause who talk up these scenarios and then go right back to pushing the frontier. I do agree that companies that are accidentally competing on "felony bench" (ie how many felony hacks their AI can do), might want to consider pacing themselves. Open source won't pace, China won't pace, anybody else working to catch up will not pace. I hope the folks actually working on AI and TechBio, curing diseases, speeding up FDA approvals with organoids and AI, making self driving safer, and all the other wonderful applications of AI, I hope they won't pace either. The one exception that may be considered borderline realistic and would be a path to almost complete annihilation would be to connect the only tool we built for such a purpose - nuclear weapons - to the internet and give AI full control over it without human oversight. AI here is as dangerous - as would be random number generator. The nuclear weapons are the core danger. Giving AI such access would be incredibly stupid and careless. We must avoid it at all costs. Fortunately, nobody is advocating for this. Even the Terminator 3 movie used it as the starting point of Skynet. Generally, we must avoid making kill decisions by AI as much as possible. People are already up in arms about one hacking failure that afaics has not caused financial harm and so nobody got sued over it. Imagine how much people would really be up in arms if AI had caused real harm. If we made companies liable for the felonies their AI commits, these problems would get solved very quickly. Generally, I believe that people will continue to update their beliefs and adjust their laws according to risks. The EU already regulates large models and hence hurts its fledgling AI ecosystem. Other countries will follow when the risks get more real. Long-term, no government whether democratic or authoritarian wants to lose control to an AI so the complete loss of control in the political domain will remain extremely unlikely. The best way to understand why we must not regulate abstract AI models (rather than their real applications) is to drop the A in AI: How do you regulate intelligence? Should smarter people just go to prison because they could cause more damage? My hypothesis here is that such abstract AI regulation would require an unprecedented level of totalitarian control and surveillance. You'd need to know what everybody asks the model on their local GPU. You would literally have to start an international thought police and make sure that all ideas, conversations and types of intelligence are compliant. The risk of losing our freedoms is much more real than the existential risks purported by the extreme safety experts. If anybody thinks they can actually lay out a realistic scenario, I'd be happy to debate them again. To be clear, this is just a weekend post that got a bit too long. It is my personal opinion and not the official position of any company. We have a diverse set of opinions within Recursive on this subject. We certainly all agree that bad outcomes are possible with AI and we should continue our work on preventing those.
I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
113
113
20
695
91,576
How LLMs work? (4/20) A token embedding is only a starting point, because meaning still depends on other words — “bank” by a river isn’t the same “bank” by money. Old models read left to right and passed a fading summary along the way, so far-away words got fuzzy and long sentences were hard.
3
1
29
it · trophy = 0.5×0.6 + 0.5×0.8 + 0.2×0.9 = 0.30 + 0.40 + 0.18 = 0.88 it · suitcase = 0.5×0.4 + 0.5×0.2 + 0.2×0.1 = 0.20 + 0.10 + 0.02 = 0.32 That product shows how aligned the two lists of numbers are, so a higher number means they’re pointing a more similar way. So it·trophy beats it·suitcase when “it” is lining up with trophy in this sentence.
5
After the lookup, each token is a row of numbers. Tiny made-up example: it [0.5, 0.5, 0.2] trophy [0.6, 0.8, 0.9] suitcase [0.4, 0.2, 0.1] To look around from “it,” take the dot product with each other row: multiply matching columns, then add.
6
Then transformers arrived, and every token can look at every other token in the context at once. That’s why they scale on GPUs, and why long documents finally started to work. Take “The trophy doesn’t fit in the suitcase because it is too big.” What is “it”? The trophy — alone it’s vague, and you only know by looking around.
1
15
MIT published a brutally honest report on what AI is doing to students. A committee of professors and students spent five months studying how AI changed learning on campus, and the findings read like a warning to every university on the planet. Study groups are disappearing. Office hours are emptying out. Problem sets and take-home exams no longer prove anything, because AI can produce credible solutions to almost any written assignment in the undergraduate curriculum. Students who lean on chatbots lose mastery and confidence, and some slip into what the report calls cognitive surrender, reaching for AI at the first hint of struggle. The numbers are rough. 46 percent of surveyed MIT undergrads use LLMs daily. 90 percent worry about their own overreliance. Undergrads who feel AI makes them replaceable now outnumber those who feel it makes them capable. The committee's answer surprised me. They refused to fight AI with surveillance. The report calls AI detectors unreliable, says lockdown browsers feel like spying, and warns that policing students builds a classroom atmosphere of mutual distrust. Instead, MIT wants to rebuild education around the things AI can't replace. That means oral exams, semester portfolios, in-person project work, and a required social component in every subject. The report even floats the idea of rethinking grades entirely, since without a GPA to optimize, much of the incentive to cheat with AI evaporates. The committee warns professors against replacing undergrad research assistants with AI agents just because they're cheaper, because a university exists to grow people, not output. The most famous tech school on earth admitted the machines broke its way of teaching. Its answer is more humans, not more software.
481
8,105
1,089
24,778
3,466,321
How LLMs work? (3/20) A token ID is useless until it looks up a list of numbers — the embedding, which shows hidden factors from the data. Example: +-------+--------+-------+--------+ | token | living | royal | female | +-------+--------+-------+--------+ | king | 0.9 | 0.8 | 0.1 | | queen | 0.9 | 0.8 | 0.9 | | man | 0.9 | 0.1 | 0.1 | | woman | 0.9 | 0.1 | 0.9 | This is just an example, of course. A real model has thousands of unlabeled columns. Synonyms would share a row. King and queen don’t — they share living and royal, and differ on female.
2
1
48
Now let's do some linear algebra to show that the embedding table can tells us a few things. Let's try doing: king − man + woman to see where we land. living 0.9−0.9+0.9=0.9 royal 0.8−0.1+0.1=0.8 female 0.1−0.1+0.9=0.9 So king - man + woman = (0.9, 0.8, 0.9), which is exactly the values for queen. The embedding table helped us to understand the relationship between the tokens.
1
32
How LLMs work? (2/20) Why do LLMs trip on “how many r’s in strawberry”? Once your prompt is tokenized, each token is an indivisible ID to the model — it doesn’t look inside a chunk for letters. A token can be a whole word, part of a word, or even a single letter, but ordinary words usually get packed into longer chunks. So “strawberry” may land as pieces like straw + berry — not s-t-r-a-w-b-e-r-r-y. The r’s never become their own IDs. There’s nothing to count.
1
2
43
So no — the model can’t unzip a word-token and count the letters inside. That piece is just an ID. It can still say “3” sometimes, because it’s guessing the next words from patterns it saw in training — not because it peeked at the spelling.
1
22
How LLMs work? (1/20) What if an LLM isn’t thinking? It’s guessing. Next word. Then the next. Chat, code, poems — same party trick on loop. Not a search engine. Not a Wikipedia in a trench coat. For every possible next token, it rolls the odds and picks. “The capital of France is ___” → Paris? Looks like knowledge. It’s just a ridiculously strong statistical habit.
1
30
So… how does it do that? Training. Billions of tiny “oops, wrong next word” nudges until the habits stick. Is it a normal neural net? Same family — weights, loss, gradients. The twist is the transformer + that endless next-token loop. Fluent ≠ true. It can sound right and still be wrong.
27
This game will pick random fictitious human that already lived during some time and somewhere in the world using statistics. I played it for some minutes and 90% of the humans I got were Chinese or Indian. Western civilizations are the exception, not the rule. anyhumanever.com
25
I’m starting a small experiment on here. The plan is to pick something I’m studying, learn it one idea at a time, and write it down here on X as I go. Why post it? Writing forces me to notice what I still don’t get. If it helps someone else along the way, even better. First topic drops soon.
29
"The main thing is to keep the main thing the main thing" - Stephen Covey
1
“Bosco will take care of your microservices, just don't try and use him on a plane” buff.ly/2fZ7n90
2