Faster, better decision-making for people helping superintelligence go well

San Francisco
Joined September 2019
I once worked with an AI researcher who became more direct. He started sharing disagreements with colleagues, standing up for himself when someone took credit for his work, telling women he was attracted to them, and texting friends “I miss you.”
16
5
4
632
105,745
Chris Lakin retweeted
Response speed is a surprisingly strong indicator of how easy it will be to work with somebody
15
19
6
545
18,256
Chris Lakin retweeted
Sooo Claude figures out who wrote the prompt, knows that that person works for Anthropic, *and* Claude's responses to that prompt are more positive compared to e.g. METR's prompt !!!
Replying to @BethMayBarnes
> Given how good the models are at author attribution there's a good chance they know who wrote each prompt, which would potentially explain that pattern if it's ~Ant ~sycophancy plus eval awareness. Seems plausible! I asked Opus 5.5 in incognito mode on my personal claude.ai account to give me its top 5 guesses for the author under a few conditions. (The correct answer is me.) * High thinking, web search on: I'm listed 5th * High thinking, no web search: I'm listed 3rd * Low thinking, prompted to answer immediately with minimal CoT: I'm listed 1st. I'm impressed by this given my relatively small online presence! (Note that it has a few more bits of info from the fact that I wrote the surrounding prompt as well, maybe other prompters would get worse results.)
4
23
2,180
one lesson here is "saying what you think can be surprisingly effective"
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
1
1
56
1,986
there's a risk in being unskillfully "too empathetic" where you interrupt the other person's natural development of autonomy
allowing people to own their problems is a huge act of kindness. this does NOT mean you can’t help them, or be kind and patient. a lot of people conflate help with “stopping someone from feeling hard feelings” or “shielding them from the consequences of their own actions.” the latter is enabling, and it takes away someone’s opportunity to learn, and prevents them from fully living in reality
1
58
1,749
Chris Lakin retweeted
Last month, we demonstrated high-rate alchemy: turning mercury into real, stable gold, beating CERN’s record by a factor of 10. Last week, we enriched isotopes in the world’s most powerful plasma centrifuge, beating the Soviet record that stood for 40 years. Now, we’re building a fusion reactor in San Francisco to achieve record power density.
546
1,293
1,226
10,401
2,376,602
there exist worlds that are actually just offense dominant and you probably don’t want to adopt a policy that burns those worlds down
7
5
1
72
2,492
is this Good?
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for: Pretraining at scale without a pretraining team. Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals. Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale? Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run. The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range. We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward. We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
686
One time I tried to find the best dentist in San Francisco and ended up deciding it was the dentist 3 doors down the hall from my old dentist
1
36
1,581
Chris Lakin retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
34
85
15
667
34,596
yeah so many people at the labs don't have much "strategic time" where they zoom out and think big picture. (or, they try to but get bottlenecked on doing it effectively)
I think most people multi-tasking with agents could benefit from slowing down most of the stuff you're doing wont really matter- what's your P0? what would it look like if you gave it your full attention?
1
19
1,078
this is a good illustration of how emotions are more like gradient updates than rewards or value functions. i think a lot of people in AI think that emotions are like rewards or whatever, but they miss that you *don't feel stressed* when you feel like you're taking effective actions — no matter how much work there is, no matter how big the problem is. when you feel your policy is optimal, there's no need for stress *so you don't feel it*
I love working on AI safety because I get to benefit the most important issue of our time, arguably. If I wasn't working on AI safety, I suspect I'd feel far more stressed and powerless about AI.
4
7
154
4,573
Chris Lakin retweeted
why aren’t more rationalists jacked?
13
2
2
75
9,373
if you wanted to bet against RSI but still believed LLMs could grow a lot more in revenue, how could you express that bet?
8
14
2,855
My prediction error decreased 75% and I’ve been so happy ever since
1
1
24
1,552
asked someone what they thought the real bottlenecks to RSI were and they replied "Better not to discuss I think". instant respect
7
2
1
255
14,174
Chris Lakin retweeted
Cofounder of Cohere : `Even in ZDR and “we won’t train on you” regimes, derivative data is usually carved out. The promise is only not to train on exactly the data you put in, rewritten data is fair game.` xitter.cf/aidangomez/status/2097…
Synthetic data derived from production user data of consumer AI tools is used for training. I’ve heard this rumour from both large labs’ employees. In particular, if you’re doing something “interesting” like working on complex math/business/software/bio problems you’re dramatically more likely to get trained on because they filter/up-weight towards those usecases where the model has the most to learn. Even in ZDR and “we won’t train on you” regimes, derivative data is usually carved out. The promise is only not to train on exactly the data you put in, rewritten data is fair game.
1
1
4
497
tbc he was a client
6
1,633
once you've resolved most of your internal conflicts and regained trust in your intuition, there's no need to use willpower because you take all the best actions without trying
I once worked with an AI researcher who became more direct. He started sharing disagreements with colleagues, standing up for himself when someone took credit for his work, telling women he was attracted to them, and texting friends “I miss you.”
9
23
2
519
12,409
wow this prototype sounds 10x better than i expected
I am making the most expressive possible instrument for live performance. Calling it the Discordion. The buttons correspond to Roman numeral chords, thumb modifies the chord, and movement + orientation in 3d space transforms the sound. There will also be a right hand component
15
1,215