AI Safety Papers (@safe_paper) header
AI Safety Papers (@safe_paper) profile picture

AI Safety Papers

@safe_paper

Sharing the latest in AI safety research.

 arXiv
Joined May 2023
  • Tweets366
  • Following283
  • Followers2.5K
103Media
Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)Photo by AI Safety Papers (@safe_paper)
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples @AlexandraSouly, @javirandor, Ed Chapman, @alxndrdavies, Burak Hasircioglu, Ezzeldin Shereen, @CarlosMougan, @mavroudisv, @ErikJones313, @hkscy, Nicholas Carlini, @yaringal, @_robertkirk @AISecurityInst
Image from AI Safety Papers (@safe_paper)
1
23
1,409
arxiv.org/abs/2510.07192
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming...

arxiv.org
182
AI Evaluation Should Work With Humans @jankulveit, Gavin Leech (@gleech), Tomáš Gavenčiak, @raymondadouglas
Image from AI Safety Papers (@safe_paper)
3
3
27
1,367
arxiv.org/abs/2608.13577
AI Evaluation Should Work With Humans

AI Evaluation Should Work With Humans

This position paper argues that the dominant paradigm of AI evaluation (which focuses on superhuman autonomous performance and so implicitly targets the goal of replacing humans) is guiding AI...

arxiv.org
4
139
Simulated Control Evals @JeffOlmo, @davidzhang37, @tylertracy321 @redwood_ai
Image from AI Safety Papers (@safe_paper)
1
1
19
922
arxiv.org/abs/2610.31593
Simulated Control Evals

Simulated Control Evals

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built...

arxiv.org
125
From AGI to ASI Tim Genewein, @FranklinMatija, @AlexLerchner, @LaurentOrseau, @SamuelAlbanie, Adam Bales, Cole Wyeth, Stephanie Chan (@scychan_brains), @IasonGabriel, Joel Z. Leibo (@jzl86), @AllanDafoe, @mhutter42, Thore Graepel (@ThoreG), @ShaneLegg @GoogleDeepMind
Image from AI Safety Papers (@safe_paper)
1
1
15
804
arxiv.org/abs/2606.12683
From AGI to ASI

From AGI to ASI

Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI organisations....

arxiv.org
1
104
Interpreting Language Model Parameters @BushnaqLucius, @danbraunai, Oliver Clive-Griffin, @BartBussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, @leedsharkey @GoodfireAI
Image from AI Safety Papers (@safe_paper)
1
4
40
2,281
arxiv.org/abs/2610.20748
Interpreting Language Model Parameters

Interpreting Language Model Parameters

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in...

arxiv.org
1
238
Debate Training Reduces Reward Hacking in RLAIF Zachary Kenton (@ZacKenton1), Lili Janzer, @rorygreig1, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, @HarriLEdwards, @sen_r, @noahysiegel, @natashajaques, @rohinmshah @GoogleDeepMind
Image from AI Safety Papers (@safe_paper)
3
4
2
52
2,079
arxiv.org/abs/2608.17776
Debate Training Reduces Reward Hacking in RLAIF

Debate Training Reduces Reward Hacking in RLAIF

We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a...

arxiv.org
2
184
CheatBench: Measuring Reward Gaming in AI Agents @longphann_, Stephen K. Yang, Jason J. Lim, @MantasMazeika96, Wenyu Zhang, Zheyuan Liu, @notRichardRen, Jingxiang Meng, Yaoteng Tan, Weiliang Zhao, @addisonwu_, Matei Anghel, @hendrycks @cais
Image from AI Safety Papers (@safe_paper)
1
3
35
1,792
arxiv.org/abs/2609.36308
CheatBench: Measuring Reward Gaming in AI Agents

CheatBench: Measuring Reward Gaming in AI Agents

Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended. In recent incidents and controlled evaluations...

arxiv.org
190
Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack @CristianTrout, @sanmikoyejo, @SashaRomanosky, @StephenLCasper, @mattbotvinick, @seanmcgregor, @Miles_Brundage, @prpaskov, @AdrienLE, @ben_s_bucknall, @kevinlwei, @Manderljung, @LSzpruch, @precognita, @tom_fehring, @RuneKvist, @RajivDattani, et al.
Image from AI Safety Papers (@safe_paper)
1
4
17
1,266
arxiv.org/abs/2607.11999
Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack

Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack

From maritime trade to commercial nuclear power, insurance has been the enabler of major economic and technological developments by pricing risk, limiting downside, and spreading best practices....

arxiv.org
160
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens @kaleybrauer, @claudiomverdun, Samuel Marks (@saprmarks)
Image from AI Safety Papers (@safe_paper)
1
1
12
798
arxiv.org/abs/2607.03502
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit...

arxiv.org
141
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble @JorioCocola, @LevMckinney, @HarryMayne5, @BetleyJan, @OwainEvans_UK
Image from AI Safety Papers (@safe_paper)
1
6
26
1,207
arxiv.org/abs/2609.10883
Story Imprinting: AI Assistants Absorb Traits from Human...

Story Imprinting: AI Assistants Absorb Traits from Human...

Language models are trained to implement a helpful AI Assistant character (e.g., Claude). We explore how finetuning on synthetic stories affects this character. Does it change the Assistant's...

arxiv.org
3
141
Load more