David Reber (@davidpreber) profile picture

#AIsafety, interpretability, and causality. PhD student in CS @ #UChicago.

Joined June 2023
Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)Photo by David Reber (@davidpreber)
Encoding semantic similarity into vector similarity is a basic building block of AI systems. When studying this, we usually just think of a single fixed notion of vector similarity (e.g., cossim). However, semantic similarity is fundamentally *contextual*---a fixed similarity notion shouldn't suffice! Turns out the information geometric lens both greatly clarifies this apparent contradiction and also yields a simple and effective technique for getting contextual semantics with vector search. @_AkashKannan_ @KihoPark_ arxiv.org/abs/2610.06919
1
6
2
44
3,795
Democracy depends on an informed electorate. But political issues and ballot measures can be confusing, obscuring the effects of one outcome versus another. Moreover, politics is personal. Once we make an initial decision about an issue, it can be hard to change our mind or see things from “the other side.” And talking about issues with those with whom we disagree can be challenging, especially when the conversation feels more like a debate than a discussion. Technology offers ways to alleviate these difficulties, but not without introducing problems of its own. The Internet and social media promised new ways for people to connect, discuss issues, and learn from each other. But in practice, both often inflame passions, solidify echo chambers, and spread misinformation. More recently, LLM chat interfaces may help people stay informed through personalized access to information, but mainstream chatbots tend to match user beliefs rather than clarifying or challenging them.12 Without the kind of pushback you’d encounter in a discussion between disagreeing friends, chatbots are ill-suited for helping people think through political issues in a balanced way. The goal of CivicChats is to address these shortcomings. Starting with ballot measures, CivicChats helps people better understand political issues through three different modes of discussion: a Q&A mode for understanding what a measure does and what’s at stake, an argumentative mode that presents competing views to your own, and a reflective mode that helps you examine and develop your own thinking.
2
17
5
35
3,472
David Reber retweeted
Most mech interp work relies on activation patching, but patching activations destroys previous computation. What if we want to use a different mechanism on the same residual stream? We propose dynamic weight grafting to interpret finetuned model weights. 🧵 1/n
2
7
1
31
5,870
Predictive Interpretability > Mechanistic Interpretability Prompting is the best method of scientific inquiry we have to study LLMs It's socially devalued because it doesn't include much d/dx,O(),etc. come to poster #3503 to talk about this or anything re: the science of LLMs
3
16
1
99
11,559
Semantics in language is naturally hierarchical, but attempts to interpret LLMs often ignore this. Turns out: baking semantic hierarchy into sparse autoencoders can give big jumps in interpretability and efficiency. Thread + bonus musings on the value of SAEs:
11
60
5
298
27,401
1/n You may know that large language models (LLMs) can be biased in their decision-making, but ever wondered how those biases are encoded internally and whether we can surgically remove them?
1
9
1
13
2,719
Reward models are a key ingredient to understanding LLMs. But what they *actually* reward can be a mystery. Turns out we can measure this directly! The trick is to use LLMs to produce "rewrites-of-rewrites" datasets to measure attributes in isolation. arxiv.org/abs/2410.11348
2
23
1
88
9,180
David Reber retweeted
Are LLMs just doing next token predictions? It is believed that if an LLM can accurately predict the next tokens in a Wikipedia entry, it essentially "learns" the information. But do pre-trained LLMs actually need to understand context sentences to solve this task? The answer is no!
5
43
2
192
40,520
David Reber retweeted
Progress in empirical ML fields like interpretability is driven by success metrics -- but how good are these metrics? @JosephMiller_ et al find "faithfulness" measures are sensitive to arbitrary design choices, calling into question previous interp claims.
1/ When you find a circuit in a language model, how do you test if it does what you think? Just accepted to COLM 2024, our new paper (@bilalchughtai_ and William Saunders), investigates this question and finds a number of common pitfalls. 🧵
1
19
978
Fundamentally, high-level concepts group into categorical variables---mammal, reptile, fish, bird---with a semantic hierarchy---poodle is a dog is a mammal is an animal. How do LLMs internally represent this structure? arxiv.org/abs/2406.01506
11
113
21
604
81,631
LLM best-of-n sampling works great in practice---but why? Turns out: it's the best possible policy for maximizing win rate over the base model! Then: we use this to get a truly sweet alignment scheme: easy tweaks, huge gains w @ybnbxb @ggarbacea arxiv.org/abs/2406.00832
5
20
1
83
16,250
Cool paper on limitations of watermarking, but I don't find it discouraging. Weak watermarking may still be very useful. Analogously, fingerprinting is imperfect, and it's easy to avoid leaving fingerprints. But fingerprinting is still really useful. arxiv.org/abs/2311.04378
2
1
22
1,765
Interested in identifiable causal representation learning with flexible nonlinear modules? 👇 A summary of our #NeurIPS paper “Nonparametric Identifiability of Causal Representations from Unknown Interventions”. 📜 arxiv.org/abs/2306.00542 @MPI_IS @CambridgeMLG @DataSciColumbia
2
18
3
118
14,558
There's an idea that LLMs encode high-level concepts linearly in representation space, and that these can be understood using geometric operations (e.g., cosine similarity) But: What does "linear" even mean? And, why would (Euclidean) geometry encode meaning? arxiv.org/abs/2311.03658
19
209
21
1,384
301,353
Agency comes in degrees, and varies along multiple dimensions. Understanding it better may help us avoid dangerously agentic systems, and find ways to preserve human agency. In this 3rd post, we explain how aspects of agency are about causal relationships alignmentforum.org/posts/Qi7…
Today, we’re starting a blog post sequence explaining how causality underpins many critical aspects of safe AGI, including agency, incentives, misspecification, generalisation, fairness, and corrigibility. We explain past work and open questions. tinyurl.com/caus-safe-agi
6
16
1,840
Today, we’re starting a blog post sequence explaining how causality underpins many critical aspects of safe AGI, including agency, incentives, misspecification, generalisation, fairness, and corrigibility. We explain past work and open questions. tinyurl.com/caus-safe-agi
2
52
5
181
37,372