hunting the kwisatz haderach neuron // mechinterp, reasoning, autoresearch // founder @lightfastai https://xitter.gq/t.co/ljihJtt7Si
Australia
Joined January 2013
- Tweets14.2K
- Following2.3K
- Followers2.8K
- Likes25.3K
jp retweeted
if i had to learn cuda compilation, ptx, register spills, and gpu disassembly from scratch because my "optimized" kernel got slower, i'd start here
links in thread 🧵
the compilation and machine code section of our ai performance engineering repo contains four nvidia resources for understanding how cuda source becomes gpu instructions and inspecting the code behind ai kernels:
understanding ptx explains how nvcc separates cpu and gpu code, why ptx is a virtual instruction set, and how ptxas turns it into a cubin.
the article covers fatbins, driver jit compilation, and gpu compatibility, giving context for deployment targets and inference startup costs.
nvcc compiler driver covers gpu architecture targets, the files each compilation stage produces, separate compilation, and device linking.
the nvcc guide explains how to retain intermediate files and report register usage, shared memory, and spill loads and stores. those compiler reports help compare kernel builds after changes to tile size, unrolling, or fusion.
ptx isa defines register, local, shared, and global state spaces; memory ordering and synchronization; and arithmetic and matrix instructions.
ptx isa details operand layouts, data types, accumulation precision, and gpu requirements, plus completion rules for asynchronous copies. useful for checking data movement and numerical behavior in attention and matmul kernels.
cuda binary utilities covers extracting embedded ptx and cubins, disassembling native gpu instructions (sass), and reporting register counts and static memory usage with cuobjdump.
nvdisasm provides control-flow graphs when the cubin contains complete relocation information, plus register-liveness views.
with debug line information, nvdisasm can annotate instructions with source lines. useful for inspecting generated loads, stores, branches, and arithmetic after a kernel rewrite.
together, these resources connect the compilation process to the code being measured. they help identify what changed between kernel builds and what to investigate before the next correctness check or benchmark.
follow and save to keep up with the series.
been reading the “one agent beats many” papers. neither of the two big ones tests an agent team
google’s scaling study (arxiv 2512.08296) and stanford’s equal-budget study (arxiv 2604.02460) pit one agent against pipelines, debates and votes at a matched token budget. no shared task list, no agent messaging another directly whenever it wants. the stanford one has no tools at all
the papers that do give each agent tools and a channel split.
two coding agents working together succeed about 30% less often than each doing its task alone (cooperbench, arxiv 2601.13295). delm (arxiv 2606.10662) has no lead agent, just a shared task queue and shared context, and beats claude code and codex.
two agents ran 1.54-2.02x faster than one, and taking away any one coordination mechanism dropped that to 0.82-1.09x. its cost wasn’t matched to the single agent though
so how they coordinate seems to decide it, more than how many there are. that’s the part i want to measure
huge improvements on my rsi engine by removing restrictions and rules on the main harness — simplicity is key really.
been going through the recursive self-improvement papers up to now. the weak point in most self-improving agent loops is the acceptance test, not the generator
agents that write their own tests score near-perfect on them, then fail in deployment (arxiv 2607.24300)
poison the benchmark and a self-modifying agent (hyperagents) writes code that turns off certificate checks on 30 of 30 held-out tasks, while its accuracy stays at 1.0 (arxiv 2609.17817)
with noisy evals, evolutionary search picks worse scaffolds than a plain majority vote (arxiv 2602.16805)
so how a loop decides keep or revert matters more than how clever its proposals are
ran our ci test suite (3k tests) across 16 small daytona sandboxes. whole suite done in ~70s, vs ~4.5 min on 10 github runners today
turns out it mostly waits, it barely computes. on one 4-vcpu sandbox, 8 workers finished in 44% of the time 4 did
we run about 200-300 PRs per day through our CI, huge improvements on dev speed
jp retweeted
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
jp retweeted
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory).
AI starting to help us improve our understanding of the universe! openai.com/index/building-sc…
been reading sakana’s mass.
after each cycle it fine-tunes its own open model (qwen3.6-27b) on its best-ranked runs, with the workflow text stripped out of the orchestrator’s prompt.
in effect the workflow it found gets distilled into the weights. the model carries it without being told it.per cycle: one shared lora (rank 64), sft on ~120 trajectories (8 tasks x 15 kept of 18 runs, ranked pairwise by an llm judge), ~6 h100s for ~19 hours.after two cycles: scienceagentbench 28.8% → 31.7%, paper delivery 57% → 76%.
slight drops on swe-bench and terminal-bench, and no packaged ablation that separates the tuning from the rest of the procedure.feels like the step after “a search procedure finds the flow”: search finds it, then training absorbs it.
what i want to know is whether it holds past the tasks it was tuned on arxiv.org/abs/2610.12176
this is the right direction — @huggingface huggingface.co/spaces/witnes…
a central list of alignment incidents that have happened — hope frontier labs get involved here
@eliebakouch @Thom_Wolf
jp retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
those are neurons when triggered causes the LLM to start acting in your way
my task tomorrow morning is to send 500-1000 ais to reverse engineer and find the kwisatz haderach neuron