LLMs are less and more than you think.
Berkeley
Joined November 2008
- Tweets1.5K
- Following587
- Followers175
- Likes1.2K
Pinned Tweet
"The natural order" is a human imagined illusion.
Nature feels no need to obey our puny human illusions of logic or order.
I take solace in the fact that there have been P(Doom) folk around for thousands of years, and they have been wrong every single time.
So far.
Nvidia/Qwen3.8-Flash-Next-FP4-Local is Amazing!
RTX6000 pro:
Tokens per second
Median 191
90th percentile 271
Peak 328
Older Docker container of Qwen3.8-Flash-Next NVFp4, same workload 85–88
"Misalignment" isn't new: We have four decades of economics literature on human 'alignment' at work. e.g. Jensen-Meckling Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure; a legal vocabulary (breach of fiduciary duty, constructive fraud), and a management toolkit (auditing, bounded authority, incentive design, termination) that are all, structurally, alignment mechanisms.
Muse, Dot, etc - Is everyone really ok, with Meta or OpenAI knowing your every email, event, purchase, human interaction, ... ?
Contracts, not prompts
An emerging programming model for agents: a written contract defines the job; a runner checks compliance, requests corrections within retry limits, and records failures.
What this looks like in software claims review:
brucedambrosio.substack.com/…
RL optimizes for verifiable outputs. What-the-code-does is verifiable cheaply. How-the-code-was-written is either gameable or requires a human eye. So RL ships working-but-sloppy code, and no one is failing. The objective function can't include what you can't measure a million times without Goodharting. Taelin's 30% isn't a bug. It's the shape of the reward.
This quoted post is unavailable.
I see huge hype around Jev. I can wire any LLM so that the task is 'output A, B, or C', then measure and renormalize the logprobs of the first output token over those three. Yup, lightning fast. Big deal. Am I missing something? Not snarky, genuinely don't understand.
This is exactly the fallacy of 'vibe prompting' - "Look mom, I typed ten words and look at the amazing thing I created!."
As LLMs evolve, they can do more and more with less and less 'prompt'. However, you get *their inference of your intention*, not your intention. The solution: contract, don't prompt.
A prompt asks for an output. A contract defines a job: purpose, inputs, terms defined once, decision rules, what is out of scope, and what makes each output field correct. The test: could a competent newcomer do the work from it alone?
A bit more, for clarity:
A prompt asks a model for an output. A contract defines a job. It opens with the purpose, says what the reader has been given and what they have not, and defines each key term once and then uses it only that way. It gives the rules for deciding, including what to do when in doubt. It says what the job does not include. For each output field it says what makes the value correct. A prompt is tuned until the output looks right. A contract is reviewed like a specification: if two careful readers could take a sentence two ways, the sentence needs refinement. The test: could a competent person new to the work do it from this document alone?
The person who checks the model's output reads the same document.
Thesis: despite the best intentions of the well-intentioned, *someone*, *somewhere*, *sometime-soon*, will develop or jailbreak , intentionally or accidentally, a rogue AI agent or swarm.
Consequence: training time 'alignment' by a subset of actors is insufficient. We need *active defenders* who can raise the cost of attack.
Who is working on this?
Happy to join the effort.
Perhaps the current belief in the generalizability of verifiable RL is just another manifestation of the 90s belief symbolic AI would scale to AGI?
9/11 - I was home in Oregon getting dressed, preparing for a flight that morning to a meeting at DARPA. My wife said: Uh, I don't think you will be flying today...
I now have Claude ask my agent for permission before changing her codebase. They negotiate and settle.
Seems only fair?
@AmandaAskell
It *seems* claude relies on docs/ for global overview and consistency. It is important, on large codebases, to remind it regularly to check for and update consistency with the actual codebase