Pinned Tweet
Start with something like PopperPad.
PopperPad is scientific memory for humans and agents: an append-only, falsification-first knowledge ledger for hypotheses, recipes, evidence, counterexamples, artifacts, and typed relationships between claims.
github.com/TheDarkLightX/Pop…
D.#dwards retweeted
Shaping & Monitoring Models through their Internal Representations
Mechanistic Shaping trains models to restrict targeted internal concepts or behaviors without hurting task performance, while keeping those concepts detectable by refitted probes.
arxivb.org/abs/2611.07071
D.#dwards retweeted
Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the KV-cache into a tree of compressed summaries. Queries search the tree from top to bottom, narrowing down which tokens deserve attention.
With just 1B tokens of additional distillation training, ZHA preserves most of the performance of full attention on models up to 35B parameters. On RULER at 256K context, it achieves 93.3% vs 96% accuracy, while attending to just 1-6% of the KV-cache.
alphaxiv.org/abs/2610.42112
D.#dwards retweeted
Exactly, without making the training procedures & datasets public, any claims of "emergent" properties in artificial intelligence (AI) solely from marketing claims by AI companies should be taken with a huge grain of salt.
Its not scientific to take the claims as facts.
D.#dwards retweeted
The statistical modeling, used in AI, is pretty much brute-force to some extent because it needs tons of data to model events as sampled from some fixed true distribution P().
That distribution is there in reality but not fixed, it's changing due to complex interactions.
Before trying to solve any problem, define the limits. This shapes the solution space. The Bekenstein bound for example.
So we have huge data centers, lots of space, RAM, storage, to store these increasingly growing models, which I see as fuzzy layered lookup tables. It's very inefficient compared to a brain running on 20 watts, but it's the best we have under classical physics.
Time complexity, space complexity, we have ways to determine if a problem can be solved in reasonable time. It's a tradeoff. You can make a lookup table big but then you lose space. You can keep it small, now you lose time. Modern AI hasn't changed this.
The gamble is time investment. You don't have infinite time. Some problems take more time than our universe has. Some problems take more atoms than our universe has. Picking problems well, is the main skill of mathematicians, programmers, and scientists.
Einstein for example spent years on E=MC2. There were hints it was tractable, from the logic side, but many people invest their whole life into a problem which they never solve. He solved it in finite time.
The same applies to people, each person has a finite amount of time to solve any problems in their life. And each problem takes some time. Hard are problems which cost a lot in time and resources for a human to solve.
Easy and hard, are really just due to not having enough resources to find a solution. Any solution which you have to solve brute force, and that takes intractable amounts of time, is hard. AES 265 is hard, it's intractable to brute force it.
If we had an efficient search algorithm, we'd have ASI. Give a problem, and with enough data and compute, it will find the grain of sand in the seemingly endless desert of candidate solutions. LLMs can't do this. And logical AI can't do this. Both aren't efficient enough.
The problem is, if you use LLMs to help you search, or you use logic, you still have the ocean of state space to search and no efficient algorithm to do it.
But while LLMs can search wide and combine many different works, it struggles to go deep. To go deep you need the logical search. The LLM is a fuzzy search, can't be sure about anything. Logic lets you zoom in, as much as you want.
With augmentation, from LLMs, now all of us can search more efficiently in a particular way. The breadth of a search is augmented. What would have taken months or years to read through in libraries, takes mere minutes to hours thanks to LLMs.
If you gave me unlimited time, I'd be able to reproduce all the great inventions and mathematics from the great minds, I'd have enough time to search brute force. But if you give me 70 or 80 years, my time is so limited, that I can't even afford to research thoroughly.
I see all problem solving as reducible to search. Whether it's me doing it, or Einstein, or Neother, or Boole. Any problem is a search. And every search is a finite search. We have limited resources (time, space, etc).
As a defender, if you've got more money, you've got more compute. Which means you have more agents, better agents, unless the attackers are state sponsored. You have swarms too, and your swarms can improve defense.
