writing as a rehearsal of language. language as a rehearsal of logic. pretraining research
Joined October 2022
- Tweets2.4K
- Following1.1K
- Followers2.2K
- Likes10.2K
Pinned Tweet
I spent the last six months trying to deconstruct Taalas's patents. We think it can be better: 100x fewer memfetches than their bitROM, better software by better quantization than their hardware team could.
So we wrote a compiler that:
> takes a huggingface checkpoint,
> quantizes the model
> descends the weights down metal layers to RTL and GDS, do your DRC, Yosys, PEX, make the electrical waveform execute a mat-vec from your huggingface checkpoint (thanks Cambricon tech papers)
in ~7000 lines of human-readable code.
we're looking for someone with contacts with a foundry/access to 7nm PDKs or contacts at a cheap EuroPTW shuttle? I'm broke and unemployed.
@itsclivetime this is what i think is the future of perplexity per picojoule. @zerohedge @zephyr_z9 you guys wanna see 40,000 tokens/sec?
One of the few papers in the last few years that's made me want to find who the author is and say thanks! The paper before this (Belief-State Transformer) and the paper that follows (Full Bandwidth Transformer) are also lovely reads!
alright my guys if you are interested in figuring out what happen when you bias a transformer towards predicting it’s own next hidden state I have a 2h interview with @jayden_teoh_ on the subject!
method is ridiculously simple and has ton of benefits like modelling new york
Progyan retweeted
alright my guys if you are interested in figuring out what happen when you bias a transformer towards predicting it’s own next hidden state I have a 2h interview with @jayden_teoh_ on the subject!
method is ridiculously simple and has ton of benefits like modelling new york
Progyan retweeted
Introducing Nestor, a new generation of scalable architectures that generalize MoE transformers and explicitly promote sparsity in both attentions and experts, hence learn structured deep representations more efficiently. Work with @TianzheC and @druv_pai: ma-lab-berkeley.github.io/Ne…
Progyan retweeted
I was going to have a nice quiet evening and then saw ETH is false
y’know there’s things you learn in your first CS class, and you just get used to them. SAT taking exponential time is one of those things… I never questioned it, smarter people have thought about this forever…
Progyan retweeted
Here's a direct link to OpenAI's PDF claiming a refutation of ETH. In particular, it's claimed that 3SAT on n variables is in about exp(n/(log n)^{1/2+o(1)}) time
github.com/openai/math/blob/…
Progyan retweeted
I converted Qwen3 1.7B to use my tokenizer.
Results: 1.5x prefill and generation, 1.5x the context (more data in same number of tokens)
Quality loss is minimal
Check it out huggingface.co/SwayStar123/Q…
Might try for much larger models now that this is proven out
Introducing Combinatorial BPE Tokenizer!
Standard BPE tokenizers have vocabulary for many variations of a core word, particularly; commonly words with a space infront of them (eg "Hello" and " Hello" as two tokens), as well as various capitalization variants.
Why not fold those all into just one core token?
Progyan retweeted
Gemini 4 Argon is underrated 😱.
When auditing the traces, we found that Pi Harness just ends the run whenever it hits a fallback (e.g. for security task) in 20% of the runs. We rerun Argon with Antigravity Harness, and now it beats Astra and sits right behind Opus 5.5. Congrats to the team @GoogleDeepMind and looking forward to the new ckpt.
Progyan retweeted
I did a deep dive into what would the interplay between RL policy and Inference frameworks look like if one were to do RL on heterogeneous systems. The post covers Blackwells for the trainer and TPUs for inference.
gauravjain.bearblog.dev/hete…
A lot of it is extrapolation from a ton of brilliant work that has been discussed in the industry and academia covering topics like Routing Replay, Score Centering and Masked Importance Sampling. Let me know what you think. Appreciate all feedback.
Progyan retweeted
Hot take: Most autoresearch today is just glorified overfitting.
Current autoresearch pipelines are essentially outer-loop searches for recipes that work in ONE specific setting. Try transferring a modded-nanogpt recipe to a different language or dataset. How much of that "optimal" recipe still holds up?
We need to move beyond static recipes toward inner-loop autoresearch: optimizers, architectures, hyperparameters, etc., continuously adapting DURING training based on the training trajectory itself.
Check out OPAL that enables this! Cool work with @SimulatedAnneal, Jia-Hao Liang, and @vincesitzmann.
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
Progyan retweeted
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:
Pretraining at scale without a pretraining team.
Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.
Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?
Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.
The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.
We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.
We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
Progyan retweeted
Ok hear me out, upcoming Macs are rumored to have a 512GB RAM (!) option. What if we just made a machine that ditched main memory? It'd be a herculean effort, plus making RAM non-volatile would suck in many ways, but fixable through AI at writing time or runtime, maybe better suited for a new generation of OSes
The giant upsides are tremendous low-level kernel + software simplifications, no more disk roundtrip, 100x throughput without caching, etc? Imagine ditching file system in favor of pure in-mem data (still standardizable at the boundary) then spill the rest to cloud like today
