Pinned Tweet
We’re so back! The calls to train models in Europe again were getting louder and louder and we worked incredibly hard to make this happen. It was a pleasure to build these teams from the ground up! Now off to do some euromaxxing. Looking forward to everyone trying the model!
We worked out efficient distributed calculation of global quantiles for quantile MoE load balancing. Additionally we found a better surrogate for straight through estimation of load errors that outperforms the common GShard loss for local balance. Check it out!
Expert load balancing is key to training Kolibri efficiently at scale.
We developed two new methods: Exact Quantile Balancing (EQB) for global balance and Load-Error Injection (LEI) for local balance. @JesseLiJiaze and @PitNeitemeier explain them in a new, visual blog post.
One of my favourite things to see here is how for DeepSeek at short seqlens we read the attention state more than once (dashed read line above solid total) due to cross-layer reuse. Past 16K, sparse attention takes over, so we read it less than once and the lines cross.
How many of your active parameters go to attention, and how many to the experts? In our new blog we derive how modern architectures allocate parameters and FLOPs, and what that means for attention state and long-context FLOPs.
Also: cute kolibris!
Every architectural choice has a cost. Choose where to pay it.
@alessio__serra and @PitNeitemeier derive from first principles how design choices allocate parameters, FLOPs and KV cache, compare open models, and share some of the tooling we used to design Kolibri.
Pit Neitemeier retweeted
Got Kolibri-1 to play Breakout completely on its own, no fine-tuning.
The more I am playing with @Aleph__Alpha’s Kolibri the more I am getting excited about its potential.
Less talk. More Breakout. 🕹️
We optimized the inference for action probabilities: around 25 ms inference per decision.
Four moves. No generated text. One shared game.
Open weights. New possibilities.
Watch 👇 tesseracted.com/kolibri-1-ch…
Cc @IlhanScheer @PitNeitemeier
Dear @Aleph__Alpha team - thank you for making Kolibri-1 open.
We care deeply about sovereign AI, and launching it on German Unity Day makes today feel especially fitting.
As a small gesture of support, we’ve hosted and made Kolibri-1 free for anyone to try for the next few days.
No GPU. No setup. Just try it.
tesseracted.com/kolibri-1-ch…
Open Models move us all forward.
We’re rooting for you. 🇩🇪
Tesseracted Labs GmbH.
tesseracted.com
#alephalpha #openweightmodels
Pit Neitemeier retweeted
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.
I explain it in detail here: aleph-alpha.com/en/blog/scal…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
Pit Neitemeier retweeted
a look at our approach to architecture, load balancing and hyperparameter transfer for Kolibri-1.
i wanted to write about it yesterday, but Pit was faster :)
Love the reactions to our release today! Here are some of my favorite details covering architecture, load balancing and hyperparameter transfer from developing and training Kolibri, our 78B-total, 3.5B-active MoE model:
To train models this sparse, load balancing needs to be dialed in. We built on the work of @Jianlin_S and team and developed Exact Quantile Balancing (EQB). Finding the exact balancing quantile naively requires keeping the tokens × experts matrix of the full global batch.
We also published our work on quantile balancing on arxiv: arxiv.org/abs/2609.28053
To transfer hyperparameters from proxy runs, we use μP across width and loss extrapolation across token horizon. I haven't seen this used this way before, and it's very satisfying to watch the LR basin emerge and the optimum shift as the target horizon grows.
For all the details, see our tech report. We love reports with the development story and ablations, so we included as many as we could while keeping it (slightly) shorter than Order of the Phoenix (just with more plots). Missing a detail? Call us out!
aleph-alpha.com/downloads/te…
Pit Neitemeier retweeted
Hybrid SWA, 3.46B active. Europe on the Pareto frontier indeed. At least they're keeping up with the Chinese literature, kind of (but Sberbank is on MLA + GDN already; Euros have too much GDP to stoop that low). Scale that up.
Pit Neitemeier retweeted
This morning we released Kolibri. Some details on how we built it in the thread.
Amazing work by Max to get this up this fast. He was actually one of the drivers to get us back into model training before he went on to found construct labs. Excited for people to try it!
Big congrats to my friends at Aleph. Glad I was still around when this team formed and got to see how much work went into building it end to end.
kolibri.constructlabs.com
We will offer inference for free for the next ~48 hours.
Hit me up if you are interested in bringing this into production or post-training it.
Pit Neitemeier retweeted
Dear @Aleph__Alpha team - thank you for making Kolibri-1 open.
We care deeply about sovereign AI, and launching it on German Unity Day makes today feel especially fitting.
As a small gesture of support, we’ve hosted and made Kolibri-1 free for anyone to try for the next few days.
No GPU. No setup. Just try it.
tesseracted.com/kolibri-1-ch…
Open Models move us all forward.
We’re rooting for you. 🇩🇪
Tesseracted Labs GmbH.
tesseracted.com
#alephalpha #openweightmodels
