Building LLMKube: K8s inference across NVIDIA, AMD, and Apple Silicon. Data stays yours. AI/ML Ops Director @ Ashley. Ham Radio: N7CPM
Gig Harbor, WA
Joined January 2018
- Tweets895
- Following616
- Followers344
- Likes2.9K
Chris Maher retweeted
Every one of you can help open source win.
If you care about open source, I need you to know that you can help and that you can make a difference.
A handful of people independently took on open source inference engineering and, through research, hillclimbing, mad scientist experiments, managed to drastically improve KV cache quality, MoE routing, perf, to the degree that several one-man teams are outperforming larger projects like vLLM, llama-cpp, and SGLang (@PeasantSmith @coldniko @Anbeeld @tbraun96 @no_stp_on_snek @Blackwellboy @coffeecup2020 , many others)
Sekai-1 is a 100m param model that scores #2 for its weight class and was trained on a single 16gb 5070 Ti (!) by one person.
YOU can compete. Start learning LLM concepts today. Read The Annotated Transformer over a weekend. Spend some spare time learning about the differences in attention mechanisms. Start hoarding your own training data. Get your feet wet. If you have access to 16gb of VRAM, start experimenting.
All it takes is one of you having a crazy idea that works well at this size and by the grace of God happens to scale. We will collectively BRUTE FORCE lab competition. By decreasing the barrier to entry we can will more labs into existence. There are so many gains on the table still, and believe me, it is possible for you to discover them.
We can't let intelligence be monopolized by these people. We need as many people learning and experimenting as possible because I know there are heroes among them that don't even know it.
Chris Maher retweeted
Be careful what you wish for, Americaโฆ
โThe rest of the world is ready and willingโ to build data centers - ARM CEO
Chris Maher retweeted
My boy @ViC305 has got a special gift for you ladies and gentleman running ONE DGX SPARK.
DEEPSEEK 1017 ON ONE DGX SPARK โก๏ธ
I REPEAT DSV4.1 ON 1 DGX SPARK โก๏ธ
SPIN IT UP BABY ๐ฅ
๐จ ๐ ๐๐จ๐ฆ๐ง ๐๐๐๐๐๐ฉ๐๐ ๐ฆ๐ข๐ ๐๐ง๐๐๐ก๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐ ๐ข๐ก ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐! ๐งช๐ฅ
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
๐ง @deepseek_ai ๐๐ฒ๐ฒ๐ฝ๐ฆ๐ฒ๐ฒ๐ธ ๐ฉ๐ฐ.๐ญ ๐๐น๐ฎ๐๐ต ๐ญ๐ฌ๐ญ๐ณ
๐ฆ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ฎ๐ ๐ท๐๐๐ ๐ญ.๐ฒ๐ฎ ๐ฏ๐ฝ๐ ๐ฒ๐
๐ฝ๐ฒ๐ฟ๐๐
๐ป ๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐ฎ๐ป๐๐๐ฒ๐ฟ-๐๐ผ๐ธ๐ฒ๐ป ๐ฎ๐ด๐ฟ๐ฒ๐ฒ๐บ๐ฒ๐ป๐!
โก ๐ฏ๐ฑ.๐ฑ ๐๐ผ๐ธ/๐ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ ๐ผ๐ป ๐๐๐๐ !
๐ ๐ญ,๐ญ๐ด๐ต ๐๐ผ๐ธ/๐ ๐ ๐๐ซ ๐ฃ๐ฅ๐๐๐๐๐!
๐ง ๐ฎ๐ฎ๐ต๐ ๐๐ง๐ซ (๐๐๐ถ๐น๐น ๐๐ผ๐ฟ๐ธ๐ถ๐ป๐ด ๐๐ผ ๐ฝ๐๐๐ต ๐ถ๐ ๐ณ๐๐ฟ๐๐ต๐ฒ๐ฟ!)
๐๐๐ ๐ข๐ก ๐ข๐ก๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐คฏ
And YES, the quantized weights are AVAILABLE NOW! ๐
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ง๐๐ ๐๐๐ฆ๐ง ๐ณ๐ฎ ๐๐ข๐จ๐ฅ๐ฆ ๐๐๐ฉ๐ ๐๐๐๐ก ๐๐ก๐ฆ๐๐ก๐!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
๐ฅ๐๐ฃ๐ฅ๐ข๐๐จ๐๐๐ก๐ ๐ ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐ก๐ง๐๐ญ๐๐ง๐๐ข๐ก ๐ง๐ฅ๐๐๐ก๐๐ก๐ ๐ฃ๐๐ฃ๐๐๐๐ก๐๐ฆ ๐๐ฆ ๐ข๐จ๐ง ๐ข๐ ๐ฅ๐๐๐๐ ๐๐ข๐ฅ ๐ ๐ข๐ฆ๐ง ๐๐ข๐๐๐ ๐๐ ๐๐ก๐ง๐๐จ๐ฆ๐๐๐ฆ๐ง๐ฆ.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! ๐ฅ
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ฅ๐๐ญ๐ฌ!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
๐งช ๐๐๐๐๐๐๐ง๐ฌ ๐ฉ๐ฆ ๐๐๐๐ฃ๐ฆ๐๐๐'๐ฆ ๐ก๐๐ง๐๐ฉ๐ ๐๐ฃ๐ด/๐๐ฃ๐ฐ ๐ฅ๐๐๐๐๐ฆ๐
๐ป ๐๐ข๐๐ โ ๐๐๐บ๐ฎ๐ป๐๐๐ฎ๐น
โ
99.08% Top-1 Agreement
โ
๐ต๐ต.๐ฒ๐ญ% ๐๐ก๐ฆ๐ช๐๐ฅ-๐ข๐ก๐๐ฌ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง!
โ
KL Divergence: 0.0186
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ โ ๐ ๐๐ง๐-๐ฑ๐ฌ๐ฌ
โ
96.04% Top-1 Agreement
โ
97.98% Answer-Only Agreement
โ
KL Divergence: 0.0349
๐ข ๐ ๐๐ง๐ โ ๐๐ฆ๐ ๐ด๐
โ
95.56% Top-1 Agreement
โ
94.19% Answer-Only Agreement
โ
KL Divergence: 0.0477
๐ ๐ข๐ฉ๐๐ฅ๐๐๐ ๐๐๐๐-๐ข๐จ๐ง ๐๐๐๐๐๐๐ง๐ฌ
๐ 94.52% Top-1 Agreement
๐ Mean KL: 0.0982
๐ฏ Median KL: 0.00019
๐ก๐๐๐ฅ-๐๐ข๐ฆ๐ฆ๐๐๐ฆ๐ฆ ๐๐ข๐๐ ๐ง๐ข๐๐๐ก ๐๐๐๐๐๐๐ง๐ฌ ๐๐ง ๐๐จ๐ฆ๐ง ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ! ๐คฏ
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ฌ๐๐ฆ, ๐๐ง ๐๐๐ง๐จ๐๐๐๐ฌ ๐ฅ๐จ๐ก๐ฆ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
๐ข๐ก๐ ๐ญ๐ฎ๐ด๐๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐.
My latest vLLM measurements:
โก Decode: 35.5 tok/s
๐ Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
๐ง๐๐๐ง'๐ฆ ๐ ๐๐๐ ๐๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐! ๐
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ช๐๐ฌ ๐ง๐๐๐ฆ ๐ ๐๐ง๐ง๐๐ฅ๐ฆ.
I have a simple goal.
๐ ๐๐๐ ๐ฃ๐ข๐ช๐๐ฅ๐๐จ๐ ๐๐ ๐๐๐๐๐ฆ๐ฆ๐๐๐๐ ๐ง๐ข ๐ฃ๐๐ข๐ฃ๐๐ ๐ช๐๐ข ๐๐ข๐ก'๐ง ๐๐๐ฉ๐ ๐๐๐๐๐ฆ๐ฆ ๐ง๐ข ๐๐๐ง๐๐๐๐ก๐ง๐๐ฅ๐ฆ.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
๐ง๐๐๐ฆ ๐๐ฆ ๐๐จ๐ฆ๐ง ๐ง๐๐ ๐๐๐๐๐ก๐ก๐๐ก๐.
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ก๐ข๐ช ๐๐ข๐ฅ ๐ง๐๐ ๐ฅ๐๐๐ ๐ง๐๐ฆ๐ง!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
๐ป HumanEval+ / MBPP+
๐ง MMLU-Pro STEM / MATH-500
๐ฌ SciCode / LiveCodeBench
๐ง BFCL / Multi-step tool chains
๐ MGSM
๐ง๐๐ ๐๐ข๐๐ ๐๐ฆ ๐ง๐ข ๐ฃ๐ฅ๐ข๐ฉ๐ ๐ง๐๐๐ง ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐๐ข๐ ๐ฃ๐ฅ๐๐ฆ๐ฆ๐๐ข๐ก ๐๐ข๐๐ฆ๐ก'๐ง ๐๐๐ฉ๐ ๐ง๐ข ๐ ๐๐๐ก ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐๐๐ง๐ฌ ๐๐ข๐ฆ๐ฆ!
Those task-level results are coming next! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐ก๐ ๐ข๐ก๐ ๐๐๐ง๐ง๐๐ ๐ง๐๐๐ฆ๐๐ฅ...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called ๐ฅ๐๐ซ๐๐ฏ.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ ๐๐จ๐๐ ๐ง๐๐๐ก๐ ๐ฌ๐ข๐จ!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
๐ง๐๐ ๐๐๐ฆ๐ง ๐ฃ๐๐ฅ๐ง ๐ข๐ ๐ข๐ฃ๐๐ก ๐๐ ๐๐ฆ ๐ช๐๐๐ง ๐ง๐๐ ๐๐ข๐ ๐ ๐จ๐ก๐๐ง๐ฌ ๐๐๐ก ๐๐จ๐๐๐ ๐ข๐ก ๐ง๐ข๐ฃ ๐ข๐ ๐๐ง. ๐
๐ ๐๐ข๐ช๐ก๐๐ข๐๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐.
๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ.
๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐๐ก๐ฆ๐ช๐๐ฅ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐๐ข๐๐๐ ๐๐ ๐๐ข๐ฅ๐ช๐๐ฅ๐! ๐๐ฅ
Chris Maher retweeted
You download a local LLM. You fire it up in Hermes. No error messages. It says it's running perfectly... but the replies are completely blank, it's operating at a crawl, or the model suddenly feels incredibly dumb.
Youve spent hours googling, tweaking flags, and pulling your hair out thinking you did something wrong.
Spoiler: Youโre not crazy. ๐ง ๐ฅ
Most "model problems" are actually hidden config traps hiding between your request and the response. Stop chasing ghosts and wasting your evenings.
The community has mapped out all 166 of these exact traps so you can diagnose your setup in under ten seconds.
If you're running your setup inside Hermes by @NousResearch , you can check your system instantly with just a single command. ๐ ๏ธ๐
github.com/Blackwellboy/modeโฆ
Chris Maher retweeted
LocalAI folks see this coming a mile away. OpenAI wonโt be the only ones.
Hi,
Tomorrow we are re-opening the Pro $200 subscriptions to new subscribers, but together with it we are also changing how we calculate the usage for it. In effect, if you do the math, it will net out at half the dollar in API spend compared to the old Pro $200 plan.
Now that it's said, let me explain why this is happening and why you will still get more work done than if you were on the Pro $200 subscription one month ago.
(a) We didn't want to compromise in other ways and are committing to not reintroducing the 5h limit, so that you can fully use the weekly usage when you want.
(b) On the subscription, we guarantee that over time you always get more work done and with an increasing level of quality. This means that you will continue to get more value per dollar spent as a result of models getting more efficient and us passing down the improvements in the form of API price reductions.
(c) We don't want to put an incentive on ourselves to artificially inflate the API list prices to make it look like you are getting a lot (and workaround it through discounts, etc). Instead we want to continue to both rapidly reduce prices and increase capabilities of models on the API. This week we introduced GPT-6 Sol and GPT-6 Luna at 50% of their previous price. Over time, we see prices go low enough that it makes sense for most to buy usage as needed without there being a significant gap between what you get in a subscription and what you get in the API for a dollar spent.
(d) Tomorrow, we are adding more things to the subscription that won't draw on the usage, I won't reveal what that is yet.
I wanted to be transparent before all the big announcements tomorrow. Lots of new exciting things are coming to the subscriptions that will make it super compelling, but I wanted to make sure to share this change ahead of time so you can all understand it before we shower you with good news.
Codexingly,
Tibo
Chris Maher retweeted
Claim your compute.
Run AI locally.
Protect your privacy.
Keep your data.
Own the model.
Work offline.
No subscriptions.
No rate limits.
No outage page.
Your thoughts stay yours.
Your IP stays yours.
Your stack stays yours.
Nothing more important than this today.
Chris Maher retweeted
DeepSeek v4.1 Flash โก๏ธ IS OUT!
DeepSeek v4.1 Flash โก๏ธ IS OUT!
DeepSeek v4.1 Flash โก๏ธ IS OUT!
DeepSeek v4.1 Flash โก๏ธ IS OUT!
DeepSeek v4.1 Flash โก๏ธ IS OUT!
DeepSeek v4.1 Flash โก๏ธ IS OUT!
huggingface.co/deepseek-ai/Dโฆ
Chris Maher retweeted
nvidia quietly dropped an official NVFP4 of qwen 3.8 flash and almost nobody noticed, it had 16 downloads when i found it.
this one is designed for the 2x dgx spark owners. 132.7GB of weights, does not fit a single box, but across two sparks it leaves about 40GB more headroom than the official fp8 and keeps more precision than the community 4 bit i have been running.
and the model is already special. i served the community q4 on one dgx spark this morning, 44 tok/s with the mtp drafter, and the depth behavior is the best i have measured in this class, barely fades where others collapse. now the same model, official modelopt quant, on the full cluster. benching it next, numbers incoming.
huggingface.co/nvidia/Qwen3.โฆ
Chris Maher retweeted
Exciting day for NVIDIA and @huggingface.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.
Thank you @ClementDelangue for coming to me.
NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. ๐ค
blogs.nvidia.com/blog/nvidiaโฆ
Chris Maher retweeted
20 hours. free up your devices!
huggingface.co/Qwen/Qwen3.8-โฆ
Chris Maher retweeted
buun-llama now supports DFlash2. This is the fastest implementation on the market. While DFlash2 restricts rollback planes to 7, I have extended this to 12. My adaptive depth also prevents DFlash2 from collapsing on prose, a problem with DFlash. Works with --mmproj-gpu-swap.
๐ค Made with AI
Chris Maher retweeted
Barrier decisively broken, just in time for this work to be worthless and deprecated by Qwen 3.8 lol. Working on DFlash2 now for 3.8, should have something good tomorrow. But if you still want to run 3.6 at 326 tok/s tg on a single 3090, feel free.
๐ค Made with AI
Chris Maher retweeted
Alright guys, wow.
Qwen 3.8 27B is basically Qwen 4.
What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model.
Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result.
On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot.
Imagine what 8 turns would do in a harness?
This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT
This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what @Alibaba_Qwen has accomplished here, and I am incredibly grateful that this model is open source.
The speed will also increase drastically with optimisations.
This is just an incredible day. What a blessing. More to come!
Chris Maher retweeted
Qwen 3.8 27B feels SO GOOOOOD!
Hereโs the thing, itโs thinking a lot, but for the first time it may be that the verbose thinking is less of a mistake than a lot of other local models.
Itโs basically doing a full build, like a model would in a harness, but entirely within its reasoning trajectory. So the final result will be as close as possible to a final output.
Thereโs much less second guessing itself in reasoning than with Qwen 3.6. Itโs very confidently peacing this together so it can have the full picture for the output.
This might be the play for smaller models. If the completeness of thought can overcome the increase in wall time, Iโm pumped, we can always make it run faster!
I think this is going to be really exceptional, I will post the output soon ๐