Building LLMKube: K8s inference across NVIDIA, AMD, and Apple Silicon. Data stays yours. AI/ML Ops Director @ Ashley. Ham Radio: N7CPM
Gig Harbor, WA
Joined January 2018
- Tweets895
- Following616
- Followers344
- Likes2.9K
Pinned Tweet
Badge in hand. Speaking at #KubeCon NA on Tuesday.
Turning Apple Silicon into Production GPU Nodes for LLM Inference on Kubernetes
Tue 11/10, 2:50 PM MT, Grand Ballroom H
CUDA vs Metal benchmarks, one operator across heterogeneous GPUs, and the real tradeoffs. Come say hi!
2
8
555
Apple Containers are really changing how I think about Macs and inference. Having a blast digging into them. How’s everybody else exploring this?
2
132
Chris Maher retweeted
Every one of you can help open source win.
If you care about open source, I need you to know that you can help and that you can make a difference.
A handful of people independently took on open source inference engineering and, through research, hillclimbing, mad scientist experiments, managed to drastically improve KV cache quality, MoE routing, perf, to the degree that several one-man teams are outperforming larger projects like vLLM, llama-cpp, and SGLang (@PeasantSmith @coldniko @Anbeeld @tbraun96 @no_stp_on_snek @Blackwellboy @coffeecup2020 , many others)
Sekai-1 is a 100m param model that scores #2 for its weight class and was trained on a single 16gb 5070 Ti (!) by one person.
YOU can compete. Start learning LLM concepts today. Read The Annotated Transformer over a weekend. Spend some spare time learning about the differences in attention mechanisms. Start hoarding your own training data. Get your feet wet. If you have access to 16gb of VRAM, start experimenting.
All it takes is one of you having a crazy idea that works well at this size and by the grace of God happens to scale. We will collectively BRUTE FORCE lab competition. By decreasing the barrier to entry we can will more labs into existence. There are so many gains on the table still, and believe me, it is possible for you to discover them.
We can't let intelligence be monopolized by these people. We need as many people learning and experimenting as possible because I know there are heroes among them that don't even know it.
#KubeCon NA 2026 is a wrap, and I'm heading home from Salt Lake City grateful and more fired up to build than ever.
The best part, as always, was the people: new connections who are working on the same hard problems, and old friends I got to reunite with.
Huge thanks to everyone who came to my talk on turning Apple Silicon into production GPU nodes for LLM inference on Kubernetes. Your questions were sharp, and they've given me a lot to build on for LLMKube.
If we started a conversation and didn't finish it, my DMs are open.
Thank you @CloudNativeFdn and everyone who made this week happen. See you at the next one!
1
4
110
Doing some hacking with my portable inference setup at the hotel before heading out to hang with the Zarf crew for a bit. This kubecon has been amazing!
1
2
72
Apple shared the next chapter of its open source container framework at KubeCon, including an experimental plugin that spins up Kubernetes clusters right on your Mac.
So I tried it with LLMKube on my M6 MacBook Pro. The operator installed in 26 seconds, the Mac joined its own cluster in 15, and phi-4-mini answered on the GPU.
How the pieces fit:
→ container cluster create gives you a local Kubernetes cluster, no Docker Desktop needed
→ LLMKube's operator runs inside it
→ the Metal agent runs natively on the same Mac and serves the model with Metal
→ pods reach it through a normal Kubernetes Service
The agent ran as is. The only change was one Helm flag: Apple's cluster has no default StorageClass, so the model cache goes per service.
Big thanks to the Apple container team for building this in the open. Excited to see where it goes, and to keep LLMKube working well alongside it.
github.com/defilantech/LLMKu…
4
8
359
Docs are up if you want to try this on your own Mac. The one-Mac quickstart now covers Apple’s container cluster too, including the one flag it needs (no default StorageClass, so --set modelCache.mode=perService): llmkube.com/docs/getting-sta…
1
9
My KubeCon demo yesterday ran from my homelab, 700 miles away.
On stage in Salt Lake City, one Mac joined my Kubernetes cluster in Gig Harbor, WA with a single command. Another served inference with Metal through the cluster, and when I pulled the plug on it mid-traffic, LLMKube's router failed over to a GB10 back in my lab.
That's the whole idea behind LLMKube for me: start with the hardware you already have, and it holds up on a KubeCon stage.
The link home was a pocket-sized @Ubiquiti UniFi Travel Router Long Range running Teleport. It put the stage on my home network and stayed up the whole talk. Thanks to the UniFi crew for making a 700-mile demo feel like it ran in the next room.
#KubeCon
8
209
Every PR one of my models writes for me gets a second pair of eyes before I see it: a read-only Qwen3.8-Flash-Next reviewer, served by LLMKube on an AMD Strix Halo.
Today, 9 minutes on a Go security bump: the code was clean, but it caught a wrong claim in the PR description.
My #kubecon presentation is over, so now it’s time to hit the LLMKube backlog like a beast 😄
1
7
221
Last-minute benchmarks on my M5 Max before my talk tomorrow: @ashxhart's TensorFold ran Qwen3.8-27B at 129 tok/s across 162K tokens of long output, 5x the same model with drafting off (24 tok/s). Not a 256-token demo: 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
Decode tok/s by workload, from TensorFold's own per-request stats:
Agent rewriting a Go file: 231 (up to 359)
K8s YAML: 160
Go package with tests: 144
Code review: 104
Prose essay: 69
That's 3x on prose, 6-7x on code and YAML, and 14x when the output copies its input. Sampling barely moved it: 129 / 134 / 125 overall at temp 0, 0.7 and 1.0. Greedy drafted output was byte-identical to undrafted, so the speedup costs no quality.
I also checked the outputs. 36/36 K8s objects pass strict schema validation, and the Go file edit builds and passes LLMKube's tests. Weak spots: unit tests written in one shot, and invented numbers in the essay. Check those with any model. In opencode it fixed its own jq errors and shipped a working kubectl plugin in under 2 minutes.
Setup: Qwen3.8-27B MLX 4-bit, thinking off, with z-lab's DFlash2 drafter, single stream, 128 GB M5 Max, served by LLMKube's Metal agent. TensorFold 0.3.4.1, because my demo has been pinned since September; 1.2.5 is current. Short templated YAML still hits 227 tok/s; long answers settle lower. Happy to share the script.
Chris Maher retweeted
Be careful what you wish for, America…
“The rest of the world is ready and willing” to build data centers - ARM CEO
My stickers arrived just in time for Kubecon! Excited to share these out for LLMKube and Socair!
1
4
113
Deck's done and I'm pumped! At #KubeCon I'm sharing how I got Kubernetes running LLM inference on Apple Silicon: what worked, what broke (a kernel panic included), and patterns you can take home to your own cluster. See you there!
Last month, Pluto Security reported a Hugging Face campaign that shipped crypto miners as small Qwen2 test checkpoints. No poisoned weights, no pickle trick: just Python that runs under trust_remote_code. The second wave's flagship scanned clean on the hub.
1
1
81
Apache-2.0, every check. Go, air-gap buildable. Try it in your browser, nothing uploaded: socair.ai/try
github.com/defilantech/socai…
1
17
Source, Pluto Security's write-up: pluto.security/blog/hugging-…
1
22