Philippe Anel (@zexigh) profile picture

Building high-perf inference engines in Rust • MoE, AVX-512, memory-bound decode • Low-level optim & AI-assisted kernels • Curious about your benchmarks

France
Joined May 2012
Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)Photo by Philippe Anel (@zexigh)
🔥 Llama.cpp just released version 1.0. It's the reference for running local, open-source AI. But can you build something faster, in Rust, without assembly (or almost)? 🎬 In this video, I compare Herbert, my inference engine, with Llama.cpp, on a CPU, no graphics card. ⚡ First result: Herbert read the question 2 times faster and wrote the answer 1.5 times faster. Too good to be true? 🔍 Digging in, I realized I wasn't comparing quite the same thing. The file you usually download for Llama.cpp, Q4_K_M, says 4 bits, but part of the weights is kept in 8 bits. Heavier, and closer to the original model. 🇨🇦 So I measured fidelity. Asked "what is the largest city in Canada", the 16-bit model answers Toronto… and every 4-bit version answers Montreal, Q4_K_M included. ⚖️ Then I redid the comparison with the same format, made from the same weights. Herbert still reads the question about 2 times faster on one CPU and 1.5 times on the other. For writing the answer, its lead shrinks to 8 to 13%, but it stays ahead. 🚀 And after months of work, Herbert is finally going public, with all the benchmarks, scripts and raw data. 🔊 The English voice is an AI clone of my own voice: I recorded this video in French. The full video (in French and English with AI dubbing) is linked 👇
9
15
62
2,075
A word on the English version: I record every video in French, then the English track is generated from that recording with an AI clone of my own voice. Recording each episode twice just isn't realistic for me right now, and I'd rather spend that time measuring things. Same script, same numbers, same visuals.
2
1
50
🔧 Want to know more about Herbert, my inference engine? The "One level down" series: 0️⃣0️⃣4️⃣ The same compiler on Apple's chips: philippe-anel.fr/s/n39ry4/ 0️⃣0️⃣3️⃣ What CUDA has that Vulkan lacks: philippe-anel.fr/s/kq57rv/ 0️⃣0️⃣2️⃣.1️⃣ Follow-up, the Arc in f16: philippe-anel.fr/s/yzg4y3/ 0️⃣0️⃣2️⃣ The same compiler on a GPU: philippe-anel.fr/s/mnumeg/ 0️⃣0️⃣1️⃣ A compiler for an LLM engine: philippe-anel.fr/s/88fvcp/ 0️⃣0️⃣0️⃣ An inference engine that writes its own code: philippe-anel.fr/s/2u74ut/
1
77
🧮 Can a CPU keep up with a GPU for running an LLM? On unified-memory machines, I measured. The answer depends on the machine, and on what you ask of it. Herbert, my inference engine, runs on the CPU (#001) and on the GPU (Metal, Vulkan, CUDA). Same model, same memory, same machine. I say it in almost every post: what limits the speed of an answer is not compute, it's how fast memory can be read, the bandwidth. Hence my bet, written on October 29 before measuring. Reading the prompt (prefill): the CPU far behind, not enough compute units. Writing the answer (decode): neck and neck, the same bandwidth bounds both. One exception, the Mac Studio, where the CPU would get crushed even when writing: its memory controller serves the GPU first, the CPU never gets the 819 GB/s. A disappointment, but it makes sense. Three machines: a MacBook Air M1 (8 GB), a Mac Studio M3 Ultra (96 GB), an Intel Core Ultra 7 258V laptop with its Arc 140V GPU (32 GB). Qwen3.5-2B and Gemma 4 E4B in Q4, 127 tokens generated, medians. 🔍 Decode, GPU over the CPU of the same machine: ▪️ MacBook Air M1: ×1.19 on Qwen (51.9 vs 43.6 tokens/s), ×1.10 on Gemma. Both pull 51 to 61 GB/s from a memory rated at 68. Neck and neck, as predicted. ▪️ Intel laptop: ×1.41 on Qwen (77.6 vs 55.1), ×1.22 on Gemma. In between. ▪️ Mac Studio: ×3.0 on Qwen (405 vs 134), ×1.9 on Gemma. The CPU pulls 160 GB/s, the GPU 480. Same memory, not the same bandwidth. ▪️ Prefill: the GPU leads everywhere: ×1.2 on the M1, ×3 on the Studio, ×8 on the Intel laptop, which sorely lacks AVX-512. ▪️ At 3,000 tokens of context, on the M1, the CPU loses 11%, the GPU 4%: the gap goes from ×1.19 to ×1.29. Attention costs the CPU more. My bet holds. On a unified-memory laptop, the CPU is enough to write the answer: the GPU mostly buys you a fast read of the prompt. On a Mac Studio, no: the bandwidth goes to the GPU. The engines measured here are Herbert's Q4 versions, the ones in my application, not the bf16 versions published with #001 and #004. Unfortunately I can't publish them yet. Testing is going well, they will come; I can't wait to show you. Along the way, Windows was throttling the Intel laptop's CPU when launched in the background: 9 tokens/s instead of 104. Fixed; a story of its own, I'll come back to it. All this to say: an inference engine is hard. Not out of reach: if I can do it, it can be done. But with the diversity of machines, you always hit a performance or stability problem somewhere. So even when my engines beat the big ones, this is no reason to question the work of the people who build them: the difficulty is real.
6
1
1
15
957
👉 The four articles of the series, on my site: #001, the CPU compiler: philippe-anel.fr/s/wxht9w/ #002, the same compiler on a GPU, with Vulkan: philippe-anel.fr/s/8baq2b/ #003, what CUDA has that Vulkan doesn't: philippe-anel.fr/s/c4t467/ #004, the same compiler on Apple silicon, with Metal: philippe-anel.fr/s/tkmva5/ Every article has its podcast at the top of the page. Sign up on the site and you get my Friday newsletter on the week's news. If you want a version that is simpler to follow, I make videos on my YouTube channel: youtube.com/@la_tete_dans_le…
4
120
⚡ On its own Mac, Herbert does better than Apple with MLX. Up to 31% faster, without a single line of MLX. Herbert is my inference engine. I measured it on my Mac Studio M3 Ultra, with six Qwen3 models in bf16, from 0.8B up to 35B, against mlx-lm, Apple's engine built on MLX. At prefill, ahead on every prompt up to 8,192 tokens. At decode, ahead on all six models. I've been working on this code for months. Dozens of versions and hundreds of hours, in Rust, in C++ and even in Swift. Some also leaned on MLX's own routines. Then I got fed up: I started over from scratch, with nothing from MLX. In Rust. I tested and rewrote it on three machines with almost nothing in common: a 96 GB Mac Studio, my old 8 GB MacBook Air M1, a fanless MacBook Air M5, with one single Herbert, which measures the hardware at load time and adapts to it. And I took two big slaps in the face (several, honestly): one night, the engine picked faster kernels... that were wrong. But above all, long prompts: at 81,920 tokens, I was 4 times behind mlx-lm 😱 I had left the attention in scalar arithmetic. Again. I hate myself when I do that 😡 Once that mistake was fixed, Herbert came back level, but the lead shrinks as the prompt grows: I explain it in the article if you're interested. 📦 The engine and the article's measurements are online, MIT-licensed. 👉 The whole story on philippe-anel.fr, with the podcast for those who want it: philippe-anel.fr/en/blog/202… 🧭 What's next? I don't know yet what I'll talk about next week. The quantized versions already run in production, and this series forces me to revisit them. But my priority now is the ultra-secure, local harness that goes with Herbert, in production at our place and with a few clients and friends. It also works with other engines, like vLLM, llama.cpp, mlx-lm, etc. We have some very serious things to show you, and we're proud of them. Consider yourselves teased 😏 What do you run locally on your Mac? One level down.
4
3
9
643
📍 A milestone. Four articles in one month: #001 a compiler for an LLM engine, on CPU #002 the same compiler on a GPU, with Vulkan #003 what CUDA has that Vulkan lacks #004 the same compiler on Apple's chips, with Metal The same Herbert on CPU, Vulkan, CUDA and Metal, level with the reference engines and often ahead.
1
3
110
🎬 The video in this post goes through all six models, from 35B down to 0.8B: prefill and decode, against mlx-lm and llama.cpp. 🤖 A button at the top of the article copies the text so you can discuss it with your AI. Six AIs already read it this morning: they all ended on the same question. I'll let you guess which one .... 🎥 These articles are not easy reading for everyone. On YouTube, Instagram and Facebook, I make short videos that are far more accessible. Have a look and tell me what you think. youtube.com/@la_tete_dans_le…
4
228
⏱️ Herbert writes its own machine code every time it starts. It should be the slowest to start. It's the first one ready to answer. Here's what the first launch costs too. The objection I heard most after my Vulkan article: "All your timings are measured warm." Fair. So I measured startup itself, from launching the server to the first token. First launch on the machine, driver caches wiped: cold. Every launch after that: warm. RTX 4080. Weights already in memory. Five runs, medians. Qwen3.5 0.8B, then 4B, in bf16. Four engines: Herbert, which writes its kernels at startup; llama.cpp, which ships them precompiled, on Vulkan and on CUDA; vLLM, which compiles at startup too, with PyTorch. ▪️ Herbert: 2.1 s cold, 0.6 s warm (4B: 2.6 and 1.0) ▪️ llama.cpp Vulkan: 3.1 and 0.8 (4B: 5.0 and 1.5) ▪️ llama.cpp CUDA: 0.8 and 0.8 (4B: 1.2 and 1.2) ▪️ vLLM: 89.5 and 37.8 (4B: 94.5 and 41.5) 🔍 What it says: ▪️ Herbert generates 38 modules of machine code in 2 to 3 milliseconds. Writing the code is free. What costs is the driver translating it for the card: 1.5 s the first time, 4 ms every time after, from its cache. Then the weights: 0.3 s for the 0.8B, 0.7 s for the 4B. ▪️ Warm, Herbert is ready before every other engine, at 0.6 s. ▪️ Cold, Herbert pays the driver once, 1.5 s, and still beats llama.cpp Vulkan, which pays the same driver. The one engine ahead of it cold is llama.cpp CUDA, whose code is compiled for the card before you download it. Precompiled for one vendor against generated on the fly for any card: 1.3 seconds, once. ▪️ vLLM: a minute and a half cold, half of it compiling. Warm, still 40 seconds, and its logs say that's no longer compilation. Python, processes, server. Default settings; its fast-start options are not measured. Two weeks ago, Herbert started after llama.cpp even warm: 4.4 s on the 4B. Not the code. The weights, copied one value at a time on a single core. I rewrote the loader that day: files read in place, the copy spread across all cores. 0.7 s instead of 4, weights on the card identical, bit for bit. That's the thing with an engine you wrote yourself: you find the bottleneck in the morning and it's gone by evening. Not measured: the disk (weights were in memory), other cards, and Herbert's CUDA engine from last Sunday. Coming. Every article has its podcast at the top of the page. Sign up on the site and you get my Friday newsletter on the week's news. 👉 The two articles these measurements come from: Vulkan, October 18: philippe-anel.fr/en/blog/202… CUDA, last Sunday: philippe-anel.fr/en/blog/202… One level down.
4
2
12
472
⚡ On an NVIDIA card, without a single NVIDIA library, Herbert writes the answer faster than TensorRT-LLM, NVIDIA's own engine. RTX 4080, three Qwen3.5 models in bf16, one request at a time. By 2 to 7%. Only slightly, but ahead, even after a 40,000-token prompt. I did not see that coming. Before measuring, a tie was the best I hoped for: "it's NVIDIA's code... I think they know how to use the power of their hardware better than I do." 🧠 Herbert, my inference engine, writes its kernels in PTX at startup, for the model and for the card in front of it. No CUDA C, no nvcc, no cuBLAS: just the driver. 🔍 Last Sunday, I wrote that Vulkan was missing two instructions. Reading the machine code its driver produces, it is missing only one: cp.async, the copy that loads the next tile while the current one is computed. Removed from the CUDA engine, it costs 5 to 14% of prefill. Nothing at decode. ⚖️ At prefill, it's a tie within a few percent, both ways. On the 2B, Herbert is 7 to 8% behind: it lacked a measured setting for that model. The article says so, with the command that fixes it. 📉 One of my predictions was right: "I'm going to get wrecked" on very long prompts. At 60,000 tokens, Herbert wrote the answer at half the speed of vLLM. One kernel was at fault, reading the same data several times. I hated myself for letting that slip through. One night of fixing: from 133 to 278 tokens per second. Keeping things in proportion: these engines are built to serve many people at once, and this bench measures one request. No data center card either: I plan to measure those on RunPod. But for me, this is an important moment. I really did not think I would get such a result so soon. 📦 The engine, the scripts, my dated predictions and the raw data are online, MIT-licensed. If a number looks wrong to you, you have what you need to check it. 👉 The full article, on my site: philippe-anel.fr/en/blog/202… 🎧 The podcast is at the top of the page: 8 minutes, a synthetic voice reading the text of my French recording, and it says so in its first sentence. With each article I try to make the podcast easier to digest than the numbers, and to make it better. If you listen to it, tell me what works and what doesn't: the tone, the length, the level of detail. One level down.
3
5
13
624
🔍 On Tuesday, DeepSeek open-sourced the inference engine that serves its models in production: DeepInfer, under Apache 2.0. I read it with one question: what does it have in common with Herbert, the engine I'm working on? At first sight, almost nothing: for now, we don't serve anywhere near the same number of requests. 1️⃣ The code. DeepInfer is written in Python on a C++ core. DeepSeek says it was originally built on vLLM. Part of the kernels are written in TileLang, DeepSeek's own language, and compiled at run time, the way vLLM does it with Triton. The rest comes from their libraries (FlashMLA, DeepGEMM, DeepEP, etc.). Herbert is written in Rust from scratch and generates the machine code, the SPIR-V or the PTX directly (more on that in the next article, on CUDA). Different in form, not in substance. 2️⃣ The scale. DeepInfer serves one model, DeepSeek-V4.1-Flash, on datacenter Nvidia GPUs (SM100) or Huawei Ascend 950 NPUs. According to DeepSeek, to be efficient they need 64 GPUs to read prompts and 64 more to generate answers, on separate machines. They don't handle one request per GPU: each one computes attention for about sixty requests and holds only a few of the model's experts. The experts talk over NVLink, inside a single supernode; between machines, the KV cache travels compressed (FP4, FP8). Herbert runs on consumer CPUs and GPUs, one request at a time. With my engine, as it stands, one GPU handles the whole request: no splitting across several GPUs, because they would have to talk over the PCI Express bus or, worse, over Ethernet. I'll come back to that in my article on Tenstorrent cards. I have code that spreads the computation across several machines. I may write about it, but in my tests there is so much data to move that the gain is negligible as long as the links between machines stay (relatively) slow. 3️⃣ The result. To make sure optimizations don't change the model's answers, DeepInfer offers "batch-invariant" inference: the same answer whatever other requests are served at the same time. For Herbert, I check at every campaign that it returns the same tokens as the Hugging Face reference. It's a bit different for the quantized version, as we'll see. This isn't about which one is faster. I have neither SM100 nor Ascend. This is a reading of the code and the docs, nothing more. The one thing to take away: whether you serve millions of requests or a single one, you end up fitting the code to the machine and checking that the result doesn't move. 📦 The repo: deepseek-ai/DeepInfer, on GitHub. My reading covers the October 21 version. 👉 Herbert against vLLM on GPU, in my latest article: philippe-anel.fr/en/blog/202…
2
3
11
488
My posts are pretty technical these days, I know. That's why every article comes with a podcast at the top of the page: the same story, told simply. Give them a listen and tell me what you think. And if you sign up on the site, you get my Friday newsletter, where I comment on the week's AI news. 👉 philippe-anel.fr/en/newslett…
1
2
107
Et c'est bien entendu disponible en Français 😊
1
1
60
⚡ Nearly three times faster than vLLM to the first token. Nearly twice as fast as llama.cpp on CUDA. With an engine that doesn't use CUDA. RTX 4080, small model, one request at a time: measured, published, reproducible at home. And I'm not hiding the cases where it loses. 🧠 Herbert, my inference engine, writes its GPU kernels at startup, for the exact shape of the model and for the card in front of it. SPIR-V for Vulkan, with no precompiled shader and no vendor library. 📊 The numbers. Qwen3.5-0.8B, short prompt: first token in 6.5 ms for Herbert, 12 for llama.cpp CUDA, 19 for vLLM. vLLM is built to serve many requests at once: at 4,000 tokens the gap with it disappears, and on the 4B vLLM answers 3.7% earlier. ⚖️ The other cases. At prefill on the 4B in f16: a tie with llama.cpp CUDA at its best setting. On the Arc 140V of an Intel laptop, in f16: Herbert is slower at prefill, down to ×0.57. At decode, it stays ahead of llama.cpp on every card. Why does it win? I turned its settings off one by one. On medium and long prompts, it's mostly cutting the prompt into large blocks. On a very short prompt, these settings explain only a small part of the lead: the rest comes from the kernels themselves. In the article: five GPUs from three vendors; two weight formats, always compared at equal format; my predictions, written before measuring; what gives Herbert its lead, measured by turning its settings off one by one. 📦 The engine in bf16 and in f16, the scripts and the raw data are online, in three MIT-licensed repos. If a number looks wrong to you, you have what you need to check it. 👉 The full article, on my site: philippe-anel.fr/en/blog/202… 🎧 The podcast is at the top of the page: 18 minutes, a synthetic voice made from my French recording, and it says so in its first sentence. One level down.
3
3
1
14
452