a 1,000 followers , time to take a break

Nanbeige4.3-33B as a live coding agent on an A6000: Given a Dijkstra function with 5 planted bugs + 4 tools, it fixed all 5 and passed 8/8 hidden edge-case tests it never saw. The map replays its code after every edit. vLLM 0.32 · FP8 weights · NVFP4 KV cache on Ampere (113K-token pool) · ~19 tok/s

study report number 3 about model inference engines. - weights load from disk once and stay on the GPU & anything to do with the tokens on the gpu too - the CPU orchestrates using the scheduler for example which is on the RAM - a model judging itself is biased, and a rubric and reference helps alot - quantization still risk regressions thats why you gotta check the probability distributions, token outputs by greedy decoding, - Query not cached: it’s read once, while keys and values are reread by every later token. - MoE big model on GPU but with the speed of a tiny model - RoPE is baked into training, past its training token limit it can produce corrupted outputs. - how do you calculate KV cache size? using vllm & sglang for example. its (gpu size) * 0.90 - model size - 2 gb( for cuda & local variables) = kv cache size . next steps: - better get why Query is so different from KVs, still don't the exact differences - more details about what part of the model lives on the gpu and ram . - conditions where the model is slow or becomes dumber all the sudden - what the schedule exactly does - more details on RoPe

I am currently trying this new study plan when it comes to learning code. I ask ai to generate a simply buggy script and give me a list of bugs to find and fix. I just don’t have the drive to write code but debugging feels more motivating . I think my learning style is to prefer almost instant feedback .

Merged my first ever fix into Emmy, @CloudRiftAI's open-source PyTorch-to-CUDA compiler. When fusion dropped a store, the program still compiled and silently returned unwritten memory. Now it fails loudly and names the kernel. Thanks @ditrifonov for the review! github.com/cloudrift-ai/emmy…

alright study time over for today: - switching between different models on the inference engine is a real consideration - while harness handles deadlocking, resource contention can still happen especially if you have TTS, ASR and LLM on the same GPU - alot of challenges revolve around the GPU having the resources - in ASR to LM to TTS. the asr always gets first priority in proceeding to ensure smooth user exp. - context length or kv cache can be set by engine - kv cache Key and Value numbers are stuff only machines can grasp not us humans. alot of it is identifying bottleneck for resources and speed and random issues popping up like the LLM part freezing up or LLM suddenly getting dumber. being able to identify supporting mlx vs lama.cpp. currently my mental model of the inference engine is this . ASR → scheduler → LLM with its KV cache and cache manager → TTS tmw is probably use vLLM and SGLang repos as point of reference when designing inference engines . - understanding why query isn't stored but kv cache are - what the scheduler does - MoE - RoPe - cases when model gets dumber

trying to understand how inference engines lately. yesterday i was clueless as a headless chicken now i have a visual of it . - scheduler + LLM model + kv cache/managers . - learning models can do parallelism shit weights are just being read & models can do batch decoding outputs too. next steps are to the many issues inference engineers ecnounter, have like kv cache getting filled up. honestly its like grasping how computers work or ram management all over again .

Flint makes a great teacher. Ran it locally on a 24GB M5 Pro, had it write + double-check 1.2k text-to-SQL pairs, then distilled a 331MB model with 0.2s per query (Flint: 7.4s). Model, data + demo: huggingface.co/datasets/alex…

pruned 35B to 11.5B supporting english and mandarin on a M5 Pro with MLX. 64 of 256 experts per layer, 22 GB → 8.9 GB. ~0.3 s to first audio, ~2× real time. huggingface.co/alexwengg/dra…

What’s funny about LLMs and this took me a while to realize is . They operate similar to computers . Kv caches are like ram or memory , starting off as a way to save recompute cost now as storage. Attention layer is like a cpu instruction loader and the forward feedback layer is like the executioner .

saw that Underdog-Bark-3.0 already has a coreml encoder. got curious how the full pipeline would look in coreml and some of my own encoder touches. resulting in 1 h of audio in 13 s (276× RTFx) and LibriSpeech test-other 4.96 % WER @ 140× on M5 Pro . hf repo coming soon.

Phonon 31m now runs on ANE ~20 ms to first audio, ~20x real time on an M5 Pro, 103 voices, and Gradium's text normalizer ported to Swift. code: github.com/FluidInference/Fl… Model: huggingface.co/FluidInferenc… Thanks @GradiumAI and @kyutai_labs for open-sourcing this 🙏

The new 18M / 7M / 2M Pocket TTS models now run on coreml in FluidAudio, open source, with the flow model on ANE. M5 Pro: 18M at 24 rtfx, 2M at 51 rtfx, TTFT ~25 ms after, under 110 MB of memory. macOS 15 / iOS 18. Model: huggingface.co/FluidInferenc… Code: github.com/FluidInference/Fl… Great work @kyutai_labs 🙏

got my first Nvidia PR merged into main 🥳🥳🥳 github.com/ai-dynamo/dynamo/…

got my first SGLang commit merged into main 🪇🪇🪇 github.com/sgl-project/sglan…

All in one month

D3 27B driving a first-person lane runner: every row is one hosted call, the model reads the next three rows as text, each move labelled with its consequence, and picks. ~220 ms per decision, 200+ rows.

d3-edge on a Mac, from a parkour head cam 🏃 Every 2 s of footage, 8 plain yes/no questions up high? long drop below? holding on? on steel? answered in ~110 ms on MLX, labels drawn live on the video with some music. Core ML + MLX ports 🤗 huggingface.co/FluidInferenc…