90% of AI 'breakthroughs' are slop. Let's build the real 10%.
Bucharest, Romania
Joined December 2021
- Tweets1.6K
- Following336
- Followers151
- Likes727
Pinned Tweet
We benchmarked Surogate's serving engine against vLLM and llama.cpp. Same hardware, published methods, losses included.
Best case: 7.0x llama.cpp on a 200 GB MoE across 8 RTX 5090s. Worst case: we lose. Details below.
VRAM just for the weights:
GB ≈ params (in billions) × bits ÷ 8
8B: 16 GB BF16, 8 GB at 8-bit, 4 GB at 4-bit
32B: 64 / 32 / 16
70B: 140 / 70 / 35
Then add KV cache, activations and runtime overhead on top. That's the part that surprises people.
Biggest model you run on your own GPU, and at how many bits?
54 GB of weights. 32 GB cards. No NVLink.
You can still fine-tune it.
Qwen3.6-27B in BF16 doesn't fit on one RTX 5090, and ZeRO-3 won't save you: it all-gathers full weight tensors onto each GPU.
The fix: stream the frozen base from pinned CPU RAM one block at a time, and pass activations between GPUs through host memory. Base weights on each GPU stay at ~2 blocks.
Biggest model you've fine-tuned on consumer cards?
Runs on 4x RTX 5090 with LoRA + FP8 weight streaming. Config and how it works: github.com/invergent-ai/suro…
1 GB.
That's Liquid AI's new LFM3-1.7B at 4-bit: open weights, 256K context. Per Liquid, it beats their previous 2.6B model with 35% fewer parameters.
The interesting fights in open weights are happening under 2B now.
Would you trust a 1.7B model to run an agent on your laptop? For which job?
$1.50 vs $3.24 per 1M tokens.
193 vs 93 tok/s.
0.33 s vs 1.20 s to first token.
GLM 5.3 vs Claude Sonnet 5.5, best provider on each, llm.buzz data through today 09:00 UTC.
None of that is quality. Where does Sonnet still earn 2x the price for you?
Per-provider price, tok/s, first-token time and uptime for every model with 2+ providers, measured: llm.buzz/benchmarks
2,442 texts/s vs 510-562.
EmbeddingGemma-300M on one DGX Spark, 16 clients, one text per request. Surogate (W8) vs vLLM (BF16): 4.3x.
Honest part: send 32 texts per request from 4 clients and it's 814 vs 745 (1.09x).
Does your RAG pipeline batch, or embed one chunk at a time?
Full table and methods (same GPU, same requests, rows where vLLM wins included): github.com/invergent-ai/suro…
Where vLLM wins: H100, Qwen3-8B FP8, 32 users with 2,048-token prompts. TTFT p50 661 ms for us, 325 ms for vLLM.
Gemma 4 31B prefill on DGX Spark, 8 users: 1,106 vs 1,597 tok/s (0.69x).
Benchmarks lie the moment you change the workload. So run your own.
github.com/invergent-ai/suro…
Try it on your hardware and reply with your numbers, wins or losses.
Flavius Burca retweeted
Mistral Large 4 went open weights today. 1.05T total, 52B active, 1M context.
The release from @MistralAI takes text and image input.
If you're serving it, list it on llm.buzz. You set the price, an automated check runs, and it goes live when it passes. We take 0%.
Flavius Burca retweeted
Announcing @flaviusburca, CTO of invergent.ai and creator of @surogate_ai, as our first speaker for the Agentic Engineering "loops & graphs" meetup.
Wed 4 Nov, 18:30, hosted by @stripe at their HQ in Bucharest.
Hot take: agents don't have a decode problem. They have a prefill problem.
Every turn resends the system prompt, the tool schemas and the whole history, then generates a few hundred tokens.
No prefix cache means you pay to re-read all of it, every single turn.
Is prefix caching actually on in your stack?
16 GiB.
That's the KV cache for ONE user at full 128k context on Llama 3.1 8B in bf16. About the size of the weights themselves.
The math: 2 (K and V) x 32 layers x 8 KV heads x 128 dims x 2 bytes = 128 KiB per token. Times 131,072 tokens = 16 GiB.
4 users at full context: 64 GiB of cache for a 16 GB model.
FP8 KV cache halves it. GQA already saved you 4x.
What runs out first on your GPU: weights or cache?
161 tok/s on both.
Gemma 4 26B A4B (open weights): $0.1375/M blended, 0.28 s to first token.
GPT-5 Nano (closed): $0.11/M blended, 0.69 s to first token.
Same output speed. Nano is 20% cheaper. Gemma answers ~2.5x sooner.
Which one runs your agent's cheap tier, and why?
Best provider per figure, 24h medians through Oct 29 12:00 UTC (test requests). Compare every provider here: llm.buzz/benchmarks