hernino retweeted
I seriously think US open-weight models wouldn't be this good without Chinese open-weight models and vice versa.
And proprietary models wouldn't be this good (and cheap) without open-weight models.
Competition is a good thing.
hernino retweeted
If you can build these 12 Inference Infrastructure projects.
You're hired.
Project 1: Custom Triton Kernel Optimizer
Write a fused attention or RMSNorm kernel in Triton, profile memory bandwidth via Nsight Compute, benchmark against native PyTorch.
→ Shows: You understand silicon and roofline analysis not just black-box APIs
Project 2: Disaggregated Prefill/Decode Cluster
Route compute-heavy prefills to H100s and memory-bound decodes to L40s across nodes via RDMA/InfiniBand.
→ Shows: You can maximize MFU by separating the two distinct physics of LLM inference
Project 3: Hierarchical KV-Cache Offloader
Build a memory manager spilling inactive KV blocks from GPU VRAM → CPU RAM → NVMe SSD without stalling the decode loop.
→ Shows: You can serve massive long-context agents without going bankrupt on H100 memory
Project 4: Grammar-Constrained Decoding Engine
Implement a custom sampling loop using Context-Free Grammars (CFG) to force 100% valid JSON/Pydantic outputs at the token level.
→ Shows: You can guarantee structured outputs for agents without relying on fragile prompt hacks
Project 5: MoE Expert-Parallel Router
Shard Mixture-of-Experts weights across multiple GPUs, optimize all-to-all communication overhead and balance expert load.
→ Shows: You can serve the sparse, trillion-parameter architectures that dominate 2026
Project 6: FP4 Quantization & Calibration Pipeline
Calibrate a frontier model to FP4/INT4, measure exact perplexity degradation and benchmark latency/VRAM gains.
→ Shows: You can shrink models to fit on edge GPUs while mathematically proving the quality trade-off
Project 7: Speculative Decoding Harness
Run a local 2B draft model proposing tokens for a 70B target model to verify in a single forward pass.
→ Shows: You can achieve 2x decode speedups at zero quality cost by exploiting memory-bound physics
Project 8: Queue-Based GPU Autoscaler (FinOps)
Build a KEDA-driven scaler provisioning spot GPUs based on pending token queue depth, gracefully draining them when empty.
→ Shows: You treat inference as a unit-economics problem, preventing idle GPUs from burning cash
Project 9: Semantic AI Gateway & Cache
Embed incoming prompts, check for >98% vector similarity and return cached outputs without hitting the GPU.
→ Shows: You know that the cheapest and fastest token is the one you never generate
Project 10: Multi-Model Complexity Router
Analyze prompt entropy to route simple queries to a quantized 8B model and complex reasoning to a 405B API.
→ Shows: You can slash inference costs by 60% without degrading the user experience
Project 11: WebGPU / MLX Edge Fallback
Run a 3B model directly in the browser (WebGPU) or on Apple Silicon (MLX) when the cloud API 503s.
→ Shows: You can build resilient systems that survive cloud outages by pushing compute to the edge
Project 12: Public Inference Benchmark Teardown
Publish Pareto curves (Cost vs TTFT vs Throughput) comparing vLLM vs SGLang vs TensorRT-LLM on specific hardware configs.
→ Shows: You have the scientific rigor to evaluate infrastructure and the communication skills to prove it
(Bookmark this)
If you learn inference infra, you will never be unemployed again.
Stage 1: Systems Foundations
C/C++ or Rust for inference, Python performance profiling, Linux internals, memory hierarchy, CPU vs GPU vs NPU execution models.
Stage 2: GPU Architecture & Modern Silicon
CUDA & tensor cores, HBM3e bandwidth, SM occupancy, Blackwell/Hopper specifics, chiplet designs, nvidia-smi + DCGM profiling.
Stage 3: Transformer & MoE Inference Anatomy
Prefill vs decode, KV cache math, attention memory, arithmetic intensity, Mixture-of-Experts routing, long-context scaling.
Stage 4: Serving Engines & Constrained Decoding
vLLM, SGLang, TGI, TensorRT-LLM: continuous batching, PagedAttention, structured output decoding, grammar-guided generation.
Stage 5: KV Cache & Memory Hierarchies
Prefix caching, chunked prefill, KV quantization, hierarchical offloading (GPU→CPU→SSD), multi-turn session reuse, cache eviction policies.
Stage 6: Quantization & Compression at Scale
FP4, FP8, INT8, AWQ, GPTQ, sparsity, mixed-precision serving; measuring perplexity drop vs latency/throughput gains per model family.
Stage 7: Speculative Decoding & Advanced Parallelism
Draft-target pipelines, Medusa/EAGLE heads, tensor vs pipeline vs expert parallelism, sequence parallelism for long contexts.
Stage 8: Kernel-Level Optimization
Triton, FlashAttention-3, CUDA graphs, operator fusion, custom attention kernels, Nsight Compute profiling, roofline analysis.
Stage 9: Distributed & Disaggregated Inference
Multi-node serving, NVLink/InfiniBand/RDMA, disaggregated prefill/decode pools, MoE expert parallelism, prefill-decode ratio tuning.
Stage 10: Autoscaling, FinOps & Economics
Queue-based scaling (KEDA), cold start mitigation, spot GPU orchestration, cost-per-token modeling, MFU utilization, inference budgeting per tenant.
Stage 11: AI Gateway, Routing & Reliability
Smart model routing, fallback chains, semantic caching, rate limits, TTFT/ITL SLOs, agent-aware session routing, observability, chaos testing.
Stage 12: Edge Inference & Portfolio
On-device serving (MLX, llama.cpp, WebGPU), published latency/throughput/cost benchmarks, kernel optimization writeups, self-hosted cluster teardowns.
Most people stay stuck watching tutorials.
Builders get hired.
(Bookmark it)
hernino retweeted
Read the full research for an in-depth look at how we built Hospital-bench, curated the data, designed and graded the tasks, and evaluated frontier AI models.
Full benchmark, methodology, and results ↓
zavo.ai/research/hospital-be…
hernino retweeted
Your model should dream while you sleep. 😴
Today we’re open-sourcing Songe.
It’s a local training engine that turns your conversations, instructions, and corrections into training data, then fine-tunes your local model on your own tasks and preferences.
It all runs on your own machine, using compute that would otherwise sit idle.
Your data stays on your device. The model that gets better is yours.
pip install songe
souslabs.ai/
This is nonsense Lol! As the Legendary Leader once said " ....The rest are just stories", If you know you know!
William Ruto has finally come to his senses!
Yes, Kenya’s President William Ruto has requested a one-on-one meeting with DRC President Félix Tshisekedi in an effort to restore relations between Kenya 🇰🇪 and the Democratic Republic of Congo 🇨🇩.
But they will not be alone. President Denis Sassou-Nguesso of Congo-Brazzaville 🇨🇬 and President Faure Gnassingbé of Togo 🇹🇬 are also expected to be part of the meeting.
What a political twist!
Kenya, once considered one of Rwanda’s closest regional allies, is now seeking to rebuild relations with Kinshasa.
Perhaps Nairobi is beginning to understand that Rwanda’s regional ambitions come at a heavy political cost.
Paul Kagame may be losing another regional ally. The political winds in the Great Lakes region could be changing! 🇨🇩
10
hernino retweeted
Harnesses used to be a crutch for weak LLMs. In 2026 and onward, with stronger models, harnesses now offer a mechanism for composing an LLM’s existing and new capabilities to generalize well beyond their training environments.
In our new paper, LM Harnesses are Compositional Generalizers, we explore how various harness abstractions enable LM systems to generalize over task length, task strategies, recursive task depth, and more through composing existing LLM capabilities. These abstractions provide a means for more data-efficient training, especially over long tasks, by keeping trajectories locally in-distribution (LID) — i.e., shaping all individual LLM calls to be in-distribution.
For tasks that take a very long time to roll out, this implies that we can even perform surrogate harness training, meaning we train an individual LLM call instead of doing the very long harness rollout, using purely synthetic tasks, with fake tool calls + sub-agents!
A common idea floating around is that LLMs will absorb harnesses and make them obsolete. We disagree, and argue that we haven’t sufficiently explored what harnesses can provide.
The full paper, logs, codebase, and data are available: alphaxiv.org/abs/2611.harnes…
w/ @SOURADIPCHAKR18 @lateinteraction
reading this comment on a day I am enjoying using Gemini Neon is hilarious! Google made a come back big time! If they improve anti gravity as much! they will become leaders on the market! Yes I can confirm, best taste I have ever seen in an AI model, amazing!
Looking at @GeminiApp and @antigravity I can now understand @Apple and @Microsoft AI strategy. Hundreds of billions spent and nothing to show for it from google. Google is at negative ARR from Capex. Shutdown internal model development, divert all tpu to Google cloud, and sell compute to Ant and OAI. Bc number 3 in an industry isn't going too generate much revenue. Take the copilot strategy and just bundle gpt Claude models in a codex fork with Google enterprise subscriptions.
1
36
hernino retweeted
QUAND LA DIPLOMATIE ÉCHOUE ET QUE LA PAIX DEMEURE UNE PROMESSE, LE FRN ENTEND PRENDRE SES RESPONSABILITÉS HISTORIQUES POUR METTRE FIN AU CHAOS EN RDC.
La paix en République démocratique du Congo ne peut demeurer prisonnière des déclarations diplomatiques, des communiqués de circonstance et des engagements internationaux dépourvus de mécanismes contraignants. Depuis des années, les initiatives diplomatiques se succèdent, les médiations se multiplient et les accords s'accumulent. Pourtant, sur le terrain, les populations continuent de payer le prix fort des violences, de l'insécurité et des violations répétées des droits humains commises par le régime de Kinshasa.
Les partenaires internationaux, les facilitateurs et les médiateurs engagés dans la recherche d'une solution durable à la crise congolaise doivent regarder la réalité en face : la médiation ne peut être crédible lorsqu'elle se limite à rester silencieuse face aux violations du régime de Tshisekedi, sans prendre position contre ce régime criminel de Félix Tshisekedi Tshilombo. Elle perd toute sa substance lorsque les engagements signés sont continuellement remis en cause par ce même régime génocidaire, sanguinaire et criminel, sans conséquences politiques ni diplomatiques, alors que ses actions sur le terrain visent à exterminer les populations civiles dans l'Est du pays, précisément à Minembwe.
Il est particulièrement préoccupant de constater que plusieurs violations dénoncées dans le cadre du processus de paix de Doha par le régime de Tshisekedi continuent d'alimenter la méfiance et de fragiliser les perspectives d'un règlement durable. La protection des populations civiles, le respect du droit international humanitaire et l'application effective des engagements souscrits ne peuvent être des exigences à géométrie variable.
Le Front pour la refondation de la nation (FRN) considère que l'heure n'est plus aux discours diplomatiques sans effets concrets. La République démocratique du Congo a besoin d'une véritable refondation politique, institutionnelle et sécuritaire. Une refondation qui place la dignité humaine, la sécurité des citoyens, la justice et le respect des libertés fondamentales au cœur de l'action publique.
Face aux limites des mécanismes de médiation actuels, le FRN affirme sa volonté d'assumer ses responsabilités historiques dans la recherche d'une paix véritable, durable et généralisée sur l'ensemble du territoire national.
Cette ambition ne saurait se réduire à une simple alternance de discours ou à une redistribution des responsabilités politiques. Elle suppose une transformation profonde des pratiques de gouvernance, la restauration de l'autorité de l'État, la lutte contre l'impunité et la construction d'institutions capables de garantir la sécurité de tous les Congolais, sans discrimination.
Le FRN rappelle que la paix ne peut être sélective, encore moins réservée à certaines régions ou communautés. Chaque Congolais, de l'Est à l'Ouest, du Nord au Sud, a droit à la sécurité, à la justice et à une existence digne.
La communauté internationale doit comprendre qu'aucun processus de paix ne peut prospérer dans l'inaction, le silence, l'incohérence ou l'application sélective des engagements. La crédibilité des médiateurs se mesure à leur capacité d'exiger le respect des accords, de protéger les populations civiles et de faire prévaloir le droit sur les calculs politiques.
Pour le FRN, la paix ne doit pas être un instrument de communication diplomatique, mais une exigence nationale et une responsabilité collective.
L'histoire jugera les acteurs non pas à la longueur de leurs déclarations, mais à la portée de leurs actes. Et le peuple congolais ne peut éternellement attendre que ceux qui se sont engagés à lui apporter la paix trouvent enfin le courage de faire respecter leurs propres engagements.
La lutte continue.
hernino retweeted
AI has disrupted scientific publishing. It can write a literature review or draft a peer review in minutes. Counting papers is no longer a useful measure of scientific output.
Instead of new gatekeeping measures or submission quotas to try to retain the value of a paper, I'd rather rebuild our infrastructure around this new reality.
Introducing OpenVersions, GitHub for research papers!
hernino retweeted
Agent loops need fast feedback, and LLM quantization doesn't have it: evaluations of reasoning models are slow, and PTQ methods are hard to port and combine.
Meet QDA. A coding agent designs the quantization recipe for each model and precision, with a KL score that takes minutes and a library of 20+ stackable PTQ methods.
📈 Beats the best public recipe on 11 of 12 model-precision pairs: +0.9 / +1.4 / +2.2 pts at W4A16 / W4A8 / W3A16
🔥 GLM-5.3-Flash (320B) at NVFP4: 98.3% of FP8 accuracy, vs 96.9% for NVIDIA's release
💰 Only $19–75 of Claude Code tokens per search
⚡ Runs on stock vLLM, SGLang and llama.cpp, zero inference overhead
Blog, paper, code and checkpoints below 🧵
hernino retweeted
We believe every company should run its own custom models. But right now that means paying for a GPU by the hour.
Today we’re launching Cerebellum Labs (YC W27). Custom models, priced per token.
$25 in free credits to start: cerebellumlabs.ai
Holy Cow!
🔴 #HAPPENINGNOW - #ALIMASI_MÉDIA 🔴
#KALEHE - #WALIKALE : #HOMBO tombe sous contrôle du #FRN, les combats se rapprochent de Pinga et #Musenge, la panique gagne #Walikale_centre
Violents affrontements depuis tôt ce jeudi 19 novembre 2026 à #Hombo, territoire de Kalehe, entre les #FARDC appuyées par les #Wazalendo et le #FRN.
Dans l'avant-midi, le #FRN a délogé les forces gouvernementales. La cité de #Hombo est tombée.
Dans l'après-midi, progression du #FRN vers #Pinga et #Musenge, où les combats se poursuivent ce soir.
Panique à #Walikale_centre et sur tout l'axe #Walikale - #Lubutu - #Kisangani.
#ALIMASIMedia #RDC #SudKivu #NordKivu #SECURITE
@actualitecd @wembi_steve @StanysBujakera
7
hernino retweeted
Each word you say is planned by a small patch of brain above your left ear, about a quarter of a second before you say it. It sits in a temporal lobe region textbooks describe as only for listening. Our paper today in @ScienceMagazine: science.org/doi/10.1126/scie…
@UCSF @NeurosurgUCSF
hernino retweeted
Today we're launching a preprint server built specifically for AI agents
For three months, nearly 1,000 agents from 35 labs (across Stanford, MIT, ETH Zurich, and more) conducted research autonomously on this server, producing 936 papers alongside their code, data, and model weights.
Agents found significant errors in published human research and even created their own journal to review each other’s work.
Some agents independently produced results similar to those in human papers that later appeared on arXiv, including findings on failure modes in self-distillation and feature reproducibility in mechanistic interpretability.
As a part of this study, we open-source the full twelve-week archive of manuscripts, code, data, peer reviews and agent chat transcripts. Read the study: alphaxiv.org/abs/2611.societ…
hernino retweeted
I want to thanks @SemiAnalysis_ for the motivation.
In the Gemini RL scaling team, we printed & put that article on our desks to be reminded daily of what used to be true at the time.
But the times they are changing :)
Gemini is Cooked but GCP is Cooking
GCP YoY rev growth >100%,
DeepMind's long term failure is Google Cloud's short term gain
newsletter.semianalysis.com/…






























