Building the future of inference through @vllm_project
San Francisco, California
Joined December 2025
- Tweets162
- Following5
- Followers7.9K
- Likes145
Our Kimi K3 result running @vllm_project TPU v7 reaches 391 tokens/s per user on @SemiAnalysis_ AgentX.
This result uses our TPU megakernels in vLLM TorchTPU and holds up on real, long-context, production-level agentic workloads. The results are publicly viewable on InferenceX, and the megakernels are open source.
1/2
We’re heading to #PyTorchCon! 🎤 Catch our talks and meet the team at Inferact booth P1.
☕ Join us, Anyscale & Nscale at the Inference & RL Coffee Club, Oct 20–21, noon–5 PM PT:
luma.com/anyscale-pjvj
Looking forward to seeing you there! 🙌
The vLLM community is coming to #PyTorchCon, Oct 20–21 in San Jose! 🎉
Explore talks on attention, KV transfer, prefix caching, the vLLM roadmap & more.
📅 Build your schedule:
events.linuxfoundation.org/p…
We can't wait to see you there! 🙌
Inferact is an API launch partner for MiniMax M3.1 Flash! 🎉
We're working with @MiniMax_AI on production inference for its latest model, built for multi-step automation, real-codebase engineering and visual reasoning.
M3.1 Flash scored 29.6 on Agents’ Last Exam (ALE), alongside 72.5% on AutomationBench-AA and 73.67% on DeepSWE—capabilities that matter when agents need to plan work, use tools and carry tasks through.
Our contribution is the serving stack: bringing our vLLM engineering and the Inferact Platform to the collaboration.
Thanks to the MiniMax team for the partnership.
Explore the model and API integration:
platform.minimax.io/docs/gui…
MiniMax M3.1 Flash: Space Performance + Bunny Speed
🚀🐰 Yes, Space Bunny - which has topped global leaderboards with 200T+ tokens - was an earlier version of M3.1 Flash.
💰Redefining the performance economics of agentic AI. M3.1 Flash achieves frontier-level intelligence across key tasks such as Cowork (Automation-Bench: 62%; ALE: 30%), Coding (DeepSWE: 74%) and Visual Reasoning (BabyVision: 91%), at 1/3 the cost of M3 and 3x the speed.
API pricing (flat rate): $0.15 / $0.60 per 1M input/output tokens, with cache reads at $0.015 / 1M tokens.
Model weights will arrive on Oct 23.
API: platform.minimax.io
M Plan: minimax.io/m-plan
MiniMax Code for desktop: agent.minimax.io/download
Inferact retweeted
Today, we are announcing an engineering collaboration with @AMD to strengthen production agentic inference on AMD Instinct GPUs in @vllm_project.
AMD brings the Instinct platform, ROCm engineering, and the AITER kernel library, while Inferact and the vLLM community bring model integration, serving features, and day-0 enablement.
Together, we are focused on agentic workloads with long-running sessions, long prefixes, and tool calling, and each configuration will ship as a public per-model serving recipe.
This work is already running in production. It shipped with this summer’s major open-model releases, and Inferact uses it on AMD Instinct clusters for our Inferact Platform customers.
Everything this collaboration produces lands in open source. Read the full announcement: inferact.ai/news/inferact-co…
Today, we are announcing an engineering collaboration with @AMD to strengthen production agentic inference on AMD Instinct GPUs in @vllm_project.
AMD brings the Instinct platform, ROCm engineering, and the AITER kernel library, while Inferact and the vLLM community bring model integration, serving features, and day-0 enablement.
Together, we are focused on agentic workloads with long-running sessions, long prefixes, and tool calling, and each configuration will ship as a public per-model serving recipe.
This work is already running in production. It shipped with this summer’s major open-model releases, and Inferact uses it on AMD Instinct clusters for our Inferact Platform customers.
Everything this collaboration produces lands in open source. Read the full announcement: inferact.ai/news/inferact-co…
Our new website is live! Along with a new visual identity for Inferact.
Inferact gives AI teams production inference they control, run optimized inference wherever your compute lives.
Come check it out inferact.ai?url
Thank you @a16z for telling the Inferact story.
Four years in, inference is still far from solved. Models keep getting bigger, workloads keep getting more varied, and the hardware landscape keeps expanding. Inferact exists to build the future of inference with the vLLM community.
We’re hiring across all teams in SF and Singapore: inferact.ai/careers
In 2022, Berkeley PhD student Woosuk Kwon thought he could make Meta's new open-weight model, OPT-175B, run faster in two weeks.
Four years and 3,000 contributors later, that project is @vllm_project, one of the largest open-source inference engines, running on ~500,000 GPUs at any given time.
He and fellow researchers Simon Mo, Kaichao You, and Roger Wang borrowed "virtual memory" from operating systems, handing out memory only as it's needed so each GPU can juggle requests of wildly different sizes. (The "v" in vLLM is for virtual.)
Their first user, Berkeley neighbor @arena, cut its GPU needs in half. Today Google, Amazon, and Meta all run vLLM in production.
In January, the core maintainers founded @inferact and raised $150M to give vLLM a full-time team.
They're hiring across eng, marketing, and devrel in SF and Singapore: a16zjobs.substack.com/p/infe…
@simon_mo_ @woosuk_k @KaichaoYou @rogerw0108
Thanks for the shoutout @SemiAnalysis_! Our team at @inferact has been optimizing @vllm_project on GB300, and those gains translate directly into better TCO. More optimizations are coming soon.
Current results on InferenceX: inferencex.semianalysis.com/
ALERT🚨🚨: @vllm_project, @Inferact & NVIDIA TOOK CO-DESIGN TO THE EXTREME WITH A GB300 NVL72 INFERENCE OPTIMIZATION TO GET 🚀 7.2x 🚀 BETTER PERF PER TCO THAN LITERALLY EVERY OTHER AI CHIP, using an inference co-design optimization called wideEP take full advantage of the NVL72 BACKPLANE. We explain below👇️ (1/3)🧵
Are you heading to @PyTorch NA next week?
We’re partnering with @anyscalecompute and @nscale to create a space where you can step away from the conference madness, grab a coffee, catch up with peers, and get some work done.
Come hang out, take a breather, and connect with the community on inference and @vllm_project.
Request your spot here: luma.com/anyscale-pjvj?utm_s…
vLLM on NVIDIA Vera Rubin NVL72 delivers more than 7.8x the throughput of GB200 on MiniMax M3 at matched interactivity, measured on AgentX.
Inferact’s team has been co-leading this effort since Rubin was announced. We have integrated Rubin optimized MSA prefill kernel and tuned AgentX performance with locality aware optimizations targeted for Rubin GPU. These are early results.
🧵 1/3
vLLM now supports NVIDIA Vera Rubin. The early results show more than 7.8x the throughput of GB200 on MiniMax M3 on AgentX.
@inferact, @NVIDIAAI, @RedHat_AI, and the vLLM community have been bringing vLLM up on Rubin since it was announced. Here is where things stand.
🧵 1/5
Together with @NVIDIAAI, Inferact led performance tuning, MiniMax Sparse Attention kernel integration, and locality-aware MoE.
We use CUDA locality domain feature to optimize MoE layer on Rubin GPU so each SMs can read MoE weights from their own locality domain. That makes MoE layers up to 1.2x faster in decode.
🧵 2/3
These changes land in open-source vLLM, and the Inferact platform benefits directly from that upstream tuning. Founded by the creators and core maintainers of vLLM, Inferact runs open models in production on your compute or ours.
Early work on hardware like NVIDIA Vera Rubin shows us where inference is heading, and we use what we learn to optimize inference for your workload.
Learn more at inferact.ai
Great collaborating with @deepseek_ai, @nvidia, and @SemiAnalysis_ to build this with the @vllm_project community.
Dropping TTFT nearly 70% at ~100K throughput reflects the joint work across models, custom kernels, and engine serving.
vLLM is the open source engine; Inferact optimizes it and brings enterprise inference to production.
xitter.cf/vllm_project/status/21…
4/ Together, TTFT drops nearly 70% at ~100K throughput.
Thanks to @deepseek_ai for the model and kernels, @nvidia for the collaboration, and @SemiAnalysis_ for AgentX. Built by @inferact and the vLLM community.
Come join us at vLLM Meetup Toronto! Our Co-founder and Lead vLLM Maintainer @rogerw0108 will be speaking on where @vllm_project is heading next, alongside engineers from Cohere and NVIDIA.
The event is free, but spots are limited. Register via the link in the thread.
Open doors for open weights 🚀
@vLLM and Cohere are co-hosting a meetup in Toronto to discuss how we're contributing and pushing the future of open source.
Plus, hear from a lead maintainer of vLLM on where vLLM is heading next and from @NVIDIAAI engineers on agentic debugging. Connect with our teams and learn something along the way 🧠
Inferact retweeted
Runtime is this week at The Midway in San Francisco.
Thanks to our partners for helping us put it together: @nvidia, @pydantic, @radixark, @sgl_project, @temporalio, and @inferact, @vllm_project 💚
Join us to learn from the engineers behind these tools.
Inferact retweeted
This work was done by @Inferact . Check out their amazing deep dive here. THANK YOU FOR YOUR ATTENTION TO THIS MATTER
inferact.ai/blog/tpu-megaker…
Thanks for the shoutout @SemiAnalysis_ ! Full breakdown linked here:
inferact.ai/blog/tpu-megaker…
ALERT ALERT ALERT 🚨 🚨 🚨 VLLM MAINTAINERS HAVE JUST SHOWN THAT TPUv7 CAN GET 700 tok/s/user, 56% BETTER PERFORMANCE THAN NVIDIA GB200 NVL72 THROUGH MEGAKERNEL OPTIMIZATION ON KIMI K3.
As we said awhile ago, the TPU externalization of software is full steam ahead. This is ultra important to follow the progress of this.
Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding.
To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8.
We are open sourcing it today.
1/2