Accelerating AI innovation with open platforms and community. The future of AI is open.
Joined May 2018
- Tweets2.7K
- Following2.1K
- Followers12.8K
- Likes1.8K
Pinned Tweet
Everything you need to start self-hosting an open LLM.
Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine.
The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models that cut an 8B from ~16GB of weights to a quarter of that while keeping 98-100% accuracy.
Walkthrough by @cedricclyburn.
Red Hat AI retweeted
🚀 Day-0 vLLM support for Sage-A1.5 (35B MoE, 3B active) from Shanghai AI Lab — agentic intelligence you can serve locally today:
vllm serve InternScience/Sage-A1.5
Congrats @sageai_official ! 🔥
🚀Sage-A1.5 by Shanghai AI Lab is here — pushing 35B agentic intelligence to the next level 🔥
🤖 35B MoE (3B active)
🧩 Agentic Orchestration & Verification — spawns parallel subagents & self-corrects on the fly
🔄 RSI Data Flywheel — evolved via real scientific discovery workflows & tens of thousands of synthesized environments
🚀 Beats frontier giants on FS-Olympiad, FS-Research, CMT & PinchBench, and hits 61.2% on SWE-Bench Pro
💻 Runs locally via vLLM（@vllm_project）, SGLang（@sgl_project）, llama.cpp & LM Studio (Q4_K_M ~20GB)
Scale the Horizon, Compose the Capabilities 👇
🔗 Model:
@huggingface
huggingface.co/InternScience…
@ModelScope2022
modelscope.cn/collections/In…
🕹️ Try it free:
discovery.intern-ai.org.cn/t…
A DFlash2 speculator for Gemma 4 31B, trained with Speculators. It drafts up to 7 tokens per step, and depending on the task the verifier accepts 4 to 7 of them on average. That lands ~1.4x throughput on coding/math workloads at high concurrency:
huggingface.co/RedHatAI/gemm…
Your LLM request doesn't go straight to the model. It gets tokenized, batched with other requests, run through prefill to build a KV cache, then decoded one token at a time. Here's the full inference pipeline in ~7 minutes: youtube.com/watch?v=615-yWyS…
vLLM Office Hours today, 2pm ET: advanced techniques for model compression.
Recent algorithmic work on observers, our AWQ and GPTQ recipes, and expert pruning with REAP and MoNE.
Plus what's new in the latest vLLM releases and.
Register: vllm.ai/office-hours
Red Hat AI retweeted
During the PyTorch Conf keynote, Abdullah Gharaibeh gave a live demo of a couple of llm-d features.
Watch live as llm-d routing and flow control provides real-time benefits for the user at scale.
youtube.com/watch?v=F-3YgAYy…
Red Hat AI retweeted
🚨🚨🚨 vLLM is all you need to reproduce DeepSeek’s performance.
With Wide-EP and EPLB, vLLM can match DeepInfer’s decode performance within 6% and generate $39/gpu/hr in revenue at DeepSeek’s off-peak pricing, at 200 P90 TPS on the AgentX workload.
Specifically:
• Under DeepSeek’s decode configuration (DeepSeek-V4.1-Flash, 131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer’s published results across batch sizes 8–256.
• On AgentX, vLLM pushes the Pareto frontier further on GB300, reaching up to 1M tokens/s/GPU and generating $38.96/gpu/hr in revenue at 196 P90 TPS, based on DeepSeek’s off-peak API pricing.
• Also, vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash, achieving up to 826 TPS per user on AgentX.
Hola, Madrid. vLLM and llm-d are coming to town: Red Hat and @nvidia are hosting an evening on open source inference, with @vllm_project, LLM Compressor, vLLM-omni, safe agents, and @_llm_d_ on Kubernetes.
12 November | MOA Institute | Register:
luma.com/gl1d2loa
Red Hat AI retweeted
Red Hat researchers, in collaboration with Purdue, developed VEX-Bench—a benchmark for assessing large language models (LLMs) in software supply chain vulnerability triage. Traditional tools often generate up to 70% false alerts. Discover the findings: red.ht/4debJqd #EMNLP2026
Amazing ~4-minute walkthrough from @LegareKerrison of how @NVIDIAAI's OpenShell sandboxes autonomous agents using declarative security policies, policy as config, not code buried in the app. youtube.com/watch?v=sXXYJQ-3…
Red Hat AI retweeted
Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉
Distributed inference is too complex to solve behind closed doors.
17 months in, llm-d is now a @CloudNativeFdn project powering fleets worldwide.
Here’s what’s inside 1.0 🧵👇
When does vLLM beat llama.cpp? What happens when a model outgrows one GPU? Can you trust an agent benchmark? Where do MCP and agent skills fit?
Red Hat AI folks have been answering these questions on @IBM's Technology channel on YouTube.
A 🧵 of deep dives for your watch list:
Want to become an AI engineer? @cedricclyburn covers the foundations: Python, APIs, RAG, embeddings, agents, observability, and deployment.
youtube.com/watch?v=XN3xNJvW…
That's the watch list. Our experts show up regularly on IBM Technology with explainers on inference, agents, and open-source tooling. Full channel:
youtube.com/@IBMTechnology/v…
