Luis Capelo retweeted
0-100k Minecraft agents in 60s!
DM me if you want your limits raised to build something like this.
Luis Capelo retweeted
Kimi K3.1 Pro from @Kimi_Moonshot is now available on Modal.
A major post-training update to Kimi K3, with expanded reinforcement learning for stronger coding, financial research, and long-horizon agentic work.
Kimi K3.1 Pro supports text and images with 1M context.
Luis Capelo retweeted
I'm joining @modal!
Software is undergoing a revolution; we need new tools and infrastructure for this new era.
@modal is the platform I want in these times, so I decided to help build it.
If you want to come build it with me: modal.jobs and my DMs are open!
📢 ADSP Episode 310 is out! 📢 In this episode, @blelbach and @code_report chat about Bryce leaving @nvidia, where he is headed next and more! 🥳 adspthepodcast.com/2026/10/3…
Luis Capelo retweeted
Alright I finally have more to share on my personal project!
Resin is a runtime for building fast LLM inference engines.
The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target.
To build an engine:
1. Import Resin's core.
2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want.
Then, boom, you have a standalone inference engine.
—
I've been hacking over the past month (early post: xitter.gq/ekzhang1/status/208950…), and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer.
For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache.
Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4).
So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work.
The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI.
Code: github.com/ekzhang/resin
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching)
I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell):
- DeepSeekV4 Flash
- Gemma 3.8 27B
- GPT-OSS
- Inkling
It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark
Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this.
See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)
Luis Capelo retweeted
We worked with @SakanaAILabs to optimize their Sakana Namazu API down to the kernel level:
→ Selective NVFP4 quantization (23k+ experts)
→ DFlash draft model trained on raw token streams (+25% acceptance)
→ In-house prefix KV routing + serverless autoscaling
Luis Capelo retweeted
Congrats to @GradiumAI on launching Phonon!
We built a voice agent with their new TTS models and Pipecat on Modal to put them to work.
Read more about Phonon and check out the demo: gradium.ai/phonon
A new frontier for on-device speech.
We're open-sourcing Phonon, a suite of TTS models under 50m parameters, setting a new standard for naturalness and accuracy across languages, with the broadest inference support to date.
Phonon-31m is available under Apache 2.0.
gradium.ai/phonon🧵
Luis Capelo retweeted
We got our hands on @nvidia's new Vera CPU and put it to the test.
We built a benchmark using real-world agentic workloads to see how it performs under contention.
In early tests, Vera delivered up to 2x the throughput of comparable x86 hardware, maintaining low latency even as hundreds of concurrent workloads competed for resources.
For Modal Sandboxes, that could mean running more agents in parallel without sacrificing latency.
Luis Capelo retweeted
The secret to Modal's speed is our custom lazily loaded container filesystem, ImageFS. ImageFS is built on a massive tiered distributed filesystem, layering local NVMe disks, regional and p2p caches, and global blob stores.
When we launched VM Sandboxes this month, we wanted to make sure they could take advantage of ImageFS in the same way our gVisor containers do. To do that, we built out a new filesystem stack that tricks a VM's ext4 driver into thinking it's talking to a normal disk.
Luis Capelo retweeted
Nemotron 3.5 Super VL from @nvidia is live on Modal today with 120B params for visual reasoning and agentic work across images, docs, and UIs.
Hear more about the thinking behind the Nemotron family: At Runtime, @ctnzr explained why “efficiency is also intelligence,” and how NVIDIA pre-trained Nemotron 3 Ultra entirely in FP4.
Luis Capelo retweeted
Muse Spark Open, a coding-optimized open weight model from @Meta, is now available on Modal.
Muse Spark Open is the most capable US open weight model to date and is trained specifically for agentic coding and long-horizon tasks; it supports native multimodal understanding and supports a 1M token context window.
Luis Capelo retweeted
Nemotron 3.5 Super VL from @nvidia is now available on Modal.
With 120B total parameters and 12B active per token, this open-weight multimodal model brings powerful visual reasoning to documents, images, and application screenshots—enabling sophisticated agentic workflows.
Luis Capelo retweeted
In early 2022, the first “real” @modal user was using us for processing large scale weather data. We were anxious about going up to them and asking them to pay ~$500 for continuing to run these jobs. (We thought it was a lot!)
Obviously the numbers are much bigger now.
But we’ve also built so much more since then, and our velocity and ambition are growing even faster. This year, we figured out how to scale to millions of concurrent sandboxes, launch RDMA clusters securely and elastically for distributed training, train state of the art speculators for inference, build a whole new sandbox runtime, and more. So many great rabbit holes, and we’re going deeper down all of them.
If this sounds interesting to you, we’re hiring! modal.jobs
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
Luis Capelo retweeted
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
Luis Capelo retweeted
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Luis Capelo retweeted
From frogs to the frontier, @ScottWu46 covered a lot at Runtime:
→ Agents are moving from tab completion to “virtual employees” that own outcomes
→ Frontier models may only need ~2% of workloads; routing handles the rest
→ 91% of Cognition’s internal Devin sessions start without a human
Luis Capelo retweeted
Folding models, protein diffusion, single-cell LLMs, and agent swarms: AI for science runs on a different model every hour and needs a different number of GPUs every day.
Listen to leaders in computational biology from @GSK, @AnthropicAI, and @achira_ai on the main stage at Runtime.
Luis Capelo retweeted
Jev launched on a Thursday. By Sunday, the @TypeSafeAI team had served a trillion tokens with no signs of slowing down.
Learn how they scaled Jev on Modal.
Luis Capelo retweeted
We threw our first conference last week in SF. Tons of cool people and talks. A very high correlation between being cool and being there.
Anyway, this is the keynote, in which I announced a bunch of Modal features. We also launched 1M sandboxes in a minute for fun!
How can you not be excited seeing so many product announcements? Come build with us modal.jobs