Vik | Practical AI & Tech

Qwen3.8-35B-A3B reportedly runs at around 50 tokens/sec on an RTX 3060 with 12GB VRAM and just 16GB of system RAM. The reported setup also reaches 550 tok/s during prompt processing, with a configured context window of 262K tokens. Reported hardware and performance: GPU: RTX 3060, 12GB VRAM System RAM: 16GB Generation: ~50 tok/s Prompt processing: ~550 tok/s Context window: 262K tokens Runtime: llama.cpp with llama-server The interesting part is how a model with 35 billion total parameters can run on hardware with these memory limits. 1. Only a fraction of the model is active per token Qwen3.8-35B-A3B uses a sparse Mixture-of-Experts architecture. Although it has roughly 35B total parameters, around 3B are activated for each token. Instead of running every parameter for every token, the model routes computation through selected experts. This can reduce generation-time computation compared with a similarly sized dense model. But there is an important distinction: fewer active parameters do not mean fewer weights to store. The full model still needs to fit across available memory, which is why quantization and CPU offloading matter. 2. The CPU helps the GPU handle the model The reported configuration uses Q4_K_M quantization, Flash Attention and a Q8 KV cache. It also offloads 26 MoE layers to the CPU. This allows the model to operate beyond what the GPU’s 12GB of VRAM could comfortably hold on its own. There are trade-offs. CPU computation, memory bandwidth and transfers between system RAM and VRAM can limit performance. With only 16GB of system RAM, memory pressure also becomes an important factor. The reported speed comes from the entire inference setup, not just the RTX 3060 or the model’s sparse architecture. 3. Can it replace an existing coding-agent model? The developer is testing Qwen3.8-35B-A3B as a potential replacement for Ornith-1.5-35B in Hermes and Pi coding-agent workflows. The comparison includes: Qwen3.8-35B-A3B Ornith-1.5-35B Qwen3.8-27B Base Qwen3.6-35B-A3B The goal is to see how these models perform on actual agent tasks: writing code, calling tools, debugging failures and completing multi-step projects with minimal intervention. Generation speed alone won’t settle that comparison. A model that produces 50 tokens/sec but repeatedly makes mistakes can be less useful than a slower model that finishes the job reliably. The 262K context figure needs similar scrutiny. A configured maximum doesn’t guarantee that the model will run comfortably at that length, maintain the same speed or use all that context effectively. For people building local AI agents, this is a promising configuration to test. It suggests that a 12GB GPU may be enough to experiment with a 35B-class MoE model, provided the quantization and CPU offloading are carefully configured. The real measure will be whether it can complete coding tasks reliably on this hardware, not just how quickly it generates tokens.