Qwen3.8-27B reportedly reached 381 tokens/sec on a single RTX 3090 with 24GB of VRAM. The same GPU and model previously ran at around 82 tokens/sec. That is approximately 4.6× higher throughput without upgrading the hardware or replacing the model weights. Developer TeksEdge achieved the reported result by changing how inference is performed, combining speculative decoding, DFlash2, context lookup and optimized verification. Here is the progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized lookup verification: 381 tok/s The jump from 138 to 381 tok/s is particularly interesting. It suggests that the cost of verifying proposed tokens was a major bottleneck in this particular setup. The model itself did not suddenly become more efficient at reasoning. The inference pipeline became better at producing and validating output without paying the full cost of ordinary token-by-token decoding for every token. Here is how the pieces fit together. 1. Speculative decoding reduces sequential generation Most language models generate text autoregressively. Each new token depends on the preceding tokens, so ordinary decoding requires a sequence of dependent operations. Speculative decoding changes that workflow. A faster draft mechanism proposes several tokens, and the main model verifies those proposals. If the predictions are accepted, the system can advance through multiple tokens during one verification step. This reduces the number of sequential generation steps needed to produce the same output. DFlash2 is designed to improve this process by making speculative decoding more effective. However, the benefit depends on draft acceptance, verification overhead and the workload. A draft mechanism that proposes useful tokens consistently can deliver substantial gains. One that frequently produces rejected tokens may deliver much less. 2. Context lookup reuses sequences already present in the input Some tasks require the model to reproduce material it has already seen. Think about copying a document, transforming structured text or generating code that contains long sequences matching the supplied context. Instead of predicting every token individually, context lookup searches for reusable sequences in the existing input and proposes matching text for verification. TeksEdge reports processing blocks of up to 16 tokens and accepting roughly 15 tokens per block in its setup. On a document reproduction task involving 25,000 tokens, the reported throughput increased from approximately 260 to 382 tok/s. That result makes sense for a workload with substantial input-output overlap. The engine has opportunities to reuse existing sequences rather than generate them entirely from scratch. Open-ended conversation is different. New explanations, unfamiliar code and original reasoning may contain fewer sequences that can be copied directly from the context. Consequently, context lookup can be extremely effective on some tasks without delivering the same gains on every coding or chat workload. 3. Verification efficiency can determine the final speed Generating candidate tokens is only half the job. The inference engine still needs to verify them against the main model. If that verification process is expensive, the system can lose much of the advantage gained from proposing multiple tokens at once. TeksEdge’s reported progression from 138 to 381 tok/s after optimizing lookup verification points to the importance of this stage. It also illustrates why inference optimization requires attention to the entire pipeline. Faster draft generation alone is not enough if verification becomes the next bottleneck. The reported stages represent successive configurations. Their individual throughput figures should not be treated as independent multipliers that can simply be combined. 4. Prefix caching addresses a different bottleneck

Qwen3.8-35B-A3B is reportedly generating around 50 tokens/sec on an RTX 3060 with just 12GB of VRAM and 16GB of system RAM. Prompt processing reportedly reaches 550 tokens/sec, with a configured context window of 262K tokens. For developers building local AI agents, this is an interesting setup because it targets hardware many people already own. Reported configuration: GPU: RTX 3060, 12GB VRAM System RAM: 16GB Generation speed: ~50 tokens/sec Prompt processing: ~550 tokens/sec Configured context: 262K tokens Inference engine: llama.cpp with llama-server How does a 35B model run on this hardware? 1. Sparse MoE keeps computation manageable Qwen3.8-35B-A3B has around 35 billion total parameters but activates approximately 3 billion parameters per token. This is a sparse Mixture-of-Experts architecture. Instead of activating the entire model for every token, it routes computation through selected experts. That reduces the computation required during generation compared with a similarly sized dense model. However, active parameters and memory requirements are different things. The full set of model weights still needs to be stored, even when only a subset is used for each token. That’s where quantization and careful memory management become essential. 2. The GPU and CPU share the workload The reported setup uses Q4_K_M quantization, Flash Attention and a Q8 KV cache. It also assigns 26 MoE layers to CPU processing, allowing the system to work around the RTX 3060’s VRAM limit. This comes with trade-offs. CPU processing and transfers between system RAM and GPU memory can become bottlenecks, and the 16GB of system RAM must accommodate the offloaded weights alongside the rest of the inference workload. The reported speed is therefore a result of the complete configuration, not simply the GPU or the model architecture. 3. The real test is replacing an existing coding agent model The developer is now evaluating Qwen3.8-35B-A3B as a potential replacement for Ornith-1.5-35B in Hermes and Pi coding-agent workflows. The comparison includes: Qwen3.8-35B-A3B Ornith-1.5-35B Qwen3.8-27B Base Qwen3.6-35B-A3B The important question is whether Qwen can write code, use tools correctly, debug errors and complete multi-step tasks without constant human intervention. A model that generates 50 tokens/sec but repeatedly breaks its code or mishandles tools may be less useful than a slower model that completes tasks reliably. Context length also deserves scrutiny. A configured 262K-token window doesn’t guarantee the same generation speed, comfortable memory usage or consistent performance at the maximum context. For local AI developers, this is worth watching. If Qwen3.8-35B-A3B performs well in real agent workflows, it could offer another route to running capable coding agents on relatively affordable hardware. The speed figures are encouraging, but reliable task completion will determine whether this setup is genuinely useful.