Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about. 💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick) My llama.cpp recipe 👉 🧠 Qwen3.5-4B Q4_K_M GGUF ⚙️ Ryzen 5 7540U — 6C/12T 🧵 --threads 9 🧵 --threads-batch 12 ⚡ --prio 2 🔄 --poll 50 📦 --batch-size 2048 📦 --ubatch-size 512 🚀 --flash-attn on 🧠 KV cache: q4_0 / q4_0 🔧 --repack 💾 --mmap 👤 --parallel 1 🚫 --device none 🚫 --gpu-layers 0 🚫 KV/op GPU offload 🚫 MTP OFF Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks). Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s. That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

Okay so imagine you're chatting with a local LLM that has allocated most of your GPU's VRAM. Then you launch a game. Instead of killing the model, LlamaForge gives the VRAM back. That's the idea behind a new experimental feature in LlamaForge called Elastic VRAM. When another application starts grabbing GPU memory 🎮 Game needs more VRAM ⬇️ LlamaForge moves model layers back to CPU 🧠 MoE experts get moved first 💬 Your existing conversation survives ⬆️ When VRAM becomes available again, layers move back to the GPU The model gets slower while it's shrunk to CPU. But it doesn't have to disappear completely just because you launched something else. Underneath, LlamaForge is running upstream llama.cpp and uses llama.cpp's slot save/restore to preserve the chat while it rearranges where the model lives. There's also a simpler Game Mode Launch game → unload model Quit game → reload model Elastic VRAM is the more interesting version because the model stays available while relinquishing GPU memory. Keep in mind that it is ... ⚠️ Experimental 🟢 NVIDIA only today ⚙️ Off by default 🧪 Not widely tested yet 🔓 LlamaForge → MIT / local Link in ALT

Are you getting ready for Kimi K3.1? It may already be testing in Arena under the anonymous name Kibitz. And there are now multiple clues pointing back to Moonshot. 🧬 Community tokenizer/fingerprint tests reportedly behave like Kimi 🗣️ In some Arena tests, Kibitz identifies itself as Kimi/Moonshot 🔀 Kimi Code's configuration now contains a model-switch target named k3.1 And the anonymous naming pattern is eye catching K2.5 → Kiwido K2.6 → Kiwire K3 → Kivine K3.1? → Kibitz K3 itself reportedly appeared anonymously in Arena roughly 30 hours before release. None of this confirms Kibitz = Kimi K3.1. Moonshot has not announced anything officially yet. But K3.1 itself looks increasingly real. Kimi Code now literally contains a migration rule pointing existing K3 sessions toward k3.1. If Moonshot follows K3 with open weights again, K3.1 could become the next ridiculous multi-GPU / multi-node Local AI project. Links in ALT

A new decision model tied for #1 on Artificial Analysis' Decision Index and its 8 points ahead of Jev 1.14. SubQ Snap 1.1 is built to make the decisions inside an AI agent. Artificial Analysis Stats 🥇 SubQ Snap 1.1 → 65 🥇 d1-ultra → 65 🥇 Elephant 9B → 65 Jev 1.14 → 57 And Snap has one unusual advantage, its context 📚 384K context Subquadratic says its Sparse Attention architecture reduces attention compute from quadratic to linear scaling with context. Snap reads the context and returns probabilities over a fixed set of allowed decisions in a single forward pass. Subquadratic reports these benchmarks 🧰 When2Call → 96.2 📉 81% fewer confident errors on its unanswerable-query test 📚 384K context 💰 $0.04 / 1M input tokens 🆓 no output-token charge This isn't a local model today as Snap 1.1 is currently offered through Subquadratic's API. Layer this between the prompt and the models Link in ALT

Nvidia released Kumo Relational, a small purpose-built AI model to predict what happens next from your relational database.👀 👉Customer churn. | Fraud. | Future demand. | Conversion. | Risk. This is not a replacement for SQL. SQL tells you what already happened. Kumo is designed to predict what happens next from the relationships and history already inside your database. And it can do this without training a separate model for every task or manually engineering features. Instead of flattening a relational database into one giant table, it works across the relationships between your existing tables. Give it historical examples in-context, up to 10,000, and ask it to perform classification or regression. Nvidia tested it across 📊 41 tasks 🗄️ 15 databases They report it beating supervised relational deep-learning systems and expert-engineered gradient-boosted-tree pipelines. On SAP SALT, Nvidia says it beat large AutoGluon ensembles by 8% MRR. You can download the weights and run it locally on CUDA. The checkpoints are tiny 🧠 Classification → ~120MB 📈 Regression → ~124MB Your model doesn't just query your private database anymore. It can try to predict what happens next from it. 🔓 NVIDIA-authored code → Apache 2.0 ⚖️ Weights → OpenMDW 1.1 Link in ALT

2 x ~$400 Snapdragon mini PCs. No GPU used and 64GB of RAM combined. And they're running a 95GB Qwen3.8-Flash-Next at 20.3 tps. 👀 The model is 31GB larger than their combined RAM. I have to build this. The setup 🖥️ 2× Lenovo IdeaCentre Mini ⚙️ Snapdragon X X1-26-100 💾 32GB LPDDR5 each 💽 512GB NVMe each 🔌 connected with one USB4 cable 💰 ~$800 total Here's the model: 🧠 Qwen3.8-Flash-Next 📦 IQ4_NL → 95GB Reported with the author's ToRc inference engine 📥 Read → 46.2 tps ✍️ Generation → 20.3 tps 💻 Coding generation → 19.2 tps And the Adreno GPUs aren't being used. ToRc uses the Snapdragon CPUs + Hexagon NPUs for some of the dense math, while the model's experts are streamed from NVMe as they're needed because the weights don't fit in RAM. The second box also isn't simply splitting normal token generation in half. For Flash-Next, ToRc pipelines MTP across the two machines over USB4. ⚠️ ToRc isn't publicly released yet and these are the author's results, not independently reproduced benchmarks. Link in ALT

This may be one of the more important Healthcare specialized Local AI weight releases. Ant open-sourced the weights for Ling-3.0-flash-Sante. The model itself isn't new as Sante has been available through APIs since September but ... What's new is 👉 you can now bring the healthcare model home. Stats 🧠 124B total parameters ⚡ only ~5.1B active/token 📚 256K context 🩺 healthcare-specialized 🔎 deep research 🤖 multi-agent collaboration 🔧 tool use It's built on the same Ling-3.0-flash architecture that already runs in llama.cpp. For reference, base Ling-3.0-flash GGUFs already exist around Q4_K_M → ~77GB IQ3 → ~54GB IQ2 → ~40–46GB IQ1 → ~28–31GB ⚠️ Those are BASE Ling-3.0-flash quants. I haven't found an official Sante GGUF yet. But now that the Sante weights are downloadable, this becomes a home model for 🖥️ 96/128GB workstations 🍎 high-memory Apple Silicon ⚡ DGX/RTX Spark-class unified memory 🔗 multi-GPU local systems And healthcare is one of the places where Local AI can be useful for private documents + research + agent workflows can stay on hardware you control instead of being sent to a cloud API. Ant also reports substantial healthcare gains in the new release, including HealthBench Professional 45.73 → 54.29 AFUMED ComplexReasoning 65.22 → 76.92 Link in ALT

Wow, that's a pretty nice bump for Ling-3.1-flash. Ant introduced the new Ling-3.1-flash-1019 checkpoint, and its Artificial Analysis score jumped Ling-3.1-flash → 41 Ling-3.1-flash-1019 → 44 The highlighted gains come from its updated agentic performance AA-Briefcase ~1400 → 1524 Elo AutomationBench-AA 62% → 68% The existing Ling-3.1-flash is already open-weight under MIT and it's big. Stats 🧠 ~560B total ⚡ ~25B active/token 📚 up to 1M context Community GGUFs already go down to IQ1_S → 142GB IQ2_XXS → 174GB IQ3_XXS → 256GB Q4_K_M → 354GB The existing Q4 has even been loaded across 4× DGX Sparks using llama.cpp RPC. ⚠️ 1019 is a new checkpoint/update, not a new Ling architecture announcement. Ant hasn't released the updated 1019 weights yet. It says open weights + API are coming soon. Link in ALT

Apparently, private Local AI roleplaying AI is a pretty serious use case. ReadyArt's new Wicked-Tutu family is already pulling serious downloads. And this is not a tiny model Stats 🧠 31B dense Gemma 4 💾 Q4_K_M → ~18.7GB 🎮 fits in 24GB-GPU territory 📚 up to 256K context 🏠 completely local Runs directly with ⚙️ llama.cpp 🦙 Ollama 🖥️ LM Studio ReadyArt trained it specifically as a private local roleplay model, with a purpose-built dataset and thousands of scenario categories. The earlier v0.1 GGUF has already passed 17K downloads, and they're iterating fast enough that a Final 31B build is already out. Local AI isn't only competing with ChatGPT but it's enabling categories where privacy + customization + no cloud at all are the product. ⚠️ The HF metadata says Apache 2.0, but the model card also adds personal/non-commercial terms, so I'd treat the licensing as unclear for now. Link in ALT

This small Local AI model could replace an entire document-processing pipeline. ATH-MaaS released OvisOCR3. It’s only a 0.8B-class multimodal model. But one local model can now do all these things 📄 document → Markdown 🧾 document → structured JSON 🔎 OCR + bounding boxes 🧮 formulas → LaTeX 📊 charts → structured data 🔑 key-value extraction 🖼️ figure captioning 🔀 flowcharts → Mermaid 📚 cross-page table reconstruction That last one is interesting. Give it consecutive document pages and OvisOCR3 can determine whether a table continues onto the next page and reconstruct the complete table. Team-reported results OCRBench → 88.8 OCRBench KIE → 90.5 Cross-page table Relation F1 OvisOCR3 → 97.07 Qwen3.8-Max → 94.47 And it’s 🧠 built on Qwen3.5-0.8B 🔓 Apache 2.0 🏠 downloadable/local ⚙️ Transformers + vLLM + SGLang 🔌 OpenAI-compatible local server through vLLM No official GGUF that I can find yet. Link in ALT

The company behind PUBG just trained its own 29B Local AI model from scratch. And the architecture is interesting. KRAFTON’s new Raon-1 mini is … 🧠 28.8B total parameters ⚡ only 4.3B active/token 💾 official Q4_K_M → 19.7GB 📚 262K context 🔓 Apache 2.0 🤖 reasoning + agent/tool use 18 of its 25 layers use fixed-state linear attention. Only 7 layers maintain a growing KV cache. KRAFTON says those layers use only ~8 KiB of BF16 KV per token. So even at the full 262K context, that’s roughly: 👉 ~2GB of growing KV cache before other runtime/state overhead. And in KRAFTON’s reported comparisons, this new 30B-A4B model beats Qwen3.6-35B-A3B on 11 of 18 listed benchmarks. Lots of new models coming for Qwen3.6-35B recently. ⚠️ Preview release. KRAFTON’s benchmark results. Upstream llama.cpp support is still pending and KRAFTON provides its own branch today. Link in ALT

Nvidia lets you build a 1TB Local AI cluster out of DGX Sparks. & Nvidia has a playbook for AI clustering them. 8 × DGX Spark 🧠 128GB each ↓ 🔥 1TB unified memory across the cluster Nvidia now officially supports manually configured clusters of up to 8 x DGX Sparks using their built-in ConnectX-7 networking. And Nvidia says that 1TB is enough to run frontier-class models like GLM-5.3 locally. They specifically call out 🧠 larger models 📚 longer contexts 🤖 multiple concurrent agents 👨‍💻 multiple developers sharing one local backend ⚙️ distributed development They list models such as → GLM-5.3 Flash → DeepSeek V4 Flash → Qwen3.8 Flash Next And later this month Nvidia Sync Model Launcher is supposed to make it easier. Link in ALT

AMD and Intel agreed on something pretty unusual. A local PyTorch AI app shouldn't have to care whether the CPU underneath it is AMD or Intel. AMD is folding its ZenDNN AI optimizations into oneDNN, the library originally developed by Intel. AMD + Intel have now aligned on oneDNN as the default open-source x86 acceleration path in PyTorch. ZenDNN has been working on refining for ... 🧠 LLM inference ⚡ INT4 / INT8 / BF16 / FP16 🧩 fused MoE operations 🚀 vLLM CPU acceleration 🔧 optimized attention/matmul Now that work is being brought into the common oneDNN ecosystem AMD + Intel are also jointly developing ACE, new x86 AI matrix instructions. The plan is for future AMD/Intel AI instructions → oneDNN → PyTorch So local PyTorch software gets one x86 target instead of separate vendor paths. ⚠️ This does NOT make llama.cpp/GGUF faster, and AMD/Intel haven't published new performance numbers from the merger. 🎯But for CPU Local AI, the two companies that spent decades fighting over x86 are now cooperating on the AI software layer. Link in ALT

Here we go! Purpose built AI generated LLMs! Then AI helped optimize the code that runs it. Recursive released Flint, a new coding/agent model designed specifically for fast local inference. Beats Qwen3.5/3.6/3.8 in some coding benchmarks! Stats … 🧠 39.8B total parameters ⚡ only ~2.5B active/token 📚 262K documented context 🤖 reasoning + tool use 🔓 Apache 2.0 And there are official GGUFs: 🟢 NVIDIA NVFP4 → 22.9GB 🍎 Apple INT4 → 26.0GB Recursive recommends 48GB-class memory for these local builds, so think: 🍎 48GB+ Apple Silicon 🟢 48GB+ NVIDIA ⚡ 64GB RTX Spark Recursive reports its optimized runtime at 🍎 M4 Pro → >125 TPs 🟢 64GB RTX Spark → >150 TPs And WIRED’s Will Knight actually ran Flint locally on a 48GB Mac and reported it was fast and successfully handled the coding jobs he tried. Side Note: Recursive built Flint using its Eureka Machine, an automated AI research system that helped 🧩 design the architecture 🧪 develop training recipes ⚙️ optimize infrastructure 💻 write hardware-specific inference kernels Nvidia says those specialized kernels improved Flint’s RTX Spark generation performance by 55% wo changing the model weights. Recursive-reported coding results SWE-bench Verified → 74.2 SWE-bench Pro v2 → 89.7 Terminal-Bench 2.1 → 54.9 LiveCodeBench v6 → 80.3 So this isn’t merely AI writing more AI code. Link in ALT

A single 24GB GPU can now run a 35B scientific research agent that spawns sub-agents and checks its own evidence. Sage-A1.5 is 35B / ~3B active, and its official Q4_K_M is just 21.7GB. Here are the new Sage-A1.5 model stats 🧠 35B MoE / ~3B active 💾 official Q4_K_M → 21.7GB 🎮 fits a single 24GB GPU at moderate context 📚 256K native context 👁️ multimodal 🔓 Apache 2.0 And it's specifically trained as a long-horizon research agent. So it is designed to 🔎 use tools/search/code 🤖 spawn parallel sub-agents ✅ inspect its own evidence 🔄 change direction when research stalls No separate critic model required for its verification mode. These are the full-precision benchmark results PinchBench Sage-A1.5 → 90.0 Kimi K3 → 88.2 GLM-5.3 → 86.6 GPT-5.6 Sol → 87.3 FrontierScience-Olympiad Sage → 81.0 Kimi → 69.0 GLM → 67.0 GPT → 73.0 SWE-Pro Sage → 61.2 Kimi → 63.3 GLM → 64.6 GPT → 64.6 ⚠️ Those benchmark scores are from the BF16 checkpoint, not the Q4. But the same model now has an official 21.7GB GGUF with llama.cpp / Ollama / LM Studio support. Link in ALT

This might be my new favorite all-purpose local AI model. Fits nicely into 16GB of VRAM for an RTX 5060/5070TI. The new SparkX2.6-4B beats Qwen3.5-9B in 80% of shared benchmarks and 97% of Gemma4-12Bs. Stats 💾 official Q4_K_M → 3.54GB 🧠 only 4B parameters 📚 native 1M context 💭 none / low / high thinking budgets 🤖 trained for local agents 🔓 Apache 2.0 It already runs with: ⚙️ llama.cpp 🦙 Ollama 🖥️ LM Studio 🍎 MLX 🚀 vLLM / SGLang In the team's reported benchmark suite Averages: Spark-X2.6-4B → 63.3 Spark-X2.5-4B → 52.1 Qwen3.5-9B → 49.1 So they're claiming a 4B model beats their tested 9B comparison overall. For normal local-agent contexts, though, a ~3.5GB reasoning/agent model with this runtime support is very interesting. Link in ALT

Underdog released Bark 3.0, and it's only a 163MB download. Runs locally w/Pytorch+CUDA or MLX On their Open ASR Leaderboard evaluation, Bark wins 🐶 Bark 3.0 → 4.94% WER → 163MB 🟢 NVIDIA Parakeet → 4.94% → 2.51GB 🌀 Whisper large-v3 → 6.04% → 9.26GB So Bark matches Parakeet's average accuracy while being 👉 ~15× smaller And beats Whisper while being 👉 ~57× smaller On recorded meetings check out these stats Bark → 7.82% WER Parakeet → 9.42% Whisper large-v3 → 13.63% And this isn't CUDA-server-only. 🏠 fully local 📱 phone-class 🍎 MLX + Core ML 🧠 Apple GPU / ANE 🪟 CPU + CUDA paths 🔓 CC-BY-4.0 I've checked and no GGUF. It uses its own compressed Bark format plus MLX/Core ML/Transformers runtimes. Bark 3.0 Plus is only 180MB and improves average WER to 4.76%. ⚠️ English only Link in ALT