Machine Learning Engineer | Researcher | Dota2 veteran
Sri Lanka
Joined May 2009
- Tweets2.9K
- Following1.7K
- Followers551
- Likes2.2K
Samitha Jayaweera retweeted
We're giving away a Hogwarts Legacy custom wrapped NVIDIA GeForce RTX 5090 to celebrate its official release with DLSS 5 🪄
To enter:
🟢 Share this post
🟢 Comment #RTXON
Terms and Conditions: nvidia.com/en-us/geforce/con…
Samitha Jayaweera retweeted
Concept Country has developed its first open-source model, Qwen Concept 27B.
With 54% less thinking tokens and 3% better performance on GPQA Diamond compared to its base Qwen 3.8 model, we have achieved 30% more useful tokens per joule without additional post-training.
Samitha Jayaweera retweeted
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: huggingface.co/spaces/huggin…
Samitha Jayaweera retweeted
We are announcing three new discoveries in mathematics, computer science, and physics about harmonic analysis, complexity theory, and quantum field theory.
openai.com/index/building-sc…
Samitha Jayaweera retweeted
The good news: you can mid-train yourself to the frontier!
At a fraction of the cost! Without distillation!
What we did: We took Qwen3.5 14B and mid- and post-trained it to beat every other open model and perform as well as the very best frontier models.
This is no simple toy task or format fixing, we are training the model to understand and reason over DNA sequences. Bio is the next frontier for AI and universal access will be crucial to make progress.
The fun part: because we train the model on both natural language and DNA sequences you can actually 'chat to your DNA': Ask it questions, ponder the impact of mutations, or ask it to generate a small sequence.
We'll soon see the frontier labs move away from math and cyber towards bio. It already started, and we believe researchers should have access to capable models.
We are releasing the full recipe, so you can do it, too!
Model: huggingface.co/HuggingFaceBi…
Blog: huggingface.co/spaces/Huggin…
Samitha Jayaweera retweeted
Life is good now. 🎉 Tell your agent to:
- create a new volume in you mac
- disable fseventsd (and spotlight) on it
- and move your projects folder to it
- remount it
For extra points move you pnpm cache folder to it.
(Start the conversation outside of your projects folder, Also you might need to kill a ton of processes)
Samitha Jayaweera retweeted
We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case).
I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves.
We're making it easy to fine tune models internally for our employees. We even have a skill for it in our internal harness that makes it easy to spin these up and start training them:
"/fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started"
I imagine we'll have some backend services that could be replaced entirely by this new architecture.
Samitha Jayaweera retweeted
Try Underdog: free, capable, private AI running locally on your device
New build and Saluki 27B are out. Under 8GB, beats Qwen 3.8 27B at tool calling
Like, repost, and comment one task you wouldn’t give a cloud agent to get an invite🦴
Today we're releasing Underdog Saluki 27B
Based on Qwen 3.8 27B, Saluki is almost 7x smaller than its full-precision counterpart while it keeps 96% of its benchmark performance and beats Qwen at tool calling
It's the best model <8GB that runs in Underdog
Saluki is available today under Apache 2.0. Try today in Underdog underdog.ai/saluki
Samitha Jayaweera retweeted
realising MLX-LM-LoRA version v5.8.6 🚀
"pip install -U mlx-lm-lora"
github.com/Goekdeniz-Guelmez…
New training algorithms added:
- KLPO: KL-Regularized Policy Optimization for Critic-Free Agentic Reinforcement Learning by @yifanzhang_
- DSLA: Directional and Similarity-aware Latent Alignment by me
Updated:
- Online training backend version 2: extreme speed increase in rollout generation and general. GRPO for example have 40% to 60% speed increase and 10% to 25% less RAM usage thanks to MLX's (by tpegolotti) new VJPs.
- Offline training: general increases in training speed and RAM usage too.
More in the comments!
Give it a try and let me know what you think 😁
Samitha Jayaweera retweeted
In light of the latest Anthropic's latest not-particularly-oblique attempt to build a case against open weights, I thought I would write down how I currently think about open weight AI models.
(1) Open weights have been deeply, irreplaceably useful for AI safety.
Samitha Jayaweera retweeted
1/ Introducing Cua Perception: Cua Driver can now see.
Nothing to click? It reads the pixels. Canvases, games, remote desktops and custom-drawn apps become labelled regions an agent can act on, parsed locally on the CPU.
github.com/trycua/cua
Samitha Jayaweera retweeted
1/ Fast Computer Use is now solved with @typesafeai Jev + Cua Driver.
Available in development preview for macOS, Windows, and Linux. We call it jev-use.
Draft #3943: github.com/trycua/cua
Samitha Jayaweera retweeted
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
Samitha Jayaweera retweeted
new course just dropped: CS 312: Deep Learning Alchemy w/ @stanfordnlp !
A very interesting new class concept from @tatsu_hashimoto :
- Assignments are worth 0% of the grade
- 85% of your course grade is 40-minute quizzes where you get a code diff for an experiment you've never run and ask you to predict validation loss differences
- The premise? that prediction == understanding. If we don't have any intuition as to the outcomes of an experiment, then it's often safe to say we don't understand it (or at least, it means we don't have beyond surface level knowledge).
And also, everything is public (in the style of cs336!)
(Answer to the question below in, and links 🧵)
Samitha Jayaweera retweeted
The VRAM barrier is officially dead.
I just ran Qwen 3.8 Flash Next (MoE) 125B A6B with a 250,000 context window on a single 24GB RTX 4090.
21 tokens/sec decode. 364 t/s prefill.
no mtp. no dflash. no kv cache quantization!
We are running datacenter models on consumer hardware.
Tested on Ubuntu 22 | CUDA 13.0 | PCIe 4.0 x16 | 110 GB DDR4 System RAM with a continuous 28k prompt across all runs.
### The Benchmarks & Scaling
# 1. Hybrid Offload (-ncmoe 40 @ 80k Context)
Offloaded 40 expert layers to the GPU, pushing VRAM to the ceiling.
./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 80000 --port 8080 -v --fit off -b 4096 -ub 4096 -ncmoe 40
Prefill: 383.85 t/s | Decode: 22.52 t/s
Footprint: 23.85 GB VRAM | 97 GB RAM
# 2. Full CPU MoE Offload (-cmoe @ 80k Context)
Pinned all 512 expert layers to DDR4 RAM (-cmoe), keeping attention on the 4090.
llama.cpp flags: (Same as above, replace -ncmoe 40 with -cmoe)
Prefill: 355.72 t/s | Decode: 20.84 t/s
Footprint: 11.66 GB VRAM (12GB+ VRAM freed up!) | 110 GB RAM
# 3. The 180,000 Context Run
Prefill: 357.75 t/s | Decode: 20.98 t/s | VRAM: 15.6 GB | RAM: 110 GB
# 4. The 250,000 Context Absolute Ceiling
./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 250000 --port 8080 -v --fit off -b 4096 -ub 4096 -cmoe
Prefill: 364.29 t/s | Decode: 20.97 t/s
Footprint: 18.3 GB VRAM (Still ~5.7 GB of VRAM headroom!) | 110 GB RAM
### Key Insights:
-b 4096 -ub 4096: doubles the prompt ingestion from ~150 to 364+ t/s.
-cmoe Free Lunch: Shifting expert layers to DDR4 RAM slashes VRAM from 24GB to 11.6GB with virtually zero decode penalty (22.5 -> 20.9 t/s), enabling the 250k context ceiling.
Qwen 3.8 Flash-Next (UD-Q4_K_XL) is a massive 111.4 GB model split across 4 shards. To run this architecture, you must build from the experimental PR branch (#27742) by @danielhanchen:
git clone && cd llama.cpp
git fetch origin pull/27742/head:qwen-next && git checkout qwen-next
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j $(nproc) --target llama-server
A single 4090 paired with 100 GB of cheap DDR4 RAM will comfortably serve production grade 125B inference.
While Qwen 3.8 27B (dense) still holds the crown for single 3090/4090 rigs, Flash Next proves 125B hybrid models are officially viable on consumer hardware.
Hugging Face GGUF link and complete performance telemetry graphs are dropped in the replies below.
GLM 5.3 Flash VS Qwen 3.8 Flash Next, which one takes the open weights crown this week?
Qwen3.8-Flash can now be run locally! 🔥
The 125B MoE model outperforms Claude-Opus-4.6 (Max).
Run on 75GB RAM via Unsloth GGUFs.
Qwen3.8-Flash-Next enables CPU RAM / unified mem setups to deliver near VRAM speeds.
Guide: unsloth.ai/docs/models/qwen3…
GGUF: huggingface.co/unsloth/Qwen3…
Samitha Jayaweera retweeted
Claude has become a language, so I built a translator.
English <-> Claudish
Samitha Jayaweera retweeted
Inference Engineering is free for everyone to read:
baseten.co/inference-enginee…
Samitha Jayaweera retweeted
When I worked at Apple, I remember seeing code that was 15 years old. Every file had a comment at the top with the date and the author’s name. You’d open a file and read through it, sometimes touching a feature no one had touched in a decade.
Everything felt very permanent, like a house made of bricks. Everyone who made a brick put their name on it before slotting it into the house. Even in a massive company, you were proud of your bricks. Each one fit so satisfyingly into the bricks around it.
When someone was renovating a room you had worked on years ago, they’d reach out and say, “I found this brick with your name on it. Can you tell me what it was doing here?”
Sometimes old bricks would get taken out during a renovation, but they were usually kept in a pile in the corner to reuse in the future.
Now you can will whole houses and neighborhoods into existence. You don’t even see the bricks go in. They just tumble in from the sky. You walk into the new house and wave your hands to push walls back, throw up a chimney, or drop in a kitchen.
It's so fucking cool. But you forget what a brick feels like.
And it’s hard to know how sad to be about that. The houses are better, they go up faster, and building them increasingly feels like magic.
Still, every once in a while you miss holding a brick.
Good moment today to remind everyone to stay frosty with AI. I was working on making a server stoppable via the CLI, and Codex decided the best path forward was to implement an unauthenticated network API to every server to shut itself down. Excellent work.
I responded with "I don't know about that." (I usually give more direct feedback but this was so absurd) and it thought for like 20 seconds and responded "Server stopping should not be part of an unauthenticated network API." Yes. lol.
By the way, the result I was looking for was that we automatically start a locally discovery Unix-socket bound server so I wanted it to discover that (using prior mechanisms already made) and gracefully stop that.
The driver is absolutely accountable here (me). My lazy prompting led AI to think I wanted to be able to stop ANY configured server I was talking to. Shame on me for thinking it was obvious here. But, I also reviewed it and found this idiocy, so, that's where good AI drivers come in. Also me.
Samitha Jayaweera retweeted
DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro.
⚡ Up to 4.6× the speed of autoregressive decoding, with the same output.
This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
inco.ai/blog/dflash2/
