everything GPU @mixedbreadai | Research, ColBERT, DSPy @Stanford
Stanford
Joined December 2013
- Tweets1.9K
- Following548
- Followers2.5K
- Likes8.7K
Herumb Shandilya 🦀 retweeted
And if you think that’s even close to a ceiling, stay tuned
Herumb Shandilya 🦀 retweeted
By now, everyone knows that single-vector embedding models are hugely limiting for modern workflows.
But they contain than you think: you can extract sparse Latent Terms from them.
And it turns out that BM25 is all you need to turn this vocabulary into a strong retriever.
Herumb Shandilya 🦀 retweeted
The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts.
We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts.
Today we're releasing @OpenJarvisAI v1.0: a personal AI assistant that lives, learns, and works on your device.
Herumb Shandilya 🦀 retweeted
New: grep for exact matching
grep → keyword / regex matching
search → fine-grained semantic retrieval
Works across uploaded content, including text, PDFs (OCR) and audio/video (transcription).
Give your agents both retrieval primitives to perform at their best.
Herumb Shandilya 🦀 retweeted
Feature: Native agentic search on Mixedbread
Search with auto-planning, exploration, and multi-hop reasoning across documents.
Built for:
- evidence discovery
- exhaustive search
- cross-document reasoning
→ Topped MADQA @snowflake with 93.4% accuracy across 18,000 PDF pages.
Herumb Shandilya 🦀 retweeted
Introducing mxbai-rerank-v3-listwise: reranking that goes beyond binary relevance.
It reads the whole candidate set, resolves conflicts, and ranks by directives like recency, source priority, and multi-step rules.
+11% NDCG@10 on average across multiple domains, modalities, and languages in runs with Wholembed v3.
Available today in preview in Mixedbread.
Herumb Shandilya 🦀 retweeted
Mixedbread just made 115h of videos accessible to my agent.
With the new @mixedbreadai v3 release, you can upload any video to your Mixedbread store and make its content accessible to your agent.
Herumb Shandilya 🦀 retweeted
Introducing Mixedbread Wholembed v3, our new SOTA retrieval model across all modalities and 100+ languages.
Wholembed v3 brings best-in-class search to text, audio, images, PDFs, videos...
You can now get the best retrieval performance on your data, no matter its format.
Herumb Shandilya 🦀 retweeted
With intelligence-per-watt (IPW), we propose a unified metric for measuring intelligence efficiency, capturing both the LM capabilities delivered and the energy required to power the AI stack, enabling a better understanding of how we scale local and cloud LLMs.
Honored to be part of Slingshots // TWO! It's been a blast working with @LaudeInstitute on the IPW project. Big thanks to @andykonwinski @bradenjhancock @ChrisRytting and the whole Laude team for all the support!
Intelligence-Per-Watt/@JonSaadFalcon @Avanika15 John Hennessy @hazyresearch @Azaliamirh (@Stanford) - Most queries don't need frontier-model horsepower. This work makes "use the right model for the job" a measurable strategy, quantifying when smaller local models can match frontier quality while cutting energy, cost, and compute.
Herumb Shandilya 🦀 retweeted
I went through both the blog and the code for Observational Memory, and really interesting direction, and I appreciate the transparency in sharing implementation details. I do have a few thoughts on the SOTA memory claim and the broader framing.
From what I can see:
1. The implementation appears heavily tuned toward performing well on LongMemEval. That's a useful signal, but it doesn't necessarily translate to robust long-term memory behavior in production environments.
2. It feels closer to context compression/context management than a durable long-term agent memory system. This will perform really well for a single long-running task
3. Both the Observer and Reflector rewrite memory in compressed form. That's helpful for token control, but compression is inherently lossy and can drop smaller details that might become important later.
4. The Reflector seems to validate success primarily via token thresholds, rather than checking whether the rewritten memory remains semantically faithful to the original. Over time, this could allow memory drift.
5. The Observer prompt may introduce assumptions (e.g., inferring that a planned action happened if enough time has passed), which risks creating incorrect memories.
6. The design appears to emphasize recency when rewriting observations. While that keeps context fresh, it may bias the system toward recent information and gradually compress away older but still important details. Durable memory systems usually need mechanisms to preserve salient long-term facts, not just recent activity.
7. The full observations block is repeatedly injected into context. This may increase token cost and introduce irrelevant noise depending on the task.
8. There appears to be limited grounding back to raw message evidence at response time, which makes it harder to detect and correct incorrect compressed memories.
9. Finally, I think we should be cautious about claiming "SOTA" based on performance on a single benchmark. LongMemEval results may demonstrate strong performance on that setup, but production workloads are much messier. Robustness, drift, grounding, and cost behavior typically show up only under sustained real-world usage.
Overall, this looks like a strong benchmark-oriented context handling. I am just less convinced that it yet qualifies as a robust, general-purpose long-term memory system. Curious how the team is thinking about these trade-offs beyond benchmark performance.
🚨 Announcing a new SOTA memory system, Observational Memory (OM), available in latest @mastra version now.
It achieves the highest scores ever recorded on LongMemEval (gpt-4o 84.2%, gpt-5-mini 94.9%)
No RAG, no graphs, no input based retrieval, just a simple constantly evolving text blob that outperforms every other memory system available. 🧵1/4
Herumb Shandilya 🦀 retweeted
New updates for the RLM paper:
We post-trained RLM-Qwen3-8B at tiny scale, the first natively recursive LM.
And it's pretty promising, even on completely unseen types of tasks! We just released the model too.
Some quick thoughts here:
Training a model to be good at long-context processing is known to be really hard. How can such a tiny model learn to be a decent RLM in general, from very little data?
We think part of the answer is that RLMs are extremely modular.
To learn to be a root RLM, the model doesn't need subtle changes to how it uses attention or something. It just needs to recognize the programmatic format of interacting with prompts-as-variables and to bake in good strategies for writing little recursive programs.
That's... not even as tricky as post-training models for solving hard math problems.
Once the sub-calls are launched, they're ~pretty much normal LLM calls, so no special training is particularly necessary (though we're sure it might help).
Obviously, this is just meant as a simple PoC that shows that tiny models can be good RLMs.
We call on future work to train native RLMs at much larger scales in terms of model size, number and variety of examples, and number of (ideally on-policy and online!) rollouts.
We just updated the RLM paper with some new stuff.
First, we just released RLM-Qwen3-8B, the first natively recursive language model (at tiny scale!).
We post-trained Qwen3-8B using only ~1000 RLM trajectories from unrelated domains to our evaluation benchmarks.
RLM-Qwen3-8B works well across several tasks and delivers a pretty large boost over using an RLM scaffold with the underlying Qwen3-8B model off-the-shelf, and even larger gains over directly using Qwen3-8B directly for long-context problems.
Herumb Shandilya 🦀 retweeted
The U.S.–China AI race won’t be decided by who builds the most datacenters, but by who deploys the most intelligence.
We call this Gross Domestic Intelligence (GDI): intelligence per watt × usable power.
If the U.S. activates its dense installed base of local AI accelerators in a hybrid local–cloud system, it could add ~30–40% inference capacity and ≈2-4× GDI for single-turn chat and reasoning queries without building any new datacenters or grid infrastructure.
Winning the GDI race means treating local compute as critical infrastructure and making hybrid inference the default.
(1/N)
Herumb Shandilya 🦀 retweeted
Introducing SpecStory Cloud🎉
We now build faster than ever.
With Copilot, Cursor, Claude Code, Codex, and other AI tools, work that once took days now happens in minutes.
But in all that speed, the context behind our work gets lost.
The reasoning, the decision logs, the learnings, and the “why” behind the code disappear the moment the chat closes.
SpecStory Cloud brings that context back:
- Connect all your AI coding sessions across tools and time, in one place where every idea, fix, and decision is remembered.
- Sync, search, and chat with your conversation histories.
- Share your prompts and sessions with teammates.
- Private by design. Sync only when you choose.
Watch the demo below 👇
Herumb Shandilya 🦀 retweeted
Data centers dominate AI, but they're hitting physical limits. What if the future of AI isn't just bigger data centers, but local intelligence in our hands?
The viability of local AI depends on intelligence efficiency. To measure this, we propose intelligence per watt (IPW): intelligence delivered (capabilities) per unit of power consumed (efficiency).
Today’s Local LMs already handle 88.7% of single-turn chat and reasoning queries, with local IPW improving 5.3× in 2 years—driven by better models (3.2×) and better accelerators (1.7×).
As local IPW improves, a meaningful fraction of workloads can shift from centralized infrastructure to local compute, with IPW serving as the critical metric for tracking this transition.
(1/N)
Herumb Shandilya 🦀 retweeted
Here's a gem for the job market: I for one would love to see Herumb @krypticmouse at places like @modal or @cerebras or any of the other cool places that are building the next generation of AI & ML training/inference infra. When Herumb puts his systems programming hat on, he's a master of just *getting things working*. I hope the right company finds him before he's off the market. 💯 highly recommend hiring him 🤗
I just learned that the one-of-a-kind Herumb Shandilya @krypticmouse, who is soon to graduate from Stanford with Master’s in CS, is on the job market for training infra, inference infra, or other ML engineering roles.
Here’s a PSA to y’all: you should hire him.
Herumb Shandilya 🦀 retweeted
I just learned that the one-of-a-kind Herumb Shandilya @krypticmouse, who is soon to graduate from Stanford with Master’s in CS, is on the job market for training infra, inference infra, or other ML engineering roles.
Here’s a PSA to y’all: you should hire him.
Herumb Shandilya 🦀 retweeted
FFmpeg makes extensive use of hand-written assembly code for huge (10-50x) speed increases and so we are providing assembly lessons to teach a new generation of assembly language programmers.
Learn more here:
github.com/FFmpeg/asm-lesson…
Herumb Shandilya 🦀 retweeted
Testing out the DSPy Rust-offspring DSRs right now. Inference is already working nicely, I just need to get the GEPA optimization working now!
