Cambridge, MA
Joined December 2022
- Tweets14.5K
- Following3.6K
- Followers39K
- Likes38K
Omar Khattab retweeted
I think I spy something new on there
Omar Khattab retweeted
The wonderful “Don’t just Fix it in Post” paper identifies predicting memorization as a key open problem.
A progress report: we can now accurately predict verbatim memorization using just three well-understood variables. Experiments by Jing Huang, using the Hubble model suite:
ALT Dot plot addressing whether memorized sequences can be predicted without training the full model. Headline: “Largely yes, with Frequency and Recency doing most of the work.” Four rows show AUC-ROC (0.5 = chance, 1.0 = perfect) for a linear predictor of verbatim extraction, one mark per Hubble model (1B or 8B params; 100B or 500B tokens). All three features: 0.82–0.94. Frequency alone: 0.81–0.93. Recency alone: 0.82–0.90. Hardness alone: 0.51–0.55, barely above chance. In every row the 1B/500B model scores lowest and the 1B/100B model highest. Footnote: held-out strings, mean of 5 seeds; Frequency = copies of the string, Recency = steps since last copy, Hardness = NLL on models that never saw it.
Omar Khattab retweeted
you can now run perplexity's decider model locally on your mac. it's private, fast, and free, with no api key needed. pplx.ai/decider-for-mac
decider for mac runs pplx-decider-flash-v1.2, our 4B decision model, fully on device. ask a yes/no question about text or an image and get a probability back. after a one-time 5.8 GB download, it serves the decisions api at localhost:8080/v1/decisions.
use it to label files, route messages, check screenshots, or run pre-commit checks, without any data leaving your mac.
confession: i've yet to have that moment of feeling that frontier LLMs are geniuses, or super-human at ~anything broadly coherent that i deeply prize
i'm not sure what it is that my work needs or what i'm presumably 'good' at, but despite extremely elaborate (& happy!) use and having worked on nothing but language models for essentially my entire career, i can only see that kind of vibe abstractly from others' (e.g., mathematicians) reactions, not in the vicinity of anything that i need or do
they're the most impressive tools i've ever used, and i essentially use my computer mostly from them now, but yeah, not yet
anyway, back to cc and codex cuz we have work to do
Omar Khattab retweeted
It's like self-supervised learning at the higher abstraction of research contributions instead of words: take a naturally occurring thing, mask out a part, see if the AI can recover it. What's the next level up of abstraction? "Pretend we know nothing about the field of acoustics; what can you recover?"
Congrats to @MangQiuyang, @lihanc02, and team. Another @LaudeInstitute Slingshot we're proud to have supported.
Omar Khattab retweeted
On this note, if you've ever thought "Man, I wish I had BEIR and NanoBEIR, but considerably better and very discriminative of model quality!", you might also want to stay tuned.
i hear it might also be fully open
Omar Khattab retweeted
extremely excited for our upcoming releases. retrievers aren't even vaguely close to anything resembling a ceiling
Omar Khattab retweeted
We've released a technical report on the released late interaction models
It's purposely kept short, simple and direct, because most of the recipe is not new (shared space is not really new as well, but pretty cool for ColBERT still)
There are still a lot of technical information if you are interested beyond the BP
We are preparing much more thorough papers for things where we did really bring a lot of novelty, pretty sure you'll enjoy those!
arxiv.org/pdf/2610.29523v1
Omar Khattab retweeted
Save 50%, only on Oct 21st!
Today, you can save on Building LLM Applications with DSPy by Serj Smorodinsky and Brett Kennedy — now in print — and other related titles: hubs.la/Q04BMk270
Learn how to build DSPy into fully agentic pipelines, integrating conversation memories, and connecting agents to external environments via MCP.
Omar Khattab retweeted
applications for slingshots more than doubled from batch 3 to batch 4.
funding open science is critical to democracy.
never in the past have scientists had to turn down so much money to do research they can talk about openly. top PhD students are receiving offers of millions of dollars per year from closed labs.
i'm proud that at Laude we support open research.
great work on slingshots batch four @bradenhancock and the @laudeinstitute team
Omar Khattab retweeted
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Omar Khattab retweeted
Incredibly grateful to @LaudeInstitute for supporting our work on WaggleHaus!
TL;DR: We believe that supporting agentic queries over massive data lakes requires rebuilding the data system from the ground up.
This includes ideas ranging from:
- making the data layer more "agent-friendly"
- to specializing the inference engine for the query workload
- to running agents within the system that continuously update views and indices for faster data access
If you're interested in a demo or becoming an industry design partner you can sign up here: wagglehaus.ai
WaggleHaus // @RussoMatthew, @tianyu_li_, @MikeCafarella, @tim_kraska, @samrmadden (@MIT, @UWMadison) — A query engine for executing semantic and agentic queries over enterprise-scale data lakes.
Omar Khattab retweeted
Check out our latest research: MemCurrent, a persistent agent memory architecture, which consistently stays on the cost-accuracy Pareto frontier on our internal and three public benchmarks. Try it in the new Agent Bricks Memory and Genie 🚀
Introducing MemCurrent, a persistent memory architecture designed to help production agents stop starting from zero.
MemCurrent combines episodic memory with derived memory across three access patterns:
• Plugins to log and load context automatically
• Tools the agent calls on demand
• An offline “dreaming” pipeline that consolidates memory in the background
Agents carry forward user preferences, prior corrections, metric definitions and successful workflows instead of rediscovering them every session.
On our internal benchmark built from real enterprise data, MemCurrent triples the agent's pass rate while using 31% fewer tokens and running 26% faster. It's built into Agent Bricks' Memory offering and already powers memory in Genie.
Read the research: databricks.com/blog/memcurre…
Omar Khattab retweeted
ScholarCatalyst is part of Slingshots Four!
Deeply “reading” the literature, i.e. knowing which earlier idea a new problem needs, takes years to get even okay at. Even the best systems do poorly on ScholarCatalyst. With @LaudeInstitute's support, we’ll train directly for this.
ScholarCatalyst // @yoonholeee, @ohmyksh, @chelseabfinn (@Stanford) — A benchmark for finding prior work that could inspire scientific research, and a retrieval system trained for this task.
Omar Khattab retweeted
Pedagogical RL // @SOURADIPCHAKR18, @NoahZiems, @amritsinghbedi3, @lateinteraction (@MIT, @LaudeInstitute) — A reinforcement learning framework that teaches AI models to generate plausible, successful reasoning trajectories they can learn from for challenging tasks.
Omar Khattab retweeted
Headlong // @NickAJalbert, @bradenjhancock, @NoahZiems, @a1zhang, @lateinteraction, @andykonwinski (@LaudeInstitute, @MIT) — A microharness for persistent agents.
