@_rishabhranjan_

Stanford CS PhD w @jure and @guestrin. Prev. CMU w @zacharylipton, IIT Delhi. Deep learning fanatic.

 Stanford, CA
rishabh-ranjan.github.io
Joined March 2022
  • Tweets88
  • Following235
  • Followers313
10Media
Our new work, TokenGate, brings MoE-like conditional computation to context tokens, resulting in up to ⚡️11x⚡️ faster prefill and ⚡️3x⚡️ lower memory!
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
2
8
500
end-to-end learning ftw 🙌 as always 😌
Liana@lianapatel_22h
Replying to @lianapatel_
(7/n) Why do TokenGated LLMs win? 🏆🏅 TokenGate flexibly learns different token-selection patterns in a task-dependent manner. eg) it keeps answer keywords for QA and stop words at the end of phrases for summarization. And the backbone co-adapts with the gate, becoming robust to pruning and re-distributing information among tokens. Both effects are possible because TokenGate enables e2e learning.
2
156
rishabh ranjan retweeted
rishabh ranjan@_rishabhranjan_17h
Replying to @kartikb753 @lianapatel_
TokenGate is extremely good at needle style tasks. Please see our NIAH (needle-in-a-haystack) results! The secret is that no token is lost completely, the inital layers can merge its info into the kept tokens!
1
1
71
Very interesting work on pruning input tokens! It’s similar to how we do latent tokenization in hierarchical models, except here the unnecessary tokens are dropped instead of being mean-pooled.
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
3
14
846
rishabh ranjan retweeted
Vignesh Kothapalli
@kvignesh142015h
The bitter lesson of scaling applies when systems are learnable end-end. Kudos to @lianapatel_ and @_rishabhranjan_ for bringing it down to the token selection stage of a forward pass. Paper: arxiv.org/abs/2610.26754

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill...

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens...

arxiv.org
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
4
8
748
rishabh ranjan retweeted
Stanford AI Lab
@StanfordAILab18h
Token-Gated LLMs achieve 11x faster processing over long-context inputs by learning to skim context and only forward tokens that matter. We saw this paradigm shift before when going from dense models to MoEs, and now TokenGate enables another generation of efficient LLMs!
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
9
12
90
8,015
rishabh ranjan retweeted
Connor Shorten
@CShorten3022h
Replying to @lianapatel_
Game changing! Congratulations Liana and team! 🔥
1
1
4
303
TokenGate follows on the footsteps of the Mixture-of-Experts layer: arxiv.org/abs/1701.06538 Hat-tip to @NoamShazeer @Azaliamirh @MaziarzKris @quocleix @geoffreyhinton @JeffDean for showing how to scale up model capacity with conditional computation and revolutionizing the field 🙇

Outrageously Large Neural Networks: The Sparsely-Gated...

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been...

arxiv.org
1
2
6
304
Also indebted to works like Switch transformer (@LiamFedus @barret_zoph), Mixtral (@AlbertQJiang @arthurmensch @dchaplot @sophiamyang @Fluke_Ellington @theo_gervet) and DeepSeekMoE (@deepseek_ai) which kept the flame alive ❤️‍🔥
1
4
127
rishabh ranjan retweeted
Shina Patel, MD@shina181220h
Replying to @lianapatel_
Conditional computing over context tokens is fire 🔥
1
1
2
221
rishabh ranjan retweeted
rishabh ranjan@_rishabhranjan_16h
Our new work, TokenGate, brings MoE-like conditional computation to context tokens, resulting in up to ⚡️11x⚡️ faster prefill and ⚡️3x⚡️ lower memory!
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
2
8
500
rishabh ranjan retweeted
rishabh ranjan@_rishabhranjan_16h
TokenGate follows on the footsteps of the Mixture-of-Experts layer: arxiv.org/abs/1701.06538 Hat-tip to @NoamShazeer @Azaliamirh @MaziarzKris @quocleix @geoffreyhinton @JeffDean for showing how to scale up model capacity with conditional computation and revolutionizing the field 🙇

Outrageously Large Neural Networks: The Sparsely-Gated...

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been...

arxiv.org
1
2
6
304
rishabh ranjan retweeted
Liana@lianapatel_20h
very cool to see TokenGate already trending on @askalphaxiv — and only a few hours since our release 👀 🚀super excited for this work. check it out and share your questions! (links in comments)
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
1
9
30
2,726
rishabh ranjan retweeted
Liana@lianapatel_22h
Introducing ✨Token-Gated Language Models✨ for up to 11x faster prefill by learning to skim context, conditionally forwarding only tokens that matter. Akin to the shift from dense models to MoEs, the next frontier of efficient LLMs is conditional computation over a new scaling axis: context tokens🧵👇
24
53
11
300
29,461
rishabh ranjan retweeted
Daman Arora
@amuseddamanOct 16
1/ RL for improving task success has made LLMs remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky! Very excited to share our new paper: Training Language Models to Fail Gracefully 🧵
24
89
18
477
56,969
rishabh ranjan retweeted
Jure Leskovec
@jureSep 29
I’m very excited to share @nvidia Kumo Tabular, a new family of foundation models for tabular data. Kumo Tabular establishes the new Pareto frontier across the entire accuracy–inference-time tradeoff. Just as importantly, we are releasing it openly: open weights, open-source software, and a permissive license for commercial use. HuggingFace: huggingface.co/nvidia/Kumo-T… GitHub: github.com/NVIDIA/structured…
32
155
34
1,102
128,219
rishabh ranjan retweeted
Liana@lianapatel_Aug 31
Looking forward to VLDB in Boston this week! I’ll be on the job market this year for CS faculty positions. I’m also co-organizing the NOVAS workshop tomorrow, featuring research at the intersection of data systems and AI, including semantic operator optimization and agentic data systems. Please reach out if you’re attending—I’d love to meet and chat!
1
5
45
5,676
Exciting to see Relational Transformers at the YC startup school: star-project.stanford.edu/ta…

Zero-Shot Predictive Models for Relational Databases

Mark Žnidar · YC ML Meetup at Startup School

star-project.stanford.edu
1
16
904
rishabh ranjan retweeted
Frank Hutter
@FrankRHutterAug 18
Super excited about our first work on relational learning, continuing to push on open source!
Prior Labs
@prior_labsAug 18
We’re happy to announce our first release in relational learning at Prior Labs, continuing our commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful, real-world impact. First and foremost, we release 𝗥𝗲𝗹𝗔𝗿𝗲𝗻𝗮-α: a unified framework for running and comparing baselines on RelBench v1 tasks. Based on learnings from tabular benchmarks like TabArena, we are standardizing data loading, evaluation protocols, tuning regimes, and adding support for systems with custom tuning. We also open-source 𝗧𝗮𝗯𝗣𝗙𝗡-𝗥𝗲𝗹: our relational harness for TabPFN-3. We initialize the (living) RelArena-α leaderboard with TabPFN-Rel and a comprehensive set of baselines. The rankings at the time of release are: • 𝗧𝗮𝗯𝗣𝗙𝗡-𝗥𝗲𝗹 is the No. 1 model submission • 𝗥𝗧-𝗣𝗹𝘂𝗥𝗲𝗹 is the No. 1 system submission Last but not least, we open-source an alpha version of the 𝗥𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲 (𝗥𝗣𝗜): enabling you to easily specify prediction tasks on your own relational database and then run any RelArena-α model, like TabPFN-Rel, in a few lines of code, all bundled as a simple PyPI package. • Read the full model report: arxiv.org/abs/2608.16319 • GitHub repository (give us a ⭐️): github.com/PriorLabs/relaren… • Announcement: priorlabs.ai/blog-posts/intr… • Docs: docs.priorlabs.ai/capabiliti… Thanks to the contributions from: @adrihayler, @KNfloege, @AlanArazi1536, @_rishabhranjan_, @jure, @LennartPurucker, @FrankRHutter & @noahholl
1
4
23
1,858
rishabh ranjan retweeted
Jure Leskovec
@jureAug 19
Excited to see this out! Relational learning needs strong benchmarks, standardized evaluation, and easy ways to bring methods to real-world relational data.
Prior Labs
@prior_labsAug 18
We’re happy to announce our first release in relational learning at Prior Labs, continuing our commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful, real-world impact. First and foremost, we release 𝗥𝗲𝗹𝗔𝗿𝗲𝗻𝗮-α: a unified framework for running and comparing baselines on RelBench v1 tasks. Based on learnings from tabular benchmarks like TabArena, we are standardizing data loading, evaluation protocols, tuning regimes, and adding support for systems with custom tuning. We also open-source 𝗧𝗮𝗯𝗣𝗙𝗡-𝗥𝗲𝗹: our relational harness for TabPFN-3. We initialize the (living) RelArena-α leaderboard with TabPFN-Rel and a comprehensive set of baselines. The rankings at the time of release are: • 𝗧𝗮𝗯𝗣𝗙𝗡-𝗥𝗲𝗹 is the No. 1 model submission • 𝗥𝗧-𝗣𝗹𝘂𝗥𝗲𝗹 is the No. 1 system submission Last but not least, we open-source an alpha version of the 𝗥𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲 (𝗥𝗣𝗜): enabling you to easily specify prediction tasks on your own relational database and then run any RelArena-α model, like TabPFN-Rel, in a few lines of code, all bundled as a simple PyPI package. • Read the full model report: arxiv.org/abs/2608.16319 • GitHub repository (give us a ⭐️): github.com/PriorLabs/relaren… • Announcement: priorlabs.ai/blog-posts/intr… • Docs: docs.priorlabs.ai/capabiliti… Thanks to the contributions from: @adrihayler, @KNfloege, @AlanArazi1536, @_rishabhranjan_, @jure, @LennartPurucker, @FrankRHutter & @noahholl
2
1
26
7,899
Load more