Research Scientist at Toyota Research Institute. Ph.D. student at @rllabmcgill, @MILAMontreal. Prev: FAIR, MSR, BorealisAI, Amazon, UF.
Seattle, WA
Joined May 2009
- Tweets439
- Following1.2K
- Followers428
- Likes570
Kushal Arora retweeted
Our white paper just came out on Arxiv arxiv.org/abs/2604.19728v1. We open-sourced it all github.com/TRI-ML/vla_foundr…. Our project website also has links to the white paper, the weights, and more videos tri-ml.github.io/vla_foundry
Kushal Arora retweeted
Releasing VLA Foundry: an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. End-to-end control from language pretraining to action-expert fine-tuning — no more stitching together incompatible repos.
Kushal Arora retweeted
ReFiNe expands on a neat idea we first presented at CoRL with Recursive Octree Auto-Decoders: that recursion can enable very high compression rates of 3D data. In ReFiNe, we use this property to represent continuous fields and can decode multiple NeRFs/SDFs with a single network.
Excited to introduce our paper, ReFiNe, at #SIGGRAPH2024 this Thursday! Learn how we encode multiple assets as continuous neural fields with high precision & low memory usage by exploiting object self-similarity.
@RaresAmbrus @robo_kat @adnothing
Webpage: zakharos.github.io/projects/…
Kushal Arora retweeted
Excited to introduce our paper, ReFiNe, at #SIGGRAPH2024 this Thursday! Learn how we encode multiple assets as continuous neural fields with high precision & low memory usage by exploiting object self-similarity.
@RaresAmbrus @robo_kat @adnothing
Webpage: zakharos.github.io/projects/…
Kushal Arora retweeted
Building language models is difficult and requires high quality preprocessing, modeling, evaluation and large scale training.
As significant collaborators in this project at TRI, the resulting 7B model DCLM-7B is a significant achievement. It is a competitor to Mistral 7B and LLaMA-7B, even though trained on less data. And it’s fully open. And that’s just the start of the competition.
Excited to see how others leverage these results to build even more capable language models and improve dataset quality.
One thing I have come to greatly appreciate over the last year is the role of data filtering in building SOTA language models.
DCLM introduces a filtered 240T dataset, a 7B open-source model that is competitive w/ Llama3 with 2-6x fewer tokens & a pipeline to build new datasets.
I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x
1
6
740
For more details, see the paper arxiv.org/abs/2406.11794v1, and the website: datacomp.ai/dclm/.
2
67
Kushal Arora retweeted
DataComp-LM: In search of the next generation of training sets for language models
- Provides a corpus of 240T tokens from Common Crawl
- Trains a LM using their filtered dataset, which performs similarly on NLU tasks w/ 6.6x less compute than Llama 3 8B
proj: datacomp.ai/dclm/
abs: arxiv.org/abs/2406.11794
Kushal Arora retweeted
Check out DataComp for language models! Open data, open code, open training recipe, and close to Llama3-8B performance. This has been a labor of love over the last year, a huge thanks to all the collaborators for helping make this happen!
Kushal Arora retweeted
I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x
Sedrick is an amazing researcher and has done amazing work on pre-training, scaling, evaluation, Japanese LMs, code models, VLMs, and more in the last year. If you are at NAACL, do get a coffee with him!
I'm attending #NAACL2024 at Mexico City this week! Excited to chat about pre-training, evaluation, and multimodality!
(also excited for🌮🌯🫔)
Kushal Arora retweeted
Recurrent models like RWKV and Mamba have gained attention recently, but these can be costly to train and iterate on.
What if we could simply...
turn Mistral/Llama/Gemma into an RNN? 🎩🪄
Presenting our work, Linearizing Large Language Models!
arxiv.org/abs/2405.06640
Kushal Arora retweeted
2024 has seen tons of cool work on RNNs (cc @RWKV_AI, @BlinkDL_AI, @GoogleDeepMind Griffin, @AI21Labs). We hope our work helps further research into linear models!
Work done at @ToyotaResearch with @MercatJean @vslevic @sedrickkeh2 @karora4u @achalddave @adnothing @tkollar
A really large in-the-wild robotics dataset from TRI colleagues And university partners, a major step in the direction of building Robotics Foundation Model.
After two years, it is my pleasure to introduce “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset”
DROID is the most diverse robotic interaction dataset ever released, including 385 hours of data collected across 564 diverse scenes in real-world households and offices
3
234
RLAIF is all the rage right now, but does just adopting the RLHF pipeline for RLAIF work?
In this work led by @archit_sharma97, we show that a lot of gains by RLAIF models are the artifact of the weak SFT data from GPT-3.5. Once updated, these advantages nearly disappear.
🧵👇
ML Internship Opportunity at @ToyotaResearch (TRI)
The language team at TRI is looking for interns for this summer. We are focused primarily on open-ended research in model scaling, code generation, datasets, efficient NLP, tokenization, RL/NLP, and alignment.
At TRI, we strongly believe in open research and open source.
Interns @ TRI will lead their own projects and have the opportunity to contribute to open source with a solid possibility for publicly releasing code, models, and datasets.
Interested?
jobs.lever.co/tri/21ac8e76-6…