Head of AI @ Wayve; ex-Research Manager @ TRI; ex-Amazon; ex Apple. CMU / MIT
san jose, ca
Joined October 2008
- Tweets113
- Following525
- Followers324
- Likes287
Thomas Kollar retweeted
Meet LA-Pose. Our latest model taking Wayve another step towards generalization at scale.
LA-Pose employs large-scale self-supervised learning, building strong motion representations for 3D perception from 10.2 million unlabeled driving video snippets, unlike today's strongest approaches that often depend on expensive, carefully curated 3D supervision.
With only a lightweight pose head and limited labelled data, LA-Pose achieves:
π· State-of-the-art camera pose estimation
π Strong zero-shot generalization across diverse driving scenarios
π·οΈ Orders of magnitude less labelled data than fully supervised 3D approaches
Our full blog post: wayve.ai/thinking/la-pose/
Explore the full paper here: la-pose.github.io/
Thomas Kollar retweeted
Nissan and #Wayve have signed a partnership agreement that will bring our next-gen #ProPILOT driver assistance tech powered by Wayve #AI to a broad range of #Nissan vehicles.
Nissan aims to first launch the next-gen tech in Japan in fiscal year 2027.
global.nissannews.com/en/relβ¦
Thomas Kollar retweeted
GAIA 3 introduces four powerful new capabilities that unlock richer and more scalable evaluation of autonomous driving systems. π
𧡠Follow the thread below to see examples of;
1. Long perturb generations π
2. Safety augmentations β οΈ
3. Semantic augmentations π€οΈπ
π
4. Embodiment transfer ππ·
GAIA 3 re-generates the same scenario as if observed from different vehicles with different camera positions. One scene, three embodiments, consistent dynamics. Ideal for testing models across different hardware setups.
These advances show how GAIA-3 brings new realism, diversity, and scale to the evaluation of end-to-end driving systems. π
Dive into the full blog: wayve.ai/thinking/gaia-3/
Every clip you see below is generated by GAIA-3.π
#GAIA3 #EmbodiedAI #AISafety #GenerativeAI #AutonomousVehicles
Thomas Kollar retweeted
It's awesome to be back in the Bay Area this week at @wayve_ai's other North American office.
I can't wait to test the massive progress the team's been making on rides around the Bay Area and city while I'm here, and to meet with our science leaders @vijaycivs @tkollar @gianlucacorrado and others to galvanise the groups at the start of an incredibly exciting #YearOfEmbodiedAI ahead!
#Science #Team #EmbodiedAI
Building language models is difficult and requires high quality preprocessing, modeling, evaluation and large scale training.
As significant collaborators in this project at TRI, the resulting 7B model DCLM-7B is a significant achievement. It is a competitor to Mistral 7B and LLaMA-7B, even though trained on less data. And itβs fully open. And thatβs just the start of the competition.
Excited to see how others leverage these results to build even more capable language models and improve dataset quality.
More details from @achalddave:
xitter.gq/achalddave/status/1803β¦
Check out DataComp for language models! Open data, open code, open training recipe, and close to Llama3-8B performance. This has been a labor of love over the last year, a huge thanks to all the collaborators for helping make this happen!
1
327
Excited to release Prismatic!
Cutting through the noise of vision-language modeling, Prismatic is a release of 42 pre-trained VLMs from the 7B to 13B scale, a codebase for rigorous evaluation and a myriad of insights for what matters for performance.
What design choices matter when developing a visually-conditioned language model (VLM)?
Check out our paper β Prismatic VLMs β and open-source training code, evaluation suite, and 42 pretrained VLMs at the 7B-13B scale!
π arxiv.org/abs/2402.07865
βοΈ + π€ github.com/TRI-ML/prismatic-β¦
![Our investigation of different axes for developing visually-conditioned language models (VLMs) spans four different axes. [Top Left] 1. Optimization procedure - should we freeze model components, or do we need multi-stage training? [Bottom Left] 2. Image processing and visual representations - how should we choose pretrained representations? [Top Right] 3. Language Models - how do base or instruct-tuned LMs affect performance? Does co-training on language-only data help? [Bottom Right] 4. Scaling Properties - are we undertraining our models? What type of added data helps?](https://pbs.twimg.com/media/GGPShHebsAAEpQX.jpg?name=orig)
ALT Our investigation of different axes for developing visually-conditioned language models (VLMs) spans four different axes. [Top Left] 1. Optimization procedure - should we freeze model components, or do we need multi-stage training? [Bottom Left] 2. Image processing and visual representations - how should we choose pretrained representations? [Top Right] 3. Language Models - how do base or instruct-tuned LMs affect performance? Does co-training on language-only data help? [Bottom Right] 4. Scaling Properties - are we undertraining our models? What type of added data helps?
xitter.gq/siddkaramcheti/status/β¦ More info on Prismatic here.
What design choices matter when developing a visually-conditioned language model (VLM)?
Check out our paper β Prismatic VLMs β and open-source training code, evaluation suite, and 42 pretrained VLMs at the 7B-13B scale!
π arxiv.org/abs/2402.07865
βοΈ + π€ github.com/TRI-ML/prismatic-β¦
ALT Our investigation of different axes for developing visually-conditioned language models (VLMs) spans four different axes. [Top Left] 1. Optimization procedure - should we freeze model components, or do we need multi-stage training? [Bottom Left] 2. Image processing and visual representations - how should we choose pretrained representations? [Top Right] 3. Language Models - how do base or instruct-tuned LMs affect performance? Does co-training on language-only data help? [Bottom Right] 4. Scaling Properties - are we undertraining our models? What type of added data helps?
161
By first developing some of the best Vision-Language Models with Prismatic at TRI:
github.com/TRI-ML/prismatic-β¦
OpenVLA was able to quickly build some of the best generalist policies for robotics. Code, data and weights are all open-source:
openvla.github.io
This is a great achievement! Congrats @moo_jin_kim @siddkaramcheti @KarlPertsch @ashwinb96 @SurajNair_1 and all collaborators.
β¨ Introducing ππ©ππ§πππ β an open-source vision-language-action model for robotics! π
- SOTA generalist policy
- 7B params
- outperforms Octo, RT-2-X on zero-shot evals π¦Ύ
- trained on 970k episodes from OpenX dataset π€
- fully open: model/code/data all online π€
π§΅π
Thomas Kollar retweeted
Recurrent models like RWKV and Mamba have gained attention recently, but these can be costly to train and iterate on.
What if we could simply...
turn Mistral/Llama/Gemma into an RNN? π©πͺ
Presenting our work, Linearizing Large Language Models!
arxiv.org/abs/2405.06640
Over the last year at TRI weβve been training Large Language Models, including results in the following areas:
Scaling: arxiv.org/abs/2403.08540
Alignment: arxiv.org/abs/2402.12366
As a part of upcoming work, we are sharing back with the open source community and releasing a performant Mamba model that weβve trained at the 7B parameter scale. More results on linear transformers upcoming.
π’ Releasing TRI's open-source Mamba-7B trained on 1.2T tokens of RefinedWeb!
Mamba-7B is the largest fully recurrent Mamba model trained and is a state-of-the-art recurrent LLM. πππ
huggingface.co/TRI-ML/mamba-β¦
At TRI, we are looking to build Robotics Foundation Models, or what we call Large Behavior Models (LBMs).
Large scale robotics datasets, such as DROID are a part of the strategy to enable broadly capable LBMs that include language, vision and action:
droid-dataset.github.io
When we started this project through our university collaborations nearly 2 years ago, I was amazed the amount of community interest in contributing to the project. TRI has been deeply involved in all stages of dataset collection and release. Looking forward to seeing what the community does with it!
cc @MashaItkina @siddkaramcheti @MercatJean @SurajNair_1 and the rest of the TRI DROID team
Introducing a new, fully open robotics dataset!
- 76k episodes
- 564 unique scenes
- 100 contributors
- 13 labs/institutions
- 3 continents
droid-dataset.github.io
A short 𧡠on the backstory