large language models @kyutai_labs
paris, france
Joined October 2012
- Tweets131
- Following174
- Followers2.9K
- Likes239
Today, we release our 🇫🇷 to 🇬🇧 simultaneous speech-to-speech translation system, called Hibiki. It runs on-device & the model, inference code and tech report are available. This is built using the same audio LLM as Moshi, showing its versatility. 🟢
Meet Hibiki, our simultaneous speech-to-speech translation model, currently supporting 🇫🇷➡️🇬🇧.
Hibiki produces spoken and text translations of the input speech in real-time, while preserving the speaker’s voice and optimally adapting its pace based on the semantic content of the source speech.
Based on objective and human evaluations, Hibiki outperforms previous systems for quality, naturalness and speaker similarity and approaches human interpreters. 🧵
Excited to release a preview of Helium-1, our 2B LLM targeting edge and mobile devices. 🚀
More to come in the future: training code, support for more languages, data pipeline, tech report & more… 🟢
Meet Helium-1 preview, our 2B multi-lingual LLM, targeting edge and mobile devices, released under a CC-BY license. Start building with it today!
huggingface.co/kyutai/helium…
I am at ICML in Vienna! Let me know if you want to chat about (or to) Moshi, multimodal LLMs, Kyutai & more.
✈️ I will be attending #NeurIPS2023: let me know if you want to chat about the future of LLMs, and how to democratize them.
🌐 We are also hiring members of technical staff and interns @kyutai_labs. Happy to talk about the lab and our mission.
/kyutai has landed! Super excited to build this new research lab. Pure focus on research. As open as it gets.
Announcing Kyutai: a non-profit AI lab dedicated to open science. Thanks to Xavier Niel (@GroupeIliad), Rodolphe Saadé (@cmacgm) and Eric Schmidt (@SchmidtFutures ), we are starting with almost 300M€ of philanthropic support. Meet the team ⬇️
Super excited by the release of LLaMA, a serie of large language models, from 7B to 65B parameters. 🎉
By training longer, LLaMA obtains GPT3 level performance with a 13B model, which can run on a single GPU. Excited to see what the research community will do with these models.
Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters.
LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B.
The weights for all models are open and available at research.facebook.com/public…
1/n
Introducing PEER, a new language model which makes text generation and editing more collaborative and controllable. It adds human in the loop, by following instructions and providing explanations.
Work lead @timo_schick.
Paper: arxiv.org/abs/2208.11663
🎉 New paper 🎉 We introduce PEER, a language model trained to incrementally write texts & collaborate w/ humans in a more natural way. It can write drafts, add suggestions, follow instructions, perform edits, correct itself & provide explanations.
Link: arxiv.org/abs/2208.11663
Very excited to introduce Atlas, a new retrieval augmented language model which is competitive with larger models on few-shot tasks such as question answering or fact checking.
Work lead by @gizacard and @PSH_Lewis.
Paper: arxiv.org/abs/2208.03299
🚨We’ve been working on better retrieval-augmented models & thrilled to present Atlas, led by @gizacard @EXGRV & myself🚨
Atlas is a end2end pretrained "RAG"-like model, beats models 50x its size on fewshot QA, sets numerous SotA on knowledge-intensive NLP arxiv.org/abs/2208.03299
New release of our Contriever project! It includes multi-lingual models which can perform cross-lingual retrieval (eg, retrieve English documents to answer a question in Swahili), the code to (pre-)train your own retrievers, and an updated version of the paper with new results.
Code for Contriever is now available!
Code: github.com/facebookresearch/…
Paper: arxiv.org/pdf/2112.09118.pdf
Additionally we trained mContriever, a state-of-the-art multilingual neural retriever, by applying a similar contrastive learning method.
🔎 Can we train dense unsupervised retrievers that are as good as BM25? With the latest contrastive learning techniques, it seems that we are getting there! Our model, the Contriever, outperforms BM25 on NQ, and is competitive on BEIR.
Paper: arxiv.org/abs/2112.09118
On BEIR, the Contriever is on-par, or outperforms, BM25 on 11 out of 15 datasets for the recall@100. Code & models will be released soon.
Joint work w/ @gizacard @mcaron31 @lucas_hosseini @riedelcastro @p_bojanowski @armandjoulin
New paper on memory efficient open domain question answering. We show that combining dimension reduction, vector quantization and passage filtering greatly reduces the memory footprint of retrieval based systems, without hurting accuracy too much.
Paper: arxiv.org/abs/2012.15156
Joint work with @gizacard, @Fabio_Petroni, @lucas_hosseini, @nicola_decao and @riedelcastro. This was part of our winning entry to the 6Gb track of the EfficientQA NeurIPS competition.
Super happy about this result too! 🚀
And thanks to the organizers for this great competition!
Super happy for winning the 6Gb track at the EfficientQA #NeurIPS competition. Our submission, lead by @gizacard w/ Lucas Husseini, @nicola_decao, @riedelcastro and
@EXGRV, achieved top position in both auto and manual evaluation. 🚀
9
New paper with Gautier Izacard (@gizacard), using distillation to train information retrieval systems! We show that attention scores of a model trained on the downstream task can be used as synthetic labels. This allows to train retrievers without document or passage annotations.
We obtain new state-of-the-art results on TriviaQA (+4.5%) and NaturalQuestions (+2.3%). We also used this technique for our winning entry to the 6 GB track of the Efficient QA competition (more on this soon).
Paper: arxiv.org/abs/2012.04584
New work w/ @gizacard (Gautier Izacard): how much do generative models for open domain QA benefit from retrieval? A lot! Retrieving 100 passages, we get 51.4 EM on NaturalQuestions, 67.6 EM on TriviaQA. 1/3
Paper: arxiv.org/abs/2007.01282
Training with QuantNoise allows to strongly compress neural networks: 80.0% accuracy on ImageNet in 3.3MB, 82.5MB accuracy on MNLI in 14MB.
Blog: ai.facebook.com/blog/trainin…
Paper: arxiv.org/abs/2004.07320
Hyper-parameter autotuning for fastText: get a 1MB text classifier while having a coffee.
With this new feature, it is possible to constrain the size of the final model, and automatically find the hyper-parameters giving the best results on a validation set.
Facebook AI researchers are releasing a new feature for the fastText library which provides hyper-parameter autotuning for more efficient text classifiers. ai.facebook.com/blog/fasttex…
New blogpost about two recent papers on Transformer networks.
Facebook AI researchers are sharing an all-attention layer to simplify the Transformer model and an adaptive attention span method to make it more efficient. Even with a much simpler architecture, these methods match or improve state-of-the-art results. ai.facebook.com/blog/making-…
New long-form question answering dataset and baselines!
Introducing long- form question answering (ai.facebook.com/blog/longfor…), a new challenge that pushes #AI to provide complex explanations rather than just simple facts. #NLP
5