Pierre Foret (@Foret_p) profile picture

QR at Citadel Securities. Ex Google AI resident.

New York, USA
Joined October 2015
Photo by Pierre Foret (@Foret_p)Photo by Pierre Foret (@Foret_p)Photo by Pierre Foret (@Foret_p)Photo by Pierre Foret (@Foret_p)Photo by Pierre Foret (@Foret_p)
Excited to share our #ICML2022 paper "Towards Understanding Sharpness-Aware Minimization"! Why does m-sharpness matter in m-SAM? Can we explain the benefits of m-SAM on simple models? Which other interesting properties does m-SAM show? Paper: arxiv.org/abs/2206.06232 🧵1/n
4
31
2
195
Pierre Foret retweeted
Excited to host @TheGradient to talk about the current state of Sharpness-Aware Minimization (SAM) and future directions next Friday 25th Feb 5pm (GMT time) Zoom details: ucl-ellis.github.io/dm_csml_…
7
2
21
Are you a strong PhD student interested in doing cutting edge research at @GoogleAI? I have an opening for student researcher position to explore open problems and extensions of Sharpness-Aware Minimization (SAM) w/ @bneyshabur. Please refer to tinyurl.com/4nfarsvt.
4
22
117
Sharpness-Aware Minimization Improves Language Model Generalization SAM substantially improves performance on SuperGLUE, GLUE, Web Questions, Natural Questions, Trivia QA, and TyDiQA by encourageing convergence to flatter minima w/ minimal overhead. arxiv.org/abs/2110.08529
1
16
1
88
Pierre Foret retweeted
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations pdf: arxiv.org/pdf/2106.01548.pdf abs: arxiv.org/abs/2106.01548 +5.3% and +11.0% top-1 accuracy on ImageNet for ViT-B/16 and MixerB/16, with the simple Inception-style preprocessing
7
124
10
469
Pierre Foret retweeted
Interesting empirical study of the geometry of the loss landscape of Vision Transformers and MLP-Mixers and study of the critical impact of Sharpness Aware Minimization (SAM) for those architectures.
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations pdf: arxiv.org/pdf/2106.01548.pdf abs: arxiv.org/abs/2106.01548 +5.3% and +11.0% top-1 accuracy on ImageNet for ViT-B/16 and MixerB/16, with the simple Inception-style preprocessing
6
28
Excited to see Sharpness-Aware Minimization (SAM optimizer) we have proposed recently (w/ @Foret_p @bneyshabur and Kleiner) is becoming a persistent component in recent state-of-the-art records 😇
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error pdf: arxiv.org/pdf/2105.13343.pdf abs: arxiv.org/abs/2105.13343 ImageNet SOTA of 86.8% top-1 accuracy after just 34 epochs of training with an NFNet-F5 using the SAM optimizer
7
39
Sharpness-Aware Minimization for Efficiently Improving Generalization (Spotlight at #ICLR2021 ) with @Foret_p, Ariel Kleiber and @TheGradient Paper: openreview.net/forum?id=6Tm1… Code: github.com/google-research/s… Video and Poster: iclr.cc/virtual/2021/poster/… 3/7
2
4
1
19
Pierre Foret retweeted
We don’t need to worry about #Overfitting anymore? Sharpness-Aware Minimization, seeks parameters that lie in neighborhoods having uniformly low loss; results in a min-max optimization formulation with efficient gradient descent #MachineLearning buff.ly/38VJTOf
14
21
Pierre Foret retweeted
Pretrained NFNet model weights (F0-F5, F6+SAM) are now available at dpmd.ai/nfnets, along with a demo Colab! All models are pretrained on ImageNet.
Our most recent work on training Normalizer-Free nets! We focus on developing performant architectures which train fast, and show that a simple technique (Adaptive Grad Clipping, or AGC) allows us to train with large batches and heavy augmentations and reach state-of-the-art.
3
33
150
Pierre Foret retweeted
High-Performance Large-Scale Image Recognition Without Normalization pdf: arxiv.org/pdf/2102.06171.pdf abs: arxiv.org/abs/2102.06171 github: github.com/deepmind/deepmind…
1
63
6
239
Happy new year everyone 🥳! We have the perfect gift to kick off your research in 2021. All data used for the competition are now open-sourced. In addition, the write-ups and talks of the participating teams are also available. You can find them at: sites.google.com/view/pgdl20…
1
8
19
This is happening right now!
Replying to @PGDL_NeurIPS
We will also have a longer event where all participants will present their solutions and participate in a roundtable discussion. This event happens at 2:00 pm EST and will be streamed at: youtu.be/30cF3PhTq-I. We hope to see you there!
1
5
It’s the last day of NeurIPS 👀 The winners of PGDL will be presenting their solutions at the Saturday NeurIPS competition tracks at 12:00 EST (neurips.cc/virtual/2020/publ…). Come to find out what 🔥 generalization measures 🔥 they discovered!
1
4
9
🔥New state-of-the-art performance on ImageNet recorded by our "Sharpness-Award Minimization" work; a very simple yet effective optimization method. Paper arxiv.org/abs/2010.01412 Performance paperswithcode.com/sota/imag…. w/ @Foret_p @bneyshabur and Ariel Kleiner.
Fine-tuning the EfficientNet l2-475 from the noisy student paper with SAM increases the imagenet accuracy from 88.2% to 88.6%, giving a new state or the art! With Ariel Kleiner, @TheGradient, and @bneyshabur paperswithcode.com/sota/imag…
1
8
70
The final results of the PGDL competition are now available! Over 3 months, we observed many creative solutions that participants proposed to tackle one of the most difficult and important mysteries of deep learning. The full leaderboard is here: sites.google.com/view/pgdl20…
1
4
1
22
Pierre Foret retweeted
More on confluencing unsupervised and multilingual MT. Great work with the awesome team: @xgarcia238, @ank_parikh , @adisid01, @Foret_p, @ThiboIbo of @GoogleResearch, #GoogleAI (1/3)
Check out our multilingual unsupervised translation work! Theory + SOTA results. Led by @xgarcia238 (1/4) 1. Multilingual View of Unsupervised MT - Findings of EMNLP 2020 (arxiv.org/abs/2002.02955 ) 2. Multilingual Unsupervised MT for Rare Languages (arxiv.org/abs/2009.11201 )
1
6
1
36
Pierre Foret retweeted
Check out our multilingual unsupervised translation work! Theory + SOTA results. Led by @xgarcia238 (1/4) 1. Multilingual View of Unsupervised MT - Findings of EMNLP 2020 (arxiv.org/abs/2002.02955 ) 2. Multilingual Unsupervised MT for Rare Languages (arxiv.org/abs/2009.11201 )
2
38
2
135
📢📢 The @NeurIPSConf competition on Predicting Generalization in Deep Learning (PGDL) has now started and participants can submit solutions on Codalab until Oct 8th, when we will start evaluating the submissions at competitions.codalab.org/com…. #NeurIPS #NeurIPS2020
2
9
4
43