Pierre Foret retweeted
Excited to share our #ICML2022 paper "Towards Understanding Sharpness-Aware Minimization"!
Why does m-sharpness matter in m-SAM? Can we explain the benefits of m-SAM on simple models? Which other interesting properties does m-SAM show?
Paper: arxiv.org/abs/2206.06232
🧵1/n
Pierre Foret retweeted
Excited to host @TheGradient to talk about the current state of Sharpness-Aware Minimization (SAM) and future directions next Friday 25th Feb 5pm (GMT time) Zoom details: ucl-ellis.github.io/dm_csml_…
Pierre Foret retweeted
Are you a strong PhD student interested in doing cutting edge research at @GoogleAI? I have an opening for student researcher position to explore open problems and extensions of Sharpness-Aware Minimization (SAM) w/ @bneyshabur. Please refer to tinyurl.com/4nfarsvt.
Pierre Foret retweeted
Sharpness-Aware Minimization Improves Language Model Generalization
SAM substantially improves performance on SuperGLUE, GLUE, Web Questions, Natural Questions, Trivia QA, and TyDiQA by encourageing convergence to flatter minima w/ minimal overhead.
arxiv.org/abs/2110.08529
Pierre Foret retweeted
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations
pdf: arxiv.org/pdf/2106.01548.pdf
abs: arxiv.org/abs/2106.01548
+5.3% and +11.0% top-1 accuracy on ImageNet for ViT-B/16 and MixerB/16, with the simple Inception-style preprocessing
Pierre Foret retweeted
Interesting empirical study of the geometry of the loss landscape of Vision Transformers and MLP-Mixers and study of the critical impact of Sharpness Aware Minimization (SAM) for those architectures.
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations
pdf: arxiv.org/pdf/2106.01548.pdf
abs: arxiv.org/abs/2106.01548
+5.3% and +11.0% top-1 accuracy on ImageNet for ViT-B/16 and MixerB/16, with the simple Inception-style preprocessing
Pierre Foret retweeted
Excited to see Sharpness-Aware Minimization (SAM optimizer) we have proposed recently (w/ @Foret_p @bneyshabur and Kleiner) is becoming a persistent component in recent state-of-the-art records 😇
Drawing Multiple Augmentation Samples Per Image
During Training Efficiently Decreases Test Error
pdf: arxiv.org/pdf/2105.13343.pdf
abs: arxiv.org/abs/2105.13343
ImageNet SOTA of 86.8% top-1 accuracy after just 34 epochs of training with an NFNet-F5 using the SAM optimizer
Pierre Foret retweeted
Sharpness-Aware Minimization for Efficiently Improving Generalization (Spotlight at #ICLR2021 )
with @Foret_p, Ariel Kleiber and @TheGradient
Paper: openreview.net/forum?id=6Tm1…
Code: github.com/google-research/s…
Video and Poster: iclr.cc/virtual/2021/poster/…
3/7
Pierre Foret retweeted
We don’t need to worry about #Overfitting anymore? Sharpness-Aware Minimization, seeks parameters that lie in neighborhoods having uniformly low loss; results in a min-max optimization formulation with efficient gradient descent #MachineLearning buff.ly/38VJTOf
Pierre Foret retweeted
Pretrained NFNet model weights (F0-F5, F6+SAM) are now available at dpmd.ai/nfnets, along with a demo Colab! All models are pretrained on ImageNet.
Pierre Foret retweeted
High-Performance Large-Scale Image Recognition Without Normalization
pdf: arxiv.org/pdf/2102.06171.pdf
abs: arxiv.org/abs/2102.06171
github: github.com/deepmind/deepmind…
Pierre Foret retweeted
Also nice to see that NFNets can also benefit from Sharpness Aware Minimization by @Foret_p et a.:
arxiv.org/abs/2010.01412
Pierre Foret retweeted
Happy new year everyone 🥳! We have the perfect gift to kick off your research in 2021. All data used for the competition are now open-sourced. In addition, the write-ups and talks of the participating teams are also available. You can find them at: sites.google.com/view/pgdl20…
Pierre Foret retweeted
This is happening right now!
Replying to @PGDL_NeurIPS
We will also have a longer event where all participants will present their solutions and participate in a roundtable discussion. This event happens at 2:00 pm EST and will be streamed at: youtu.be/30cF3PhTq-I. We hope to see you there!
Pierre Foret retweeted
It’s the last day of NeurIPS 👀 The winners of PGDL will be presenting their solutions at the Saturday NeurIPS competition tracks at 12:00 EST (neurips.cc/virtual/2020/publ…). Come to find out what 🔥 generalization measures 🔥 they discovered!
Pierre Foret retweeted
🔥New state-of-the-art performance on ImageNet recorded by our "Sharpness-Award Minimization" work; a very simple yet effective optimization method. Paper arxiv.org/abs/2010.01412 Performance paperswithcode.com/sota/imag…. w/ @Foret_p @bneyshabur and Ariel Kleiner.
Fine-tuning the EfficientNet l2-475 from the noisy student paper with SAM increases the imagenet accuracy from 88.2% to 88.6%, giving a new state or the art!
With Ariel Kleiner, @TheGradient, and @bneyshabur
paperswithcode.com/sota/imag…
Pierre Foret retweeted
The final results of the PGDL competition are now available! Over 3 months, we observed many creative solutions that participants proposed to tackle one of the most difficult and important mysteries of deep learning.
The full leaderboard is here: sites.google.com/view/pgdl20…
Pierre Foret retweeted
More on confluencing unsupervised and multilingual MT.
Great work with the awesome team: @xgarcia238, @ank_parikh , @adisid01, @Foret_p, @ThiboIbo of @GoogleResearch, #GoogleAI
(1/3)
Check out our multilingual unsupervised translation work! Theory + SOTA results. Led by @xgarcia238 (1/4)
1. Multilingual View of Unsupervised MT - Findings of EMNLP 2020 (arxiv.org/abs/2002.02955 )
2. Multilingual Unsupervised MT for Rare Languages (arxiv.org/abs/2009.11201 )
Pierre Foret retweeted
Check out our multilingual unsupervised translation work! Theory + SOTA results. Led by @xgarcia238 (1/4)
1. Multilingual View of Unsupervised MT - Findings of EMNLP 2020 (arxiv.org/abs/2002.02955 )
2. Multilingual Unsupervised MT for Rare Languages (arxiv.org/abs/2009.11201 )
Pierre Foret retweeted
📢📢 The @NeurIPSConf competition on Predicting Generalization in Deep Learning (PGDL) has now started and participants can submit solutions on Codalab until Oct 8th, when we will start evaluating the submissions at competitions.codalab.org/com…. #NeurIPS #NeurIPS2020




























