Andy Brock (@ajmooch) header
Andy Brock (@ajmooch) profile picture

Dimensionality Diabolist, Seeker of Optima

Joined October 2016
Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)Photo by Andy Brock (@ajmooch)
Our most recent work on training Normalizer-Free nets! We focus on developing performant architectures which train fast, and show that a simple technique (Adaptive Grad Clipping, or AGC) allows us to train with large batches and heavy augmentations and reach state-of-the-art.
Introducing NFNets, a family of image classification models that are: *SOTA on ImageNet (86.5% top-1 w/o extra data) *Up to 8.7x faster to train than EfficientNets to a given accuracy *Normalizer-free (no BatchNorm!) Paper: dpmd.ai/06171 Code: dpmd.ai/nfnets
6
64
4
343
Literally resnets dude
very surprising that fifteen years of hardcore computer vision research contributed ~nothing toward AGI except better optimizers we still don't have models that get smarter when we give them eyes
2
53
5,977
Never will be.
Pre-training is not dead
5
12
137
16,313
Ah yes, the obscure "University of Edinburgh," which famously did not produce this year's physics Nobel laureate
another very impressive thing about the VR-CLI paper is that there are only two authors. a phd student + professor from... Scotland, of all places A+ research can come from anywhere
1
16
1,160
💪
Introducing Gemini 2.5 Pro Experimental. The 2.5 series marks a significant evolution: Gemini models are now fundamentally thinking models. This means the model reasons before responding, to maximize accuracy -- and it’s our best Gemini model yet. Blog - goo.gle/4c3NitO
2
46
3,734
Interested in helping us make Gemini Pro even better? The Gemini pre-training team is looking for a Research Scientist in London to push the boundaries of LLM scaling: understanding, predicting, and improving. ♊️🚀 Apply here: boards.greenhouse.io/deepmin…
Replying to @GoogleDeepMind
2.0 Pro Experimental is our best model yet for coding and complex prompts, refined with your feedback. 🤝 It has a better understanding of world-knowledge and comes with our largest context window yet of 2 million tokens - meaning it can analyze large amounts of information.
21
1
63
33,928
Open robotaxi door Cupholder already has a delicious milkshake Check the logo It's a Faemo Almost got me! Remember to check the True Name of your ride, every time!
4
572
Oh no, I have learned [DETAIL] about [CRAFT OR TRADE] and now whenever I see [THING] in [MEDIA] I am hopelessly distracted by [DETAIL]
5
541
Counterpoint: JAX is literally the greatest library for neural network research, and it's only the stuff it doesn't do that gets in the way
Someone needs to say it: As someone who started using it in its early days (2019 when I interned at Google) and tries it again occasionally - JAX is terrible! It might be due to my software engineering skills, but it's very hard to debug, and the speed improvement isn't worth it
5
7
2
111
31,115
Hi Matteo, this idea was originally proposed (in a more general form, incorporating an arbitrary number of momentum buffers, albeit with a specific way to pick betas + no alpha) as AggMo, Lucas et al, ICLR 2019: openreview.net/forum?id=Syxt…; you may wish to consider citing
Stop discarding your old gradients! Introducing AdEMAMix, a novel (first-order) optimizer capable of outperforming Adam. Let’s have a thread on momentum and the surprising relevance of very old gradients. A joint work with @GrangierDavid and @PierreAblin #ml #optimization 1/🧵
1
3
45
8,651
Andy Brock retweeted
Exciting News from Chatbot Arena! @GoogleDeepMind's new Gemini 1.5 Pro (Experimental 0801) has been tested in Arena for the past week, gathering over 12K community votes. For the first time, Google Gemini has claimed the #1 spot, surpassing GPT-4o/Claude-3.5 with an impressive score of 1300 (!), and also achieving #1 on our Vision Leaderboard. Gemini 1.5 Pro (0801) excels in multi-lingual tasks and delivers robust performance in technical areas like Math, Hard Prompts, and Coding. Huge congrats to @GoogleDeepMind on this remarkable milestone! Gemini (0801) Category Rankings: - Overall: #1 - Math: #1-3 - Instruction-Following: #1-2 - Coding: #3-5 - Hard Prompts (English): #2-5 Come try the model and let us know your feedback! More analysis below👇
Today, we are making an experimental version (0801) of Gemini 1.5 Pro available for early testing and feedback in Google AI Studio and the Gemini API. Try it out and let us know what you think! aistudio.google.com
79
386
243
1,624
1,339,529
instead of a laser beam, the Death Star should smash through planets like the Kool-Aid man
1
6
872
Super excited for this launch! Had a great time working with a really amazing team getting this out!! Check it out:
Gemini 1.5 Pro - A highly capable multimodal model with a 10M token context length Today we are releasing the first demonstrations of the capabilities of the Gemini 1.5 series, with the Gemini 1.5 Pro model. One of the key differentiators of this model is its incredibly long context capabilities, supporting millions of tokens of multimodal input. The multimodal capabilities of the model means you can interact in sophisticated ways with entire books, very long document collections, codebases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more. Gemini 1.5 was built by an amazing team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google. @OriolVinyals (my co-technical lead for the project) and I are incredibly proud of the whole team, and we’re so excited to be sharing this work and what long context and in-context learning can mean for you today! There’s lots of material about this, some of which are linked to below. Main blog post: blog.google/technology/ai/go… Technical report: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” goo.gle/GeminiV1-5 Videos of interactions with the model that highlight its long context abilities: Understanding the three.js codebase: youtube.com/watch?v=SSnsmqIj… Analyzing a 45 minute Buster Keaton movie: youtube.com/watch?v=wa0MT8Ow… Apollo 11 transcript interaction: youtube.com/watch?v=LHKL_210… Starting today, we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI. Read more about this on these blogs: Google for Developers blog: developers.googleblog.com/20… Google Cloud blog: cloud.google.com/blog/produc… We’ll also introduce 1.5 Pro with a standard 128,000 token context window when the model is ready for a wider release. Coming soon, we plan to introduce pricing tiers that start at the standard 128,000 context window and scale up to 1 million tokens, as we improve the model. Early testers can try the 1 million token context window at no cost during the testing period. We’re excited to see what developer’s creativity unlocks with a very long context window. Let me walk you through the capabilities of the model and what I’m excited about!
2
7
1
95
14,360
So excited to see Gemini 1.5 released to the world! It was a blast working with such an amazing team.
Gemini 1.5 Pro - A highly capable multimodal model with a 10M token context length Today we are releasing the first demonstrations of the capabilities of the Gemini 1.5 series, with the Gemini 1.5 Pro model. One of the key differentiators of this model is its incredibly long context capabilities, supporting millions of tokens of multimodal input. The multimodal capabilities of the model means you can interact in sophisticated ways with entire books, very long document collections, codebases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more. Gemini 1.5 was built by an amazing team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google. @OriolVinyals (my co-technical lead for the project) and I are incredibly proud of the whole team, and we’re so excited to be sharing this work and what long context and in-context learning can mean for you today! There’s lots of material about this, some of which are linked to below. Main blog post: blog.google/technology/ai/go… Technical report: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” goo.gle/GeminiV1-5 Videos of interactions with the model that highlight its long context abilities: Understanding the three.js codebase: youtube.com/watch?v=SSnsmqIj… Analyzing a 45 minute Buster Keaton movie: youtube.com/watch?v=wa0MT8Ow… Apollo 11 transcript interaction: youtube.com/watch?v=LHKL_210… Starting today, we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI. Read more about this on these blogs: Google for Developers blog: developers.googleblog.com/20… Google Cloud blog: cloud.google.com/blog/produc… We’ll also introduce 1.5 Pro with a standard 128,000 token context window when the model is ready for a wider release. Coming soon, we plan to introduce pricing tiers that start at the standard 128,000 context window and scale up to 1 million tokens, as we improve the model. Early testers can try the 1 million token context window at no cost during the testing period. We’re excited to see what developer’s creativity unlocks with a very long context window. Let me walk you through the capabilities of the model and what I’m excited about!
2
9
1,477
I was leading long context in Gemini for a while now, and today I’m proud to share what the team has achieved: over 1M context in a large-scale foundation model. Big shoutout to @TeplyashinDenis and @machelreid - without you this would not have happened! youtube.com/watch?v=wa0MT8Ow…
9
21
5
250
97,456
10,000,000 tokens achieved internally! 🚢 🚢 🚢
Gemini 1.5 has arrived. Pro 1.5 with 1M tokens available as an experimental feature via AI Studio and Vertex AI in private preview. Then there’s this: In our research, we tested Gemini 1.5 on up to 2M tokens for audio, 2.8M tokens for video, and 🤯10M 🤯 tokens for text. From Shannon’s 1950s bi-gram models (2 tokens), and after being mesmerized by LSTMs many years ago able to model 200 tokens, it feels almost impossible that I would be talking about hundreds of thousands of tokens in context length, let alone millions. ♊️💙 Tech report: goo.gle/GeminiV1-5
3
2
66
6,880
Just watched Saltburn, aka Get In
1
3
954
Things I'm looking forward to this year: -Vending machine you can haggle with -Fridge with twitter (not for you, it's the fridge that tweets) -Doorbell that makes you solve a sphinx riddle to get in -Toaster that screams
18
1,579
In service of keeping my inner child alive, today I was defeated by a childproof cap.
1
10
1,268
Having so much fun working on this every day! 😊 Building the best model with friends and lots of TPUs. 🚀
The Gemini era is here. Thrilled to launch Gemini 1.0, our most capable & general AI model. Built to be natively multimodal, it can understand many types of info. Efficient & flexible, it comes in 3 sizes each best-in-class & optimized for different uses blog.google/technology/ai/go…
1
3
70
10,975
Andy Brock retweeted
Gemini Nano improve on the efficiency frontiers. They are multimodal as well, see results in the paper. Nano series: At 1.8B and 3.25B parameters packs so much to provide high utility on device First foundation model on the device! android-developers.googleblo…
Replying to @sundarpichai
Gemini Nano is super efficient for tasks that are on-device. Android developers can sign up for an early access program for Gemini Nano via Android AICore and Pixel 8 Pro users can already see it rolling out in features like Summarize in Recorder and Smart Reply in Gboard + much more to come! blog.google/products/pixel/p…
4
12
6
129
57,076