Fatma retweeted
Introducing MiniMax M3: The First Open-Weights Model to Combine Three Frontier Capabilities
- Coding & Agentic Frontier: 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench Hard, 74.2% MCP Atlas
- MiniMax Sparse Attention scales context to 1M
- Natively Multimodal from Step Zero
API: platform.minimax.io
Token Plan: platform.minimax.io/subscrib…
🚀New! MiniMax Code: code.minimax.io
Weights & Tech Report in ~10 Days
Fatma retweeted
one of the most important things I know about deep learning I learned from this paper: "Pretraining Without Attention"
this what I found so surprising:
these people developed an architecture very different from Transformers called BiGS, spent months and months optimizing it and training different configurations, only to discover that at the same parameter count, a wildly different architecture produces identical performance to transformers
this may imply that as long as there are enough parameters, and things are reasonably well-conditioned (i.e. a decent number of nonlinearities and and connections between the pieces) then it really doesn't matter how you arrange them, i.e. any sufficiently good architecture works just fine
i feel there's something really deep here, and we may be already very close to the upper bound of how well we can approximate a given function given a certain amount of compute. so we should spend more time thinking about other questions, such as what that function should actually look like (what data? which objective function?) and how to make it more efficient
Fatma retweeted
My lecture notes on regularization techniques in machine learning
dropbox.com/scl/fi/ivyb3wswq…
Check out the section on double descent suggested by @ylecun :)
Sigmoid Loss for Language Image Pre-Training paper (CLIP with sigmoid loss instead of softmax )
xitter.gq/giffmana/status/169264…
What makes CLIP work?
The contrast with negatives via softmax?
The more negatives, the better -> large batch-size?
We'll answer "no" to both in our ICCV oral🤓
By introducing SigLIP, a simpler CLIP that also works better and is more scalable, we can study the extremes.
Hop in🧶
1
141
So happy that our paper SPARF got a #CVPR2023 highlight! It is the result of my internship at #Google in @fedassa team.
SPARF: Neural Radiance Fields from Sparse and Noisy Poses
Arxiv: arxiv.org/abs/2211.11738
Website: prunetruong.com/sparf.github…
Code coming very soon!
1/n
70
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow xitter.gq/chriswolfvision/status…
I highly recommend you check out @WeinzaepfelP and colleagues poster on the "CroCo v2" vision foundation model ... and his T-Shirt ❤️.
The model is used for lots of downstream tasks (monocular depth, stereo, optical flow, pose, robotics/navigation).
1
80
vision transformers need registers xitter.gq/TimDarcet/status/17077…
Vision transformers need registers!
Or at least, it seems they 𝘸𝘢𝘯𝘵 some…
ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”.
Just add new tokens (“[reg]”):
- no artifacts
- interpretable attention maps 🦖
- improved performances!
87
Fatma retweeted
I've been told the traffic in Paris will be horrible, I didn't realize they were referring to the #ICCV poster session.
Fatma retweeted
#CVPR2023 is mostly posters this year, which is good! Poster sessions can be great. But a lot of people are not so great at presenting their poster. Here's some of the advice I give grad students when preparing for poster sessions.
Fatma retweeted
It in insane to me that @ftm_guney can get an EU research grant to fund her lab as a professor (very big deal) but cannot get a visa from the EU to come to a conference (should be a very small thing??!!!!!) Visa privilege is a real thing and smthg we don’t discuss enough in CV??l
Fatma retweeted
Vision transformers need registers!
Or at least, it seems they 𝘸𝘢𝘯𝘵 some…
ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”.
Just add new tokens (“[reg]”):
- no artifacts
- interpretable attention maps 🦖
- improved performances!
Fatma retweeted
#ICCV2023 Program Stats – at Paris, France







































