Ọládípọ̀ AF

🚀 Introducing MMMU-Pro: A more robust version of MMMU arxiv.org/pdf/2409.02813v1 After launching MMMU, we received valuable feedback from the community: 1️⃣ Some questions were answerable without even seeing the images. 2️⃣ Models didn’t always "know" the answer but found shortcuts from the options provided. 3️⃣ Performance was heavily tied to LLMs, with minimal impact from the vision module. To tackle these issues, we implemented the following improvements: 🔍 1. Filtering Text-Only Answerable Questions 🔄 2. Augmenting Candidate Options up to 10 by Human Experts. 🖼️ 3. Vision-Only Input Setting: where questions are embedded directly in images, requiring the model to rely purely on visual input. ✨ Why We Added Vision-Only Input Setting? 1. From a foundational perspective, this setting forces AI to genuinely "see" and "read" at the same time—challenging a core human cognitive skill: the seamless integration of visual and textual information. 2. From an application standpoint, this approach mirrors how users naturally interact with AI systems—by sharing screenshots or photos, without meticulously separating text from images. 📊 Key Results: Performance on MMMU-Pro is notably lower compared to MMMU, ranging from 16.8% to 26.9% across various models. The ranking of models is generally similar to the original but we also observe less robust ones— for example, GPT-4o mini proved less robust than GPT-4o and other proprietary models, showing significant drops in performance on the augmented set. 🔬 More in-depth analysis can be found in the threads below! 👇