Rayat Imtiaz Hossain retweeted
📊 Why do Large Vision-Language Models struggle with charts?
Even the most advanced LVLMs often look at the wrong parts of a chart—fixating on irrelevant axes or labels—leading to wrong answers and making their reasoning hard to interpret. In domains like finance, medicine, or scientific research, this lack of interpretability is a huge barrier to trust.
✨ That’s where our new work ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement comes in.
We introduce ChartGaze, the first eye-tracking dataset for chart reasoning, capturing how humans actually look at charts when answering questions. With this, we:
🔹 Show that LVLM attention diverges significantly from human gaze.
🔹 Propose gaze-guided attention refinement to teach models where to look.
🔹 Improve both accuracy (+2.56%) and interpretability, making attention maps more human-like and reasoning more trustworthy.
What’s inside ChartGaze?
✅ 1,620 real-world charts from Statista, Pew, and beyond
✅ 4,638 high-precision human gaze maps
✅ 4,638 chart QA pairs across trend analysis, extremum finding, and comparison reasoning
🚀 The result: LVLMs that don’t just answer better—but reason in a way that’s more aligned with human cognition.
👉 Paper: arxiv.org/pdf/2509.13282
💻 Code: github.com/alisalamatian1/Ch…
📊 Dataset: huggingface.co/datasets/alis…
A huge thanks to my amazing co-authors and mentors: Ali Salamatian, @CyuanWan , @RayatHoss , and my supervisors Leonid Sigal and @careninigiusepp — for making this work possible! 🙏
#AI #MachineLearning #DeepLearning #NLP #ComputerVision #MultimodalAI #Interpretability #Charts #Visualization #Research #ChartQA #LVLMs
Rayat Imtiaz Hossain retweeted
I’m angry to learn that women wearing hijabs were the target of a violent attack in London, and a mosque in Winnipeg was the target of hateful harassment.
Islamophobia kills. Whenever we see it, we’ll confront it — and we’ll work to keep our communities safe from it.
Rayat Imtiaz Hossain retweeted
Eid Adha Mubarak everyone which is coinciding with @CVPR this year in Seattle. I will there in 3 days there. Looking forward to the talks, workshops, posters and meeting good people ^_^. Come visit us in our poster 383, June 21st 10:30-12:00 @RayatHoss. arxiv.org/abs/2404.11732
Rayat Imtiaz Hossain retweeted
Our @CVPR 2024 work is out where we discuss mutliscale visual prompting, novel-to-base causal attention and transductive prompt tuning to utilize the unlabelled data when prompting image segmentation models. #CVPR2024. arxiv.org/pdf/2404.11732.pdf, Kudos to
@RayatHoss
Rayat Imtiaz Hossain retweeted
Phenaki. Text to video using VQ encoders and masked transformers. Simple yet powerful. Love the 2 min video generation. phenaki.video/





