Arijit Ray.|

Research Scientist @ Google |

Projects CV Github

Google Scholar
 

I aim to build intelligent tools that complement* human abilities.

About me:

I am a Research Scientist at Google, working on developing AI models that can complete complex enterprise tasks. I completed my Ph.D., advised by Kate Saenko and Bryan Plummer at Boston University, and by Ranjay Krishna at the University of Washington. I received my M.S from Virginia Tech, advised by Devi Parikh.

I am fascinated by how humans accomplish remarkable things simply by reasoning while observing their surroundings and interacting with the visual world. Hence, my research aims to advance AI models’ capability to reason while visually interacting with an environment.

I also enjoy building creative ecosystems that help people discover the access to the right community. I have helped start and organize networking events across five schools in the Boston area for conversations around how AI researchers can help practioners from diverse verticals.

google

Student Researcher, Google, 2025
ai2

Student Collaborator, Allen AI (PRIOR), 2024–present
googleX

AI Resident, Google X Moonshot Labs, 2023
meta_ai_fair

Research Intern, Meta AI (FAIR), 2022

Selected Highlights:

  • 2026: I graduated and started as a Research Scientist at Google.
  • 2026: I was recognized as an Outstanding Reviewer for CVPR 2026 (top 5%).
  • 2025: Excited to spend my summer at Google, working with the Pixel team and Leonidas Guibas from Deepmind.
  • 2024: Our paper on feedback-guided autonomous driving got accepted to CVPR 2024 as a highlight.
  • 2023: One of my student’s paper on diverse reactions to multimodal content got accepted as an oral at an ICCV Workshop 2023.
  • 2022: Excited to be spending my summer as a Research Scientist Intern at Meta (Facebook) AI (FAIR), working on the compositionality of large vision-language models.
  • 2022: I started a PhD; We started [AI+X] of BU and Harvard, where we brainstorm research/venture ideas on how AI can impact different verticals.
  • 2019: I won runners-up at the SRI CVT Shark Tank Competition that supported my mini-project on understanding image-text content to reduce radicalization of opinions on social media.
  • 2017: The weed vs plant detection system I helped develop for precision fertilizing played a key part in the acquisition of Blue River Technologies by John Deere for 305 million USD.
  • 2014: Our UAV to help locate natural disaster victims was featured in national news: Deccan Chronicle, Indian Express
  • 2012: I won an Academic Merit Scholarship from SRM University that waives a part of my undergraduate tuition.

Selected publications (all):

2026

PLE
Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee, Yifei Ming, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
PLE: Physical-Engineering’s Last Exam
arXiv 2026
Webpage arXiv Code Data
TL;DR: A real engineer-in-the-loop benchmark to test agents’ capability to design the physical world.

2025

Mull-Tokens
Arijit Ray, Ahmed Abdelkader, Chengzhi Mao, Bryan A. Plummer, Kate Saenko, Ranjay Krishna, Leonidas Guibas, Wen-Sheng Chu
Mull-Tokens: Modality-Agnostic Latent Thinking
CVPR 2026 (Findings)
Webpage arXiv Code Model
TL;DR: Instead of text reasoning or explicit image thoughts, using just a few multimodal thinking tokens is more effective for visual reasoning tasks.
SIMS-V
Ellis Brown, Arijit Ray, Ranjay Krishna, Ross Girshick, Rob Fergus, Saining Xie
SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
arXiv 2025
Webpage arXiv Data
TL;DR: Using simulators to create spatially-rich video training data for multimodal language models. Our 7B-parameter model fine-tuned on just 25K simulated examples outperforms the larger 72B baseline and achieves competitive performance with proprietary models on real-world spatial reasoning benchmarks.
Abhay Deshpande, Yuquan Deng, Arijit Ray, Jordi Salvador, Winson Han, Jiafei Duan, Kuo-Hao Zeng, Yuke Zhu, Ranjay Krishna, Rose Hendrix
GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation
CORL 2025
Webpage arXiv Data Code
TL;DR: Use textual reasoning (e.g., one should grasp “handle” of cup to drink tea) and visual reasoning (e.g., in this image, the cup is grasped by the “handle”) along with an object grasping dataset (data of 6 DOF arm positions to grasp objects) to generate high-level task description (e.g., how should I grasp cup to drink tea?) to precise 6DOF grasp data.
SAT
Arijit Ray, Ellis Brown, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna*, Kuo-Hao Zeng*, Kate Saenko*
SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
COLM 2025
Webpage arXiv Data Code
TL;DR: Simulated spatial aptitude data (SAT) using physics engines can improve spatial reasoning in real images and videos for MLMs while maintaining pretraining commonsense. Spatial annotation quality matters more than visual realism.

2024

R2D3
Arijit Ray, Dina Bashkirova, Reuben Tan, Kuo-Hao Zeng, Bryan A. Plummer, Ranjay Krishna, Kate Saenko
R2D3: Imparting Spatial Reasoning by Reconstructing 3D Scenes from 2D Images
Tech report 2024
Report
TL;DR: Reconstructing a 3D environment with interactive objects in a physics simulator from just a single image.
FED
Jimuyang Zhang, Zanming Huang, Arijit Ray, Eshed-Ohn Bar
FED: Feedback-Guided Autonomous Driving
CVPR 2024 (Highlight)
Paper
TL;DR: Understanding natural language feedback to refine the next waypoint prediction in autonomous driving.

2023

Cola
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, Kate Saenko
Cola: A Benchmark for Compositional Text-to-image Retrieval
NeurIPS 2023
arXiv Project Page & Data
TL;DR: Multimodal layers are the most responsible for compositional reasoning rather than the visual or textual encoders. Tuning multimodal layers over frozen representations are more effective than tuning similar amount of parameters in the individual encoders or even the entire model.
Lasagna
Dina Bashkirova, Arijit Ray, Rupayan Mallick, Sarah Adel Bargal, Jianming Zhang, Ranjay Krishna, Kate Saenko
Lasagna: Layered Score Distillation for Disentangled Object Relighting
arXiv Project Page & Data
TL;DR: Synthetically generated examples using 3D graphics engines are effective in teaching physics-aware edits like relighting if we use score-distillation to avoid overfitting.
Socratis
Katherine Deng, Arijit Ray, Reuben Tan, Saadia Gabriel, Bryan A. Plummer, Kate Saenko
Socratis: Are Large Multimodal Models Emotionally Aware?
ICCV Workshops 2023 (oral), Workshop on Emotionally and Culturally Intelligent AI
arXiv Project Page & Data
TL;DR: A preliminary benchmark to test MLMs on why different people may feel different emotions for diverse reasons while viewing the same image-text content.

2021

User-targeted content generation
Ajay Divakaran, Karan Sikka, Arijit Ray, Xiao Lin, Yi Yao
User-targeted content generation using multimodal embeddings
US Patent App. 17/191,698
Webpage
VQA Error Regions
Arijit Ray, Michael Cogswell, Xiao Lin, Kamran Alipour, Ajay Divakaran, Yi Yao, Giedrius Burachas
Knowing What VQA Does Not: Pointing to Error-Inducing Regions to Improve Explanation Helpfulness
Applied AI Letters (Wiley) 2021
PDF arXiv Project Page

2019

VQA Consistency
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, Giedrius Burachas
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
EMNLP 2019, also at CVPR-W 2019 VQA and Visual Dialog Workshop
arXiv BibTeX Data
TL;DR: Use a learned teacher module to reward VQA models for predicting consistent answers to entailed questions.

2016

VQA Relevance
Arijit Ray, Gordon Christie, Mohit Bansal, Dhruv Batra, Devi Parikh
Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions
EMNLP 2016
PDF Code Video
TL;DR: Image captions are a surprisingly strong baseline to identify whether a question is relevant to an image or not.

Press Coverage

Hobbies

I derive joy from a positive gradient, however incremental, in personal growth — both in myself and the people I care about. When not training AI models, I enjoy making latte art, going to techno fests, swimming, and skiing. I believe fascinating things happen when you bring bright ambitious personalities together in a room. Through the years, I have organized events across BU, Harvard, and MIT to foster such community. In middle school, I opened a research society to encourage students to take an interest in science by constructing simple gadgets. We won multiple accolades in city and state-level exhibitions.

I also admire well-crafted snippets that convey a glimpse of something deep about human lives and society. Hence, I maintain a list of my favorite quotes.

Contact

Best way to reach me would be to drop an email to array at bu dot edu.