SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

Arijit Ray1, Jiafei Duan2 †, Ellis Brown2 †, Reuben Tan1,4, Dina Bashkirova1, Rose Hendrix3, Kiana Ehsani3, Aniruddha Kembhavi3, Bryan A. Plummer1, Ranjay Krishna2,3*, Kuo-Hao Zeng3*, Kate Saenko1*
1Boston University, 2University of Washington, 3Allen AI, 4Microsoft Research (MSR) 5New York University
*equal advising, † joint second author
Paper arXiv Code 🤗 Model Data Data v2 Reasoning Data lmms-eval
COLM 2025

✨ News

  • [2026.2.4] ✨ Richer Training Data ✨ We have released high-quality visual reasoning data for perspective questions with rendered images from the new perpsective as reasoning here: Reasoning Data Train your models on this and let us know how it works!
  • [2026.2.4] Check out an updated version of SAT that should work with current versions of Huggingface datasets and python versions: Data v2
  • [2026.2.4] We released a Qwen2.5-VL model trained on a SAT and Video-R1 mixture as a strong baseline. Check it out here: 🤗 Model

SAT provides dynamic spatial training experiences to Multimodal Language Models: requiring reasoning about ego and object motions that go beyond simple static relationships in existing datasets.

SAT Teaser

Abstract

Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships and not dynamic awareness of motion and space---i.e. reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos---even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.

Approach

SAT Approach We take actions in a 3D simulator and use privileged 3D information about the assets. We use natural language descriptions of the assets and make question-answer pairs based on how the 3D nature of the scene changes with the actions taken.

Simulated Training Data

SAT data

Results

Open-source MLMs struggle on dynamic spatial reasoning as well as large proprietary models

SAT Tasks
Simulated SAT data improves spatial performance on real static benchmarks
SAT Tasks
Fine-tuning on image-based SAT also improves performance videos, VSI-Bench (Yang et al, 2024)
SAT Tasks
Dynamic movements further help static perception!
SAT quality
Annotation quality is more important than realism!
SAT quality

BibTeX

@misc{ray2025satdynamicspatialaptitude,
      title={SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models}, 
      author={Arijit Ray and Jiafei Duan and Ellis Brown and Reuben Tan and Dina Bashkirova and Rose Hendrix and Kiana Ehsani and Aniruddha Kembhavi and Bryan A. Plummer and Ranjay Krishna and Kuo-Hao Zeng and Kate Saenko},
      year={2025},
      eprint={2412.07755},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.07755}, 
    }