WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
WM-VLM adds a world-model branch to vision-language models to generate intermediate visual states for spatial reasoning, outperforming baselines by up to 39.25 points on mental rotation tasks.
Published Sep 28, 2026▲ 11 on Hugging FaceCode ★ 1arXiv ↗
Only vote on papers you've read. Sign in with GitHub to vote.
WM-VLM delivers striking gains on synthetic spatial reasoning by generating verifiable intermediate visual states, with ablations confirming real dependence on them, though its world-model branch remains confined to controlled rotation benchmarks without physical deployment or non-world-model…
Abstract
Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional vision-language models (VLMs) reason primarily through language. We investigate whether VLMs can solve spatial problems by reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for reasoning. We programmatically construct spatial reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both language and visual space.