Adaptive Information Flow
Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/
Problem
Text tokens attend to too many irrelevant visual tokens
- Text-to-image attention is often spatially dispersed over both relevant and irrelevant regions
- Irrelevant visual tokens introduce noisy information into the decoding path
Source: Khayatkhoei et al., ICLR 2025
Analysis
Measuring Visual Token Dynamics
Not all visual tokens contribute equally
Removing distracting interactions can improve prediction
Method
Estimating visual-token importance from token dynamics
- Important tokens tend to become strongly activated at particular layers
- Irrelevant tokens show more irregular activation patterns
- Measures this randomness using entropy over the layer-wise token dynamics
Information Flow Modulation
- Selected high-entropy tokens are masked only in their interaction with text tokens
- Visual-to-visual attention remains unchanged
Adaptive Mask Ratio Selection
- Rank visual tokens by token entropy
- For each candidate mask ratio, mask high-entropy tokens first. Then recompute the visual attention distribution
- Select the ratio that maximizes the shift from the original distribution
Discussion
Computational Cost
- Low additional inference overhead (approximately equivalent to generating one additional token)
- No retraining or auxiliary network is required
Limitation
- Performance may depend on prompt quality
- AIF relies on visual-to-text interaction induced by the user prompt
- Long or indirect prompts may make relevant visual regions harder to identify
- In such cases, token dynamics may provide a less reliable importance signal