Just deciding when a diffusion VLM looks at the image can cut decoding FLOPs to about a third 👀
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
📌 Overview
LOWN is a training-free, threshold-free decoding method for diffusion VLMs such as LLaDA-V. Instead of processing the image at every step, it cheaply checks whether a step needs the image at all.
🚧 The problem
・Diffusion VLMs use bidirectional attention, so KV caching does not apply and the image is recomputed at every step
・Visual tokens are 89–97% of the input, so most compute goes to re-processing an unchanged image
・Removing the image changes the unmasked token at only 22.8% of steps on average
🔧 Method
・Keep the predictions from a FULL (image-conditioned) forward as an anchor
・At each step, run a TEXT-ONLY forward with image tokens removed
・The longest prefix whose predictions match the anchor is "certified" and unmasked without a FULL forward
・If too few are certified, run FULL and unmask disagreement-first
📊 Results
・On LLaDA-V, 99.6% of baseline score at 0.353x decoding FLOPs
・With RedVTP, 0.223x FLOPs and the best score on all ten benchmarks
・Standalone, 57.4% less generation time on average
・With Fast-dLLM, 38–50% less generation time, though scores fall below Fast-dLLM alone
⚠️ Caveat
Savings shrink on chart and document reading, where visual dependence persists.
#
DiffusionLM# #
VLM#