Diffusion-based vision-language models (such as LLaDA-V) recompute the visual tokens for the entire image at every denoising step, while visual tokens account for 89–97% of the input. The authors found that removing the image only changes the tokens being revealed in an average of 22.8% steps. At each step, LOWN first predicts the masked tokens without looking at the image, and only actually processes the image when this prediction is inconsistent with the prediction from the previous step when the image was viewed. Consistent parts are considered "certified" and can be directly reused; the number of tokens revealed corresponds to the number of tokens certified.
Likes 91 · Code Stars 7
Reasons
- Training-free and no threshold tuning required: The authors claim that LOWN retains 99.6% of LLaDA-V's score across ten benchmarks, using only 35.3% of the decoding FLOPs on average.
- Composable: The authors claim it can be generalized to other LLaDA-based diffusion vision-language models, is orthogonal to visual token pruning, and can further accelerate speed under KV caching.
- Run comparisons with two commands: In the COCO bear example in the README, for the same image, LLaDA-V uses approximately 1930 TFLOPs, while LOWN uses approximately 812 TFLOPs, a relative 0.421 (the authors state this is an analytical estimate for the decoder and LM head, based on a single-image example).
Usage
- Environment: Linux, Python 3.11, and an NVIDIA GPU with 32 GB or more of VRAM (PyTorch CUDA 12.6).
- Run: git clone github.com/hyeon9698/LOWN → bash setup.sh (creates a local .venv; automatically downloads LLaDA-V and SigLIP2 weights on first run) → bash evaluate.sh --gpu 0, with results and FLOPs written to outputs/result.json.
- Use your own image: bash evaluate.sh --gpu 0 --method lown --image path_to_image --prompt "Describe this image."
Risks
- Only applicable to diffusion-based vision-language models like LLaDA-V; not applicable to common autoregressive vision-language models.
- The README provides a single-image demo and FLOPs estimates, but no reproduction scripts for the ten benchmarks, nor wall-clock latency; the repository does not specify a license (it contains LLaDA-V code, subject to its license).
- The score retention and compute savings figures are as reported by the authors.
Open · Code · Paper Page