Diffusion Instruction Tuning
Lavender aligns vision-language model attention with Stable Diffusion during supervised fine-tuning, boosting accuracy up to 30% with minimal training data.
Published Feb 4, 2025▲ 2 on Hugging FacearXiv ↗
Only vote on papers you've read. Sign in with GitHub to vote.
Lavender delivers striking vision-language gains by aligning VLM attention with Stable Diffusion instead of adding encoders, yet its 30% claims remain hard to isolate from data curation since it needs 130k examples and lacks clear ablation…
Abstract
We introduce Lavender, a simple supervised fine-tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation models such as Stable Diffusion. Specifically, Lavender aligns the text-vision attention in the VLM transformer with the equivalent used by Stable Diffusion during SFT, instead of adapting separate encoders. This alignment enriches the model's visual understanding and significantly boosts performance across in- and out-of-distribution tasks. Lavender requires just 0.13 million training examples, 2.5% of typical large-scale SFT datasets, and fine-tunes on standard hardware (8 GPUs) in a single day. It consistently improves state-of-the-art open-source multimodal LLMs (e.g., Llama-3.2-11B, MiniCPM-Llama3-v2.5), achieving up to 30% gains and a 68% boost on challenging out-of-distribution medical QA tasks. By efficiently transferring the visual expertise of image generators with minimal supervision, Lavender offers a scalable solution for more accurate vision-language systems. All code, training data, and models will be shared at https://astrazeneca.github.io/vlm/.