MMFlash targets faster speculative decoding for vision-language models
The MMFlash authors released code and draft-model artifacts for speeding up speculative decoding in vision-language models, with support for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it targets. They report that MMFlash adaptively prioritizes visually dependent tokens and uses sparse attention for long-video inputs, achieving lossless decoding speedups of up to 3.58× across multimodal benchmarks. These benchmark results come from the authors and have not been independently validated by the Reddit report.
MMFlash targets a specific bottleneck in multimodal inference: draft models accept fewer image-dependent tokens, and its released artifacts are designed to improve those predictions without changing the target model’s output.
| Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively. ⭐ Updates 💪 Highlights 💻 Models The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results. submitted by /u/pmttyji [link] [comments] |
#MMFlash#Qwen3.5-4B#Qwen3.5-9B#Qwen3.8-27B#Gemma 4 E4B-it#DFlash#EAGLE-3#SGLang#vLLM