SparseDecoding

Headlines · 1

1. Oct 7
   SparseDecoding: decoding-aware pruning for faster LLM inference

   The paper introduces SparseDecoding, a training-free network pruning framework aimed at the decoding stage of LLM inference. It builds calibration matrices from layer-wise activations collected during dense-model autoregressive generation (excluding prefill) so the pruning objective matches decoding activations, and pairs this with an N:M sparse matrix-vector kernel using bitmask indexing and fixed-step traversal. The authors report that on Llama-3.1-8B, Llama-3.3-70B and Qwen3-14B/32B it outperforms fixed-text calibration on long-form generation benchmarks, with up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my SparseDecoding experienceAsk about SparseDecoding