SparseDecoding: decoding-aware pruning for faster LLM inference
The paper introduces SparseDecoding, a training-free network pruning framework aimed at the decoding stage of LLM inference. It builds calibration matrices from layer-wise activations collected during dense-model autoregressive generation (excluding prefill) so the pruning objective matches decoding activations, and pairs this with an N:M sparse matrix-vector kernel using bitmask indexing and fixed-step traversal. The authors report that on Llama-3.1-8B, Llama-3.3-70B and Qwen3-14B/32B it outperforms fixed-text calibration on long-form generation benchmarks, with up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.
It highlights a distribution-shift problem in Hessian-based pruning — calibrating on natural text but decoding with self-generated tokens — and tries to turn that fix into real speedups via a decoding-oriented sparse kernel.
Computer Science > Machine Learning
arXiv:2610.12327 (cs)
[Submitted on 8 Oct 2026]
Title:SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Authors:Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang
View a PDF of the paper titled SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference, by Qitong Wang and 5 other authors
#SparseDecoding#Llama-3.1-8B#Llama-3.3-70B#Qwen3-14B#Qwen3-32B#NVIDIA A100