NarrowSpec enables narrower speculative-decoding drafts
A paper submitted to arXiv on November 18, 2026 proposes that speculative-decoding draft models need not have the same hidden width as their target models. The authors report that DFlash and EAGLE-3 drafts using half or one-quarter of the target width largely preserve token acceptance. In tests with Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4) running in SGLang, the authors report end-to-end decoding speedups of 1.06–1.12x on an NVIDIA B200 and 1.13–1.44x on a DGX Spark (GB10) at batch size 1 and temperature 0.
The reported results indicate that narrower draft models can improve inference efficiency while largely preserving acceptance in the tested configurations. The findings are currently author-reported results from a newly submitted preprint and have not been independently verified in the supplied source.
Computer Science > Computation and Language
arXiv:2611.23835 (cs)
[Submitted on 18 Nov 2026]
Title:NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding
Authors:Alagappan Valliappan, Ankit More, Dheevatsa Mudigere, Jatin Mitra
View a PDF of the paper titled NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding, by Alagappan Valliappan and 3 other authors
#NarrowSpec#DFlash#EAGLE-3#Qwen3-8B#GPT-OSS-20B#SGLang#NVIDIA B200#DGX Spark