Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Qwen presents SpecOPD for speculative decoding

AI summary

Qwen researchers presented SpecOPD, a branch-parallel on-policy distillation framework that parallelizes draft sampling and target supervision across independent proposal-tree branches. The approach uses an E2E-TV surrogate to target acceptance length and was evaluated with 8B, 27B, and 125B targets, three drafter architectures, and four task domains. Qwen reports up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput versus sequential-rollout OPD, and up to 3.56× generation speedup over autoregressive decoding.

In-depth insight

SpecOPD directly addresses a central bottleneck in speculative decoding by parallelizing training supervision and targeting acceptance length, with Qwen reporting substantial inference-efficiency gains across multiple model and task configurations.

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD under native configurations, and up to 3.56× generation speedup over autoregressive decoding. submitted by /u/pmttyji [link] [comments]

#Qwen#SpecOPD#DFlash#DSpark#DFlash V2

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.