Quantized LLM Serving Gains Depend on Batch Size
A meta-analysis of public vLLM and SGLang measurements reports that 4-bit quantization delivers roughly 1.35–2.12× speedups at batch size 1, while W4A16 gains fall to about 1.01–1.02× in the cited batch-256 tests. One H20 measurement found FP8 retained a 1.68× gain at a long prompt, but the analysis rates the large-batch W4A16 finding low confidence and notes that FP8 was not measured in that comparison. It also finds that all reported HumanEval differences fall within estimated benchmark noise, while selected AWQ and GPTQ cases show GSM8K losses of about 4 points. The underlying timings come from heterogeneous engine versions, GPUs, metrics and workloads rather than a unified benchmark.
The review gives inference teams a workload-sensitive basis for evaluating quantization. Its public measurements support considering INT4 for small-batch latency and memory reduction, and FP8 for high-throughput serving, while emphasizing that deployment-specific validation is necessary.
TL;DR:三位独立研究者把 9B–20B 开源模型在 vLLM/SGLang 上量化的全部公开实测翻了个底朝天,发现所有计时都能用一条「内存带宽 vs 计算」的规则解释:小 batch 时 4-bit 量化提速约 2 倍,大 batch 时增益几乎归零,这时候 FP8 才是还能打的配方。同时,大量被广泛引用的「量化掉点」数字(包括所有 HumanEval 差值)其实落在基准的采样噪声里。文末附 7 条量化评测报告清单,建议部署前对照。
资源
先问一个可能正戳中你的问题:你们团队要把一个 14B 的开源模型放上 vLLM 或 SGLang 自托管,老板问「量化能省多少卡」,算法同学提醒「量化会不会掉点」——然后你去搜资料,得到的是一堆互相打架的答案:模型卡说 AWQ 提速 2.8 倍几乎无损,某基准报告说 GPTQ 掉了 4 分,还有帖子说量化根本没用。
谁对?最近挂在 arXiv(AILET Vol. 1110)上的一篇 letter 给出了目前最接近「标准答案」的东西:Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs。三位独立研究者(Kadvani、Faujdar、Tallam)把截至 2026 年 10 月 5 日能核实的公开测量全部收拢,逐条对照原始来源核验,得出了一些可能让你意外的结论。
#vLLM#SGLang#Qwen2.5-14B-Instruct#Qwen3-14B#Gemma 3 12B#Mistral NeMo#AWQ#GPTQ#FP8#gpt-oss-20b#NVIDIA A100#NVIDIA H100