Qwen FP8
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Qwen3.8-27B benchmark finds 8-bit KV caches nearly halve cache use
A community author tested 35 weight quantizations of Qwen3.8-27B in llama.cpp, ExLlamaV3 and vLLM with 16-bit, 8-bit and, where supported, 4-bit KV caches. The test set included tool-use examples and text from multiple domains, with contexts up to 64,000 tokens. In llama.cpp and ExLlamaV3, 8-bit caches nearly halved cache size and changed KL divergence by less than 0.00012 in either direction. vLLM’s FP8 cache increased KL divergence by 0.0004 to 0.0006. The author did not recommend 4-bit caches: they produced measurable KL penalties that grew with context, including about 15% to 30% more added KL divergence at 16,000 to 32,000 tokens than in the first 4,000 tokens. vLLM did not support a 4-bit KV cache option in the test.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my Qwen FP8 experienceAsk about Qwen FP8
Nobody has shared their experience with Qwen FP8 yet.