Qwen3.8-27B benchmark finds 8-bit KV caches nearly halve cache use
A community author tested 35 weight quantizations of Qwen3.8-27B in llama.cpp, ExLlamaV3 and vLLM with 16-bit, 8-bit and, where supported, 4-bit KV caches. The test set included tool-use examples and text from multiple domains, with contexts up to 64,000 tokens. In llama.cpp and ExLlamaV3, 8-bit caches nearly halved cache size and changed KL divergence by less than 0.00012 in either direction. vLLM’s FP8 cache increased KL divergence by 0.0004 to 0.0006. The author did not recommend 4-bit caches: they produced measurable KL penalties that grew with context, including about 15% to 30% more added KL divergence at 16,000 to 32,000 tokens than in the first 4,000 tokens. vLLM did not support a 4-bit KV cache option in the test.
The measurements provide practical guidance for local inference users facing limited VRAM, showing that 8-bit KV caches can substantially reduce cache use with very small measured changes in output distribution. The results are from one community author’s test rather than independent laboratory validation.