Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Qwen3.8-27B benchmark finds 8-bit KV caches nearly halve cache use

AI summary

A community author tested 35 weight quantizations of Qwen3.8-27B in llama.cpp, ExLlamaV3 and vLLM with 16-bit, 8-bit and, where supported, 4-bit KV caches. The test set included tool-use examples and text from multiple domains, with contexts up to 64,000 tokens. In llama.cpp and ExLlamaV3, 8-bit caches nearly halved cache size and changed KL divergence by less than 0.00012 in either direction. vLLM’s FP8 cache increased KL divergence by 0.0004 to 0.0006. The author did not recommend 4-bit caches: they produced measurable KL penalties that grew with context, including about 15% to 30% more added KL divergence at 16,000 to 32,000 tokens than in the first 4,000 tokens. vLLM did not support a 4-bit KV cache option in the test.

In-depth insight

The measurements provide practical guidance for local inference users facing limited VRAM, showing that 8-bit KV caches can substantially reduce cache use with very small measured changes in output distribution. The results are from one community author’s test rather than independent laboratory validation.

#Qwen3.8-27B#llama.cpp#ExLlamaV3#vLLM#NVIDIA NVFP4#Qwen FP8

Original · 4 min readRead the original
Group chat

No comments yet. Start the conversation.