Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Qwen FP8

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Qwen3.8-27B benchmark finds 8-bit KV caches nearly halve cache use

    A community author tested 35 weight quantizations of Qwen3.8-27B in llama.cpp, ExLlamaV3 and vLLM with 16-bit, 8-bit and, where supported, 4-bit KV caches. The test set included tool-use examples and text from multiple domains, with contexts up to 64,000 tokens. In llama.cpp and ExLlamaV3, 8-bit caches nearly halved cache size and changed KL divergence by less than 0.00012 in either direction. vLLM’s FP8 cache increased KL divergence by 0.0004 to 0.0006. The author did not recommend 4-bit caches: they produced measurable KL penalties that grew with context, including about 15% to 30% more added KL divergence at 16,000 to 32,000 tokens than in the first 4,000 tokens. vLLM did not support a 4-bit KV cache option in the test.

    r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my Qwen FP8 experienceAsk about Qwen FP8

Nobody has shared their experience with Qwen FP8 yet.