Reddit benchmark pits three NVFP4 engines for Qwen3.8-Flash-Next on one RTX 5090
A Reddit user posting in r/LocalLLaMA benchmarked three NVFP4 inference engines — basalt, capy-flash and ninfer-ext — running Qwen3.8-Flash-Next on a single NVIDIA RTX 5090 with 32 GB, sweeping context in 16k steps up to the full 262,144 tokens with MTP speculation and int8 KV cache and reporting zero prefix-cache hits. The post says basalt was fastest at every context depth at 8.1k–9.2k tok/s prefill and 176–265 tok/s greedy decode, capy-flash second at 6.8k–8.0k tok/s and 138–198 tok/s, and ninfer-ext third at 4.2k–6.3k tok/s and 73–106 tok/s. The author also reports that sampled decoding at temperature 1.0 cost 7–10% of decode throughput for every engine and nothing in prefill, and that tuning changed results substantially; the numbers are one user's measurements and have not been independently verified.
The post publishes full-context, single-GPU numbers for three community NVFP4 engines on the same model and hardware, and reports that per-engine tuning shifted decode throughput markedly — basalt went from 122/114 tok/s at 33k/131k context on stock settings to 248/228 tok/s after CPU worker and memory changes.
| Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines. Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode = generated tokens / decode time. The fastest engine in each row is in bold. Each data point is a fresh, non-repeating prompt: the servers reported zero prefix-cache hits. Engine configuration (after tuning) Greedy decoding, temperature 0 Prefill, tokens/s Decode, tokens/s Sampled decoding, temperature 1.0, top_p 0.95, top_k 20 Prefill, tokens/s Decode, tokens/s Summary across all 16 points MTP draft acceptance, mean over the 16 points Headline Tuning that changed the result submitted by /u/giveen [link] [comments] |
#Qwen3.8-Flash-Next#NVIDIA RTX 5090#basalt#capy-flash#ninfer-ext#Intel Core Ultra 9 285K