Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090
The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.
The benchmark offers concrete single-GPU performance data for locally running Qwen 3.8 FN, but interpretation requires caution because the author develops Basalt, engine configurations differ, each request received only one cold run, and llama.cpp’s decode results are not directly comparable because MTP was disabled.
| Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording. I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results: Decode, short context Decode, short context Decode at context depth Decode at context depth Prefill at context depth Prefill at context depth All the numbers Best value per column in bold. Decode, tokens per second Prefill, tokens per second Time to first token, seconds Takeaways How it was run What each engine ran submitted by /u/jesdga95 [link] [comments] |
#Qwen3.8-Flash-Next NVFP4#Basalt#Infernix#capy-flash#Strata#ninfer-ext#ninfer#llama.cpp#RTX 5090#NVIDIA NVFP4