七种 Qwen 3.8 FN 推理引擎在单块 RTX 5090 上的基准测试
AI 总结
Basalt 开发者在一块配备 128 GB 内存的 RTX 5090 上测试了七种 Qwen 3.8 FN 推理引擎。报告称 Basalt 与 Infernix 整体领先：Basalt 的结构化输出解码速度为每秒 354 个 token，25K 上下文下为每秒 190 个 token；两者在 25K 上下文下的首字延迟均为 3.94 秒。各引擎均使用自己的工具转换 NVIDIA 的 NVFP4 权重，并按照自身文档进行配置。所有引擎都运行相同的七个贪心请求，每个请求只进行一次冷启动测试，没有预热。
深度见解
该基准测试为在本地运行 Qwen 3.8 FN 提供了具体的单卡性能数据，但解读时需要谨慎：作者是 Basalt 的开发者，各引擎配置不同，每个请求仅进行一次冷启动测试，而且 llama.cpp 关闭了 MTP，因此其解码结果无法直接比较。
| Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording. I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results: Decode, short context Decode, short context Decode at context depth Decode at context depth Prefill at context depth Prefill at context depth All the numbers Best value per column in bold. Decode, tokens per second Prefill, tokens per second Time to first token, seconds Takeaways How it was run What each engine ran submitted by /u/jesdga95 [link] [comments] |
#Qwen3.8-Flash-Next NVFP4#Basalt#Infernix#capy-flash#Strata#ninfer-ext#ninfer#llama.cpp#RTX 5090#NVIDIA NVFP4