六款推理引擎在RTX 5090上的基准测试
TensorSharp作者在单张RTX 5090 32 GB上测试了六款运行Qwen3.8 27B的推理引擎，并对比启用和禁用DFlash2推测解码的表现。NInfer在推测解码和并发总吞吐量上领先，llama.cpp的单请求普通解码速度最高；测试还覆盖32K提示词、图像输入和四轮工具调用。由于量化检查点、推测解码器和KV格式不同，结果反映的是完整配置，作者同时披露自己是TensorSharp的开发者。
该测试为本地推理提供了详细参考，但因各引擎配置不同，不能将其结果直接解释为运行时代码本身的性能差异。
Disclosure: I’m the author of TensorSharp. This English summary was adapted and edited with LLM assistance from the full benchmark report.
The comparison covers llama.cpp, vLLM, SGLang, NInfer, velox-inferer, and TensorSharp on the same RTX 5090 32 GB, including decode throughput, long prompts, concurrent requests, image input, and a small tool loop.
Setup: Ubuntu 24.04, EPYC 9354, one engine running at a time. All requests used greedy decoding with thinking off. Context was configured at 40,960 tokens per sequence, with tested prompts up to approximately 32K. TensorSharp used its direct-CUDA backend, --backend cuda.
Before the numbers, the main qualification: the engines used different quantized checkpoints, drafters, and KV formats.
- TensorSharp / llama.cpp: the same NVFP4 GGUF target and Q4_K_M DFlash2 drafter.
- vLLM / SGLang: NVIDIA’s NVFP4/FP8 safetensors target and an FP8 drafter.
- NInfer / velox: a mixed NVFP4/FP8
.ninfertarget and Q8 drafter.
The results describe those complete configurations. They don’t isolate the performance of the runtime code alone.
#Qwen3.8 27B#DFlash2#RTX 5090#llama.cpp#vLLM#SGLang#NInfer#velox-inferer#TensorSharp