Six Inference Engines Benchmarked on RTX 5090
A TensorSharp author benchmarked six inference engines running Qwen3.8 27B on one RTX 5090 32 GB, with and without DFlash2 speculative decoding. NInfer led speculative and aggregate concurrent throughput, while llama.cpp recorded the highest plain single-request decode rate; the tests also covered 32K prompts, image input, and a four-turn tool loop. The report attributes results to complete configurations because checkpoints, drafters, and KV formats differed, and its author disclosed being TensorSharp’s creator.
The results offer a detailed local-inference comparison, but they describe differing runtime configurations rather than runtime code alone, limiting conclusions about the engines themselves.
Disclosure: I’m the author of TensorSharp. This English summary was adapted and edited with LLM assistance from the full benchmark report.
The comparison covers llama.cpp, vLLM, SGLang, NInfer, velox-inferer, and TensorSharp on the same RTX 5090 32 GB, including decode throughput, long prompts, concurrent requests, image input, and a small tool loop.
Setup: Ubuntu 24.04, EPYC 9354, one engine running at a time. All requests used greedy decoding with thinking off. Context was configured at 40,960 tokens per sequence, with tested prompts up to approximately 32K. TensorSharp used its direct-CUDA backend, --backend cuda.
Before the numbers, the main qualification: the engines used different quantized checkpoints, drafters, and KV formats.
- TensorSharp / llama.cpp: the same NVFP4 GGUF target and Q4_K_M DFlash2 drafter.
- vLLM / SGLang: NVIDIA’s NVFP4/FP8 safetensors target and an FP8 drafter.
- NInfer / velox: a mixed NVFP4/FP8
.ninfertarget and Q8 drafter.
The results describe those complete configurations. They don’t isolate the performance of the runtime code alone.
#Qwen3.8 27B#DFlash2#RTX 5090#llama.cpp#vLLM#SGLang#NInfer#velox-inferer#TensorSharp