ninfer

Headlines · 3

1. 5d
   Six Inference Engines Benchmarked on RTX 5090

   A TensorSharp author benchmarked six inference engines running Qwen3.8 27B on one RTX 5090 32 GB, with and without DFlash2 speculative decoding. NInfer led speculative and aggregate concurrent throughput, while llama.cpp recorded the highest plain single-request decode rate; the tests also covered 32K prompts, image input, and a four-turn tool loop. The report attributes results to complete configurations because checkpoints, drafters, and KV formats differed, and its author disclosed being TensorSharp’s creator.

   r/LocalLLaMA · 🔥 0
2. Oct 24
   capy-flash v0.7.4 expands local Qwen3.8 inference

   capy-flash v0.7.4 is a standalone local inference engine for Qwen3.8-27B and Qwen3.8-Flash-Next, designed around NVIDIA NVFP4 quantization and multi-GPU tensor parallelism. The author reports broader single-GPU support, persistent context caching, faster long-prompt processing, and compatibility with Claude Code, Codex CLI, and OpenCode; the post also includes local throughput benchmarks, but these results have not been independently verified.

   r/LocalLLaMA · 🔥 0
3. Oct 23
   Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090

   The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my ninfer experienceAsk about ninfer