ninfer-ext

Headlines · 2

1. Oct 23
   Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090

   The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.

   r/LocalLLaMA · 🔥 0
2. Oct 22
   Reddit benchmark pits three NVFP4 engines for Qwen3.8-Flash-Next on one RTX 5090

   A Reddit user posting in r/LocalLLaMA benchmarked three NVFP4 inference engines — basalt, capy-flash and ninfer-ext — running Qwen3.8-Flash-Next on a single NVIDIA RTX 5090 with 32 GB, sweeping context in 16k steps up to the full 262,144 tokens with MTP speculation and int8 KV cache and reporting zero prefix-cache hits. The post says basalt was fastest at every context depth at 8.1k–9.2k tok/s prefill and 176–265 tok/s greedy decode, capy-flash second at 6.8k–8.0k tok/s and 138–198 tok/s, and ninfer-ext third at 4.2k–6.3k tok/s and 73–106 tok/s. The author also reports that sampled decoding at temperature 1.0 cost 7–10% of decode throughput for every engine and nothing in prefill, and that tuning changed results substantially; the numbers are one user's measurements and have not been independently verified.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my ninfer-ext experienceAsk about ninfer-ext