Basalt
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 6
- Basalt developer tests lossy MoE routing options
A Basalt developer reported an experiment with Qwen 3.8 Flash Next that tested two opt-in, lossy inference options: routing bias, which favors experts already on the GPU, and tail skip, which omits lower-ranked experts not resident on the GPU. Both reduce transfers from RAM but can change outputs. In a simulated small-card configuration, the shipped settings increased prose throughput by 56%, while ladder and fact scores fell by 5.2 and 2.3 points, respectively. The developer’s deliberately worst settings reduced fact accuracy from 38.1% to 2.7%. The test covered one model on one rig, used one seed for ladders and three for facts, and measured speed with single cold runs. Basalt ships both flags disabled by default and warns when they are enabled.
r/LocalLLaMA · 🔥 6 - Community post reports 5,500 tok/s prompt processing on one RTX 5080
A r/LocalLLaMA post reported that Basalt achieved 5,500 tokens/s in prompt processing while running IQ3_S Qwen3.8 Flash Next on a single RTX 5080. The author also cited peak generation rates of 275 tokens/s for code and tool calls and 190 tokens/s for reasoning traces, using 64GB DDR5-6400 memory and a Core Ultra 9 285K; no independent measurement was supplied.
r/LocalLLaMA · 🔥 0 - Qwen3.8 Flash Next IQ3_S benchmarked on RTX 5080
A Reddit user reported benchmarking Qwen3.8 Flash Next IQ3_S with Strata/Basalt on an RTX 5080 with 16GB VRAM, an Intel Core Ultra 9 285K, 62GB DDR5-6800, 262K context and an int8 KV cache. In 103 opencode agent-coding requests with contexts up to 130K, it averaged 5,011 tok/s for prompt processing and 153 tok/s for generation. A 15K–207K context sweep averaged 5,475 tok/s for prompt processing and 154.9 tok/s for generation, with a peak of 5,662 tok/s and 186.6 tok/s, respectively. The agent test recorded a generation peak of 262 tok/s, the report’s best single second, exceeding the title’s “up to 250 tok/s” claim.
r/LocalLLaMA · 🔥 0 - Seven Qwen 3.8 FN Inference Engines Benchmarked on One RTX 5090
The developer of Basalt benchmarked seven Qwen 3.8 FN inference engines on an RTX 5090 with 128 GB of RAM. The report’s results placed Basalt and Infernix ahead overall: Basalt recorded 354 tokens per second for structured output and 190 tokens per second at 25K context, while both reached a 3.94-second time to first token at 25K. Each engine converted NVIDIA’s NVFP4 weights with its own tool and was configured according to its documentation. Every engine received the same seven greedy requests, with one cold run per request and no warm-up.
r/LocalLLaMA · 🔥 0 - Reddit benchmark pits three NVFP4 engines for Qwen3.8-Flash-Next on one RTX 5090
A Reddit user posting in r/LocalLLaMA benchmarked three NVFP4 inference engines — basalt, capy-flash and ninfer-ext — running Qwen3.8-Flash-Next on a single NVIDIA RTX 5090 with 32 GB, sweeping context in 16k steps up to the full 262,144 tokens with MTP speculation and int8 KV cache and reporting zero prefix-cache hits. The post says basalt was fastest at every context depth at 8.1k–9.2k tok/s prefill and 176–265 tok/s greedy decode, capy-flash second at 6.8k–8.0k tok/s and 138–198 tok/s, and ninfer-ext third at 4.2k–6.3k tok/s and 73–106 tok/s. The author also reports that sampled decoding at temperature 1.0 cost 7–10% of decode throughput for every engine and nothing in prefill, and that tuning changed results substantially; the numbers are one user's measurements and have not been independently verified.
r/LocalLLaMA · 🔥 0 - Developer runs DeepSeek V4.1 Flash IQ1_S on RTX 5090 at 53.6 tok/s
A developer posting on r/LocalLLaMA reports running DeepSeek V4.1 Flash 1017 at Unsloth's UD-IQ1_S quantization in Basalt, an inference engine they are developing, on an RTX 5090 plus an RTX 5060 Ti 16 GB and 128 GB of DDR5. With all 107 GiB of experts held in RAM and the 5090 used as an expert cache, the setup reached 53.6 tok/s decode and about 610 tok/s prefill, versus roughly 20 tok/s for a llama.cpp PR #30817 baseline on the same hardware and quantization. Letting the 5060 Ti compute some experts raised decode to 63.5 tok/s, but the author reports no perplexity or KL-divergence quality measurements yet, only greedy-match agreement with llama.cpp on test prompts.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my Basalt experienceAsk about Basalt
Nobody has shared their experience with Basalt yet.