NVIDIA Boosts Ling-3.0-flash MoE Inference on DGX Spark
An NVIDIA Technical Blog post details three optimizations for Ling-3.0-flash on NVIDIA DGX Spark: the Humming MXFP4 MoE backend, online FP8 quantization of the LM head, and NEXTN multi-token-prediction speculative decoding. Profiling showed batch-size-1 decoding was memory-bound, with weight reads dominating, and the Mixture-of-Experts computation plus the output-layer projection standing out as the costliest operations. On a synthetic workload the baseline ran at 27.71 ms TPOT and 36.09 tok/s, while NEXTN MTP reached 9.72 ms and 102.88 tok/s; across the full 1,319-question GSM8K evaluation the three optimizations together raised decode throughput 42.5% and cut mean end-to-end latency 24.5%, with no statistically detectable accuracy change (96.89% to 96.82%). The post also warns that combining online FP8 LM-head quantization with MTP requires the draft model to share the entire LM-head module, otherwise the framework silently falls back to a configuration slower than baseline.
The report gives a deployable, measured optimization stack for MoE inference on DGX Spark, including a silent-degradation integration pitfall and a caveat that the 185.1% synthetic throughput gain does not establish gains on other workloads.
Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA DGX Spark, profiling showed that decoding was memory-bound: reading model weights drove execution time. Two operations stood out—the mixture-of-experts (MoE) computation and the output-layer projection that produces scores across the model’s vocabulary.
Those findings pointed to two ways to improve performance: reduce the weight traffic in each decode step and generate more tokens per verification round. This post examines three techniques that address those costs:
- Humming MXFP4 MoE kernels to improve execution of the routed-expert computation.
- Online FP8 quantization of the output layer, or LM head, to reduce its weight-read volume.
- Multi-token prediction (MTP) speculative decoding with NEXTN to draft and verify multiple tokens together.
This post explains how each optimization works, measures its effect, and examines an integration issue that can make the combined configuration slower than the baseline. Ling-3.0-flash serves as the implementation example, with code and deployment instructions in the Ling cookbook.
Model and baseline configuration
The evaluation used Ling-3.0-flash-fp4, a mixture-of-experts model with 124 billion total parameters and 5.1 billion active parameters per token. The model has 42 layers: 35 Kimi Delta Attention (KDA) linear-attention layers and seven multi-head latent attention (MLA) layers. Each MoE layer routes each token to eight of 512 routed experts, alongside one shared expert.
#NVIDIA#Inclusion AI#Ling-3.0-flash#NVIDIA DGX Spark#Humming#FlashInfer#SGLang#NEXTN#GSM8K#FlashInfer CUTLASS