Inclusion AI

Headlines · 1

1. 5d
   NVIDIA Boosts Ling-3.0-flash MoE Inference on DGX Spark

   An NVIDIA Technical Blog post details three optimizations for Ling-3.0-flash on NVIDIA DGX Spark: the Humming MXFP4 MoE backend, online FP8 quantization of the LM head, and NEXTN multi-token-prediction speculative decoding. Profiling showed batch-size-1 decoding was memory-bound, with weight reads dominating, and the Mixture-of-Experts computation plus the output-layer projection standing out as the costliest operations. On a synthetic workload the baseline ran at 27.71 ms TPOT and 36.09 tok/s, while NEXTN MTP reached 9.72 ms and 102.88 tok/s; across the full 1,319-question GSM8K evaluation the three optimizations together raised decode throughput 42.5% and cut mean end-to-end latency 24.5%, with no statistically detectable accuracy change (96.89% to 96.82%). The post also warns that combining online FP8 LM-head quantization with MTP requires the draft model to share the entire LM-head module, otherwise the framework silently falls back to a configuration slower than baseline.

   NVIDIA Technical Blog · 🔥 1

Experience and discussion from the community

Share my Inclusion AI experienceAsk about Inclusion AI