Humming

Headlines · 2

1. 5d
   NVIDIA Boosts Ling-3.0-flash MoE Inference on DGX Spark

   An NVIDIA Technical Blog post details three optimizations for Ling-3.0-flash on NVIDIA DGX Spark: the Humming MXFP4 MoE backend, online FP8 quantization of the LM head, and NEXTN multi-token-prediction speculative decoding. Profiling showed batch-size-1 decoding was memory-bound, with weight reads dominating, and the Mixture-of-Experts computation plus the output-layer projection standing out as the costliest operations. On a synthetic workload the baseline ran at 27.71 ms TPOT and 36.09 tok/s, while NEXTN MTP reached 9.72 ms and 102.88 tok/s; across the full 1,319-question GSM8K evaluation the three optimizations together raised decode throughput 42.5% and cut mean end-to-end latency 24.5%, with no statistically detectable accuracy change (96.89% to 96.82%). The post also warns that combining online FP8 LM-head quantization with MTP requires the draft model to share the entire LM-head module, otherwise the framework silently falls back to a configuration slower than baseline.

   NVIDIA Technical Blog · 🔥 1
2. Oct 21
   Ant Group and vLLM open-source Humming, a quantized-inference kernel library

   Ant Group and the vLLM community have open-sourced Humming, a general-purpose kernel library for quantized inference, hosted at github.com/vllm-project/humming. The project says it supports NVIDIA GPU architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, and uses a unified quantization representation covering integer and floating-point weight/activation formats, bit widths and scaling schemes for dense and MoE workloads. The authors report that Humming matches or beats vLLM Marlin, FlashInfer and DeepGEMM on many workloads in their own GEMM microbenchmarks, and say it exploits two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that are not exposed in the PTX type system.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my Humming experienceAsk about Humming