Ant Group and vLLM open-source Humming, a quantized-inference kernel library
Ant Group and the vLLM community have open-sourced Humming, a general-purpose kernel library for quantized inference, hosted at github.com/vllm-project/humming. The project says it supports NVIDIA GPU architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, and uses a unified quantization representation covering integer and floating-point weight/activation formats, bit widths and scaling schemes for dense and MoE workloads. The authors report that Humming matches or beats vLLM Marlin, FlashInfer and DeepGEMM on many workloads in their own GEMM microbenchmarks, and say it exploits two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that are not exposed in the PTX type system.
The post argues that the memory and compute savings from quantization only become lower latency and higher throughput with efficient kernels, so a shared kernel library spanning many quantization schemes and GPU generations targets a growing maintenance burden in inference stacks.
Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference. Humming offers the following features:
- Broad device support: Supports NVIDIA GPU architectures from Turing onward, along with CUDA-compatible devices such as T-Head's Zhenwu 810E.
- Extensive quantization support: A unified architecture supports diverse combinations of weight and activation formats for dense and MoE workloads, covering a wide range of integer and floating-point types, bit widths, and scaling schemes.
- Strong execution performance: Significantly outperforms existing implementations from vLLM Marlin, FlashInfer, and DeepGEMM across many representative workloads.
Repository: github.com/vllm-project/humming
Background
As models and inference workloads grow, memory capacity, memory bandwidth, and compute costs increasingly constrain deployment. This is especially apparent in MoE models: each token activates only a subset of experts, but the full set of weights still needs to be stored. DeepSeek-V3, for example, has 671 billion parameters, with 37 billion activated per token. Quantization lowers the precision of weights and activations to reduce memory usage and data movement, and can also improve compute efficiency.
Quantization requirements vary across models and GPU generations. Common configurations include WNA16, W8A8, W4A8, and W4A4, where W and A specify weight and activation bit widths, and N denotes a variable weight bit width. Even at the same precision, schemes such as INT4 and NVFP4 differ in numeric format and scaling. Scaling granularity, scale representation, group size, and symmetric or asymmetric quantization further expand the configuration space.
#Ant Group#vLLM#Humming#NVIDIA#T-Head#Zhenwu 810E#vLLM Marlin#FlashInfer#DeepGEMM#Blackwell#DeepSeek-V3