vLLM Marlin

Headlines · 1

1. Oct 21
   Ant Group and vLLM open-source Humming, a quantized-inference kernel library

   Ant Group and the vLLM community have open-sourced Humming, a general-purpose kernel library for quantized inference, hosted at github.com/vllm-project/humming. The project says it supports NVIDIA GPU architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, and uses a unified quantization representation covering integer and floating-point weight/activation formats, bit widths and scaling schemes for dense and MoE workloads. The authors report that Humming matches or beats vLLM Marlin, FlashInfer and DeepGEMM on many workloads in their own GEMM microbenchmarks, and say it exploits two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that are not exposed in the PTX type system.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my vLLM Marlin experienceAsk about vLLM Marlin