vLLM adds day-0 support for Recursive's Flint 40B MoE
vLLM said it supports Recursive's Flint from day 0, with both the NVFP4 and BF16 checkpoints running on a single GPU. Flint is a 40B-parameter mixture-of-experts model that activates about 2.5B parameters per token, uses sliding-window attention in most of its 49 layers and carries a 262K-token context. vLLM reports that a single request decodes at roughly 500 tokens/s on one GB200 even with 100K tokens of context, and that its Flint-specific kernels cut NVFP4 decode from 4.81 ms to 1.78 ms and reduced a forward pass from 1,096 kernels to 346.
The post documents how a serving stack was retuned kernel by kernel for a sparse, mostly local-attention MoE, including a low-latency decode path for batches of up to eight requests.
vLLM supports Recursive's Flint from day 0. Both its NVFP4 and BF16 checkpoints run on a single GPU, and a single request decodes at about 500 tokens/s on one GB200, even with 100K tokens of context.
Flint is a 40B-parameter MoE that activates only about 2.5B parameters per token, with sliding-window attention in most layers and a 262K-token context. It ships as two separate checkpoints: NVFP4 (quantization-aware trained) and BF16.
Quick start
The day-0 Docker image vllm/vllm-openai:flint (x86_64 and arm64, CUDA 13.0) has Flint support built in:
# NVFP4 checkpoint (GGUF): config and tokenizer come from the BF16 repo
docker run --gpus all --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:flint \
RecursiveSI/flint-nvfp4-gguf \
--hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \
--enable-auto-tool-choice --tool-call-parser glm47 \
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none
# BF16 checkpoint
docker run --gpus all --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:flint \
RecursiveSI/flint \
--enable-auto-tool-choice --tool-call-parser glm47 \
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none