int8-vllm fork adds support for Nemotron 3.5 Bolt 74B
A community maintainer said the int8-vllm fork of vLLM now supports original Nvidia Nemotron 3.5 Bolt 74B checkpoints. The update adds fused NVFP4 and IQ2_xxs dequantization, a configurable INT8 weight cache, and AMD Infinity Storage model loading. The maintainer said the latter cut Bolt's initial load time from about seven minutes to two minutes in his test; the supplied report does not independently verify the claim.
The update provides an INT8-oriented community option for serving Nemotron 3.5 Bolt 74B on older AMD and Nvidia hardware, according to the maintainer. Its reported performance, numerical quality and loading improvements have not been independently verified in the supplied report.
Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.
https://github.com/curvedinf/int8-vllm
This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.
In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.
I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.
Ive been running it in production for a couple weeks and it has been very stable. Enjoy!
submitted by /u/1ncehost [link] [comments]
#int8-vllm#Nemotron 3.5 Bolt 74B#vLLM#ROCm#AMD Infinity Storage#MI100#MI210#MI250#NVIDIA Volta#NVIDIA Ampere