NVIDIA releases 3B and 8B Nemotron-Labs-Sigma diffusion models
NVIDIA released Nemotron-Labs-Sigma models in 3B and 8B sizes, including instruction-tuned variants. According to the supplied model documentation, the 8B instruct model denoises Gaussian noise in a 16-dimensional continuous token-embedding space and decodes parallel token blocks with configurable DDPM or DDIM sampling steps. The documentation says the models are ready for commercial or non-commercial use. NVIDIA claims competitive reasoning accuracy with leading masked discrete diffusion language models, but the supplied report does not independently substantiate that comparison.
Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.
This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.
This model is ready for commercial or non-commercial use.
Highlights
- Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
- Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
- Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
- Standard
transformersloading viatrust_remote_code; no custom kernels, flash-attention, or training framework required for inference.
#NVIDIA#Nemotron-Labs-Sigma-8B#Nemotron-Labs-Sigma-8B-instruct#Nemotron-Labs-Sigma-3B#Nemotron-Labs-Sigma-3B-instruct