NVIDIA发布3B与8B Nemotron-Labs-Sigma扩散模型
NVIDIA发布了3B和8B规模的Nemotron-Labs-Sigma模型,其中包括指令微调变体。据所提供的模型文档,8B指令模型在16维连续词元嵌入空间中对高斯噪声去噪,并通过可配置的DDPM或DDIM采样步骤并行解码词块。文档称这些模型可用于商业或非商业用途。NVIDIA声称其推理准确率可与领先的掩码离散扩散语言模型竞争,但所提供的报告并未独立验证这一比较。
Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.
This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.
This model is ready for commercial or non-commercial use.
Highlights
- Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
- Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
- Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
- Standard
transformersloading viatrust_remote_code; no custom kernels, flash-attention, or training framework required for inference.
#NVIDIA#Nemotron-Labs-Sigma-8B#Nemotron-Labs-Sigma-8B-instruct#Nemotron-Labs-Sigma-3B#Nemotron-Labs-Sigma-3B-instruct