Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

NVIDIA releases 3B and 8B Nemotron-Labs-Sigma diffusion models

AI summary

NVIDIA released Nemotron-Labs-Sigma models in 3B and 8B sizes, including instruction-tuned variants. According to the supplied model documentation, the 8B instruct model denoises Gaussian noise in a 16-dimensional continuous token-embedding space and decodes parallel token blocks with configurable DDPM or DDIM sampling steps. The documentation says the models are ready for commercial or non-commercial use. NVIDIA claims competitive reasoning accuracy with leading masked discrete diffusion language models, but the supplied report does not independently substantiate that comparison.

Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.

This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.

This model is ready for commercial or non-commercial use.

Highlights

  • Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
  • Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
  • Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
  • Standard transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.

#NVIDIA#Nemotron-Labs-Sigma-8B#Nemotron-Labs-Sigma-8B-instruct#Nemotron-Labs-Sigma-3B#Nemotron-Labs-Sigma-3B-instruct

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.