Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

NVIDIA SEED Cuts Prefill Work and Compresses Hybrid LLM State

AI summary

NVIDIA researchers proposed SEED, a method that uses shallower layers and lightweight emitters to write memories needed for later generation while retaining full-depth computation during decoding. On Qwen3.8-Flash-Next, parameter-dependent prefill computation fell about 25%, from 12.1 to 9.1 GFLOP per non-final input token. SEED also stores recurrent state in a compressed representation, reducing Qwen3.8-Flash-Next's per-request state from 113.2 MB to 29.2 MB immediately after compression and averaging 43.4 MB over the compression cycle. The report describes tests on five hybrid models and an arXiv preprint. In GB300 GPU and SGLang tests replaying coding-agent sessions, NVIDIA reported 16%–42% higher peak throughput per GPU; gains varied by deployment and workload.

In-depth insight

SEED targets two distinct costs in hybrid-model serving: reducing deep computation during prefill and reducing recurrent-state storage during decoding. The report also states that it does not compress the attention KV Cache, so its memory benefit can be smaller in long contexts where the KV Cache dominates.

机器之心 2026-11-19 12:09 北京

NVIDIA 拆解混合模型的"读写不对称性"。

Qwen3.8-Flash-Next 有 48 层。处理一段输入时,SEED 让除最后一个以外的 token 只运行前 31 层,再由轻量的记忆写入组件,为后续 17 层生成 KV Cache 和循环状态。到了生成阶段,每个 token 仍然运行完整的 48 层。

这个改动利用了一个区别:为后续生成写下记忆,所需的深层计算可以远少于预测下一个 token。SEED 在五种已训练的混合大模型上测量了这种差别,并把它用于减少预填充计算。

#NVIDIA#SEED#Qwen3.8-Flash-Next#SGLang#Nemotron-3.5-Lightning#Nemotron-3.5-Super#Kimi-Linear#GLM-5.3-Flash#Gated DeltaNet#Kimi Delta Attention#Mamba-2#GB300

Original · 10 min readRead the original
Group chat

No comments yet. Start the conversation.