NVIDIA Links Full-Stack Codesign to Higher AI Memory Efficiency
NVIDIA published two technical posts on November 5 describing a “full-stack codesign” approach that designs GPUs, CPUs, networking, storage and software together to increase useful output per byte of memory during AI inference. The company says HBM is the fastest, most expensive and most contested tier, where model weights and KV cache compete, while CPU memory, local SSDs and shared storage trade speed for capacity and lower cost per bit. The approach includes NVFP4 for weights and KV cache, weight distribution across up to 72 GPUs in NVL72 systems, large-capacity LPDDR in the Vera CPU, Dynamo KV-cache offloading and reuse, CMX context memory storage, and keeping RAG indexes outside HBM. Citing its own SemiAnalysis AgentX results, NVIDIA says Vera Rubin NVL72 delivers up to 30 times the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. A separate comparison says Vera Rubin delivers up to 30 times the throughput per dollar at 125 tokens per second per user, while HBM capacity per GPU increased about 3% from Blackwell Ultra. These figures have not been independently verified in the supplied reports.
The posts shift the AI infrastructure discussion from raw HBM capacity toward useful tokens per watt and per dollar, while detailing memory-hierarchy techniques such as NVFP4, Dynamo, Vera CPU and CMX that infrastructure builders could evaluate. The headline performance multiples remain vendor-reported without independent confirmation.
Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term.
NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack.
But the approach isn’t to simply pack more memory onto the GPU. It’s full-stack codesign. NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory.
That matters because an AI factory runs on a memory and storage hierarchy. GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier — with model weights and KV cache competing for limited capacity. Below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit.
How much useful work a byte does depends on how well the whole hierarchy is used: shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it’s needed.
#NVIDIA#Vera Rubin NVL72#GB300 NVL72#NVFP4#NVIDIA Dynamo#NVIDIA Vera CPU#NVIDIA CMX#SemiAnalysis AgentX#DeepSeek R1#H200#NVIDIA BlueField-4