Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

ArtificialAnlys: DeepSeek V4.1 Flash agent throughput up to 1.5x on GB300 NVL72

AI summary

ArtificialAnlys reports that DeepSeek V4.1 Flash running on NVIDIA GB300 NVL72 gained up to 1.5x tokens per second per agent over the past month on its AA-AgentPerf benchmark, which tests datacenter systems running realistic agentic coding workloads. A single agent reached 848 tok/s, up from 596 (1.42x), while throughput per GPU rose roughly 1.6x at about 80 tok/s per agent. ArtificialAnlys attributes the gains to a newer vLLM version, new vLLM kernels for DeepSeek V4.1 on GB300, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and running prefill and decode on the same GPUs in NVIDIA’s updated Dynamo + vLLM setup.

In-depth insight

ArtificialAnlys attributes the higher measured throughput to updates across the inference serving stack, highlighting the role of software and system configuration in DeepSeek V4.1 Flash performance on GB300 NVL72.

DeepSeek V4.1 Flash speeds on GB300 NVL72 have increased by up to 1.5x tok/s per agent over the past month on AA-AgentPerf, our benchmark of datacenter systems running realistic agentic coding workloads

This improvement comes from a newer vLLM version, new vLLM kernels for DeepSeek V4.1 on GB300, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and running prefill and decode on the same GPUs in NVIDIA’s updated Dynamo + vLLM setup

➤ Faster speeds: 848 tok/s for a single agent, up from 596 (1.42x)

➤ Higher throughput: ~1.6x the throughput per GPU at ~80 tok/s per agent

#DeepSeek#DeepSeek V4.1 Flash#NVIDIA GB300 NVL72#vLLM#FlashInfer#NVIDIA Dynamo#ArtificialAnlys#AA-AgentPerf

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.