Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

18GB More VRAM Raises 125B MoE Decode Throughput to 256.2 tok/s

AI summary

A two-run, single-variable test on one RTX 5090 compared Qwen3.8-Flash-Next Q2_0 inference while other containers occupied about 18GB of VRAM versus when the GPU was available. Releasing that memory expanded the expert cache from 5,784 to 19,530 slots, raised measured decode throughput from 140.6 to 256.2 tok/s, and reduced energy use from 1.634 to 1.314 joules per token. The report says cross-content comparisons are limited and several capacity-planning figures are extrapolated rather than directly measured.

In-depth insight

The telemetry-rich test gives local AI operators a concrete view of how expert-cache capacity affects MoE inference speed and energy use, while its single-machine limits constrain broader comparisons.

中小企业跑125B 大模型:18GB 显存之差,到底值多少吞吐

作者:吴佳浩 撰稿时间:2026-10-24 测试环境:

  • GPU:RTX 5090(32GB VRAM,驱动 580.142,CUDA 13.0,PCIe Gen3 ×16,实测 13.9 GB/s)
  • CPU:Intel Xeon Gold 5218 ×2(32 物理核 / 64 线程,2 个 NUMA 节点,无 AVX512-BF16)
  • Memory:125.6 GiB RAM
  • OS:Ubuntu 20.04.3 LTS(kernel 5.15.0-58)
  • Engine:Strata v0.2.2(engine 0.1.47,CUDA 13.0,官方容器镜像)
  • Model:Qwen3.8-Flash-Next Q2_0(125B 级 MoE,24,576 专家槽)
  • 两轮配置完全相同:CONTEXT=32k / KV=int8 / PARALLEL=1 / VISION=cpu / MTP spec4
  • 唯一变量:第一轮同机其他容器占了 GPU1 的 18,106 MiB;第二轮停掉它们

引言

面向工程师与研究者的体系化分析。 第一篇讲清了机制。这一篇只做一件事:把机制里的每一段,用实测数据钉死。 我把同一台机器、同一份模型、同一套参数跑了两轮,唯一区别是显卡上有没有别人的 18 GB。然后我从引擎日志里挖出 2,890 条逐请求遥测(每条都带「专家缓存命中率」和「PCIe 溢出比例」),把「显存 → 吞吐」这条链逐段拆开。 你可以把这篇当成一份「容量规划参考表」:给定显存,我告诉你能期待多少 tok/s;给定速度目标,我告诉你需要多少显存。 注:全文所有性能数字均来自本机两轮真实压测(合计 1,507 次请求、1,362 个资源采样点、317 个服务端指标采样点),未经验证的推算会明确标注。因为阿拉好奇,所以停掉别人占的卡,一道来白相相——看看这 18 GB 是"锦上添花"还是"生死攸关"。

#Qwen3.8-Flash-Next Q2_0#NVIDIA GeForce RTX 5090#Strata v0.2.2#Intel Xeon Gold 5218

Original · 18 min readRead the original
Group chat

No comments yet. Start the conversation.