Roofline for LLM Inference: Performance Budgets from Compute and Bandwidth
A personal-blog article, presented as the first installment of an eight-part “Inference INFRA Methodology” series, applies the Roofline model to GPU inference. It estimates latency with T ≥ max(W/F, Q/BW) and effective performance with P ≤ min(F, I×BW), using arithmetic intensity, the hardware ridge point and actual HBM traffic to diagnose GEMM and attention bottlenecks. H20 examples and the article’s own measurements distinguish when quantization, batching and operator fusion may help. A series schedule covers communication, distributed inference, hardware, attention, MoE, PD/EPD and core optimizations. The benchmarks are self-reported and have not been independently verified here.
The framework turns a vague impression that inference is slow into a latency budget and a diagnostic process for choosing optimization strategies. Its calculations and examples offer practical guidance, although the reported benchmark results come from a single self-published source.
从算术强度、硬件转折点和实际 HBM 流量出发，判断 GEMM 与 Attention 的瓶颈，并区分量化、合批和融合的适用条件。
本文为「推理 INFRA 方法论」系列第 1/8 篇。发布日期按 2026 年 1–11 月编排；内容统一整理于 2026-11-01，包含年内后续更新的模型与实验资料。
任何一段 GPU 上的计算，时间都花在两件事上：算（执行浮点运算）和搬（把数据在存储单元和计算单元之间搬运，通信也算在搬运内）。硬件对这两件事各有一个上限：算力 F（FLOP/s）和带宽 BW（Byte/s）。理想流水下两者重叠
T ≥ max( W/F , Q/BW )
↑算的时间 ↑搬的时间
通常来说在优化模型之前，需要解答几个问题：
① 性能的物理上限是多少？ P ≤ min(F, I×BW)，P为有效算力，F为算力上限，I×BW为带宽侧上限（算子的算术强度×带宽），时延下限 T ≥ max(W/F, Q/BW)。实测贴到这条线就该收手；离得远才有优化空间。它还能在没写一行代码时做时延预算——比如”这个模型在 H20 这张卡上，每生成一个词最快多少毫秒”是可以纸面算出来的，这直接服务于服务质量目标（SLO，Service Level Objective，即对时延的承诺指标）的制定。