A Practical Framework for Profiling and Tuning vLLM Inference
This tutorial explains how to evaluate LLM inference services by balancing throughput, request latency, and memory utilization, with latency further divided into TTFT and TPOT. It recommends profiling bottlenecks before tuning vLLM parameters, and maps options such as batched-token limits, concurrency, prefix caching, chunked prefill, and speculative decoding to different workload goals. The author also warns that some listed vLLM commands, environment variables, and metric names are invalid, and that NVIDIA-specific tooling must be replaced when running on Ascend NPUs.
The article provides a practical troubleshooting framework for choosing vLLM configuration changes based on workload priorities and measured bottlenecks rather than trial-and-error parameter tuning.
① 这篇讲什么——怎么量化地衡量和优化推理服务:吞吐、延迟、显存三者怎么权衡,看哪些指标、调哪些参数。
② 它的核心观点对不对——**"不可能三角"的思路是对的,而且正好解释了你之前那个疑问**:为什么 P 节点 token 预算大、并发小,D 节点预算小、并发大——那就是这套权衡在 PD 分离里的具体落地。
③ ⚠️ 但它的 API 和指标名基本是编的——python -m vllm.entrypoints.benchmark、VLLM_PROFILING_ENABLED=1、还有那些 vllm_xxx 指标名(少了 vllm: 前缀、好几个根本不存在)都别当真。而且它是 NVIDIA 视角,你是昇腾 NPU,工具名要换。
下面从零讲。
一、先建立三个指标的直觉(餐厅版)
| 指标 | 是什么 | 餐厅比喻 | 谁最关心 |
|---|---|---|---|
| 吞吐量 Throughput | 单位时间总共吐多少 token(Tokens/s) | 一小时总共出多少道菜 | 成本/离线批处理 |
| 延迟 Latency | 单个请求多久完成 | 一桌客人等多久上菜 | 交互/聊天体验 |
| 显存利用率 | 显存用了多少(决定能装多少 KV) | 仓库堆得多满 | 能不能同时接更多客 |