Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Stanford CS336 Lecture 10 on LLM Inference Optimization

AI summary

A Stanford CS336 Lecture 10 report explains why autoregressive LLM inference can be slow and how to evaluate performance using TTFT, TPOT, latency, throughput, and memory usage. It covers KV Cache sizing, GQA and MLA, quantization, speculative decoding, continuous batching, and PagedAttention. The report distinguishes Prefill’s generally compute-intensive characteristics from Decode’s bandwidth-sensitive bottlenecks and emphasizes that acceleration methods should be evaluated for model quality, stability, and cost. Its Llama 2 13B and H100 figures are idealized estimates derived from course formulas, not real-world service benchmarks.

In-depth insight

The report provides a systematic framework for diagnosing inference bottlenecks and selecting optimizations, while clearly distinguishing course-derived estimates from deployment benchmarks. It also stresses that speed improvements must be assessed alongside quality, stability, and business cost.

课程:Stanford CS336(2026 Spring),Lecture 10 — Inference 阅读目标:理解大模型推理为什么慢、性能如何衡量,以及 KV Cache、注意力结构、量化、投机解码和推理调度各解决什么问题。

一、核心结论与知识框架

一句话总结:大模型的自回归解码难以并行,往往受到显存带宽与 KV Cache 读取开销限制;优化推理需要同时关注模型结构、内存占用和请求调度。

graph TB
    A[大模型推理优化] --> B[性能分析]
    A --> C[模型优化]
    A --> D[解码算法]
    A --> E[推理系统]
    B --> B1[TTFT和TPOT]
    B --> B2[计算强度]
    B --> B3[吞吐量和延迟]
    C --> C1[减少KV Cache]
    C --> C2[量化]
    C --> C3[剪枝和蒸馏]
    D --> D1[投机解码]
    E --> E1[连续批处理]
    E --> E2[分页注意力]

#vLLM#SGLang#TensorRT-LLM#llama.cpp#DeepSeek-V2

Original · 16 min readRead the original
Group chat

No comments yet. Start the conversation.