Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

C++17 Project Implements vLLM-Style Paged KV Cache Mechanics

AI summary

The second installment of the personal learn-vllm-paged-kvcache project implements and explains physical block allocation, block tables, slot mapping, reference counting, and prefix-cache reuse for paged KV Cache management. Written in dependency-free C++17, it uses deterministic integer simulation instead of GPU tensors; its author reports 61 unit tests, two stress tests, and CI across Linux/GCC, Windows/MSVC, and ThreadSanitizer, without independently verified results.

In-depth insight

The project provides a CPU-only, testable reference for core vLLM-style KV Cache memory-management concepts, but it is an independent learning reproduction rather than an official vLLM implementation or release.

手写 vLLM 分页 KV Cache(二):深入物理块分配、Block Table 与 Slot Mapping

系列导读:本系列采用 Top-Down(自顶向下) 的顺序拆解 vLLM 核心机制。在第一篇《手写 Continuous Batching 调度器》中,我们站在全局视角看清了请求如何流转、显存不足时如何抢占重算;本篇向下深潜一层,把“KV Cache 在显存里如何切块、如何通过页表寻址、以及物理块分配器如何管理生命周期”从零推演一遍;第三篇再展开跨请求共享与写时复制(CoW)。 全文代码来自我的学习仓库 learn-vllm-paged-kvcache:纯 C++17、零第三方依赖、无需 GPU,cmake --build 之后 ctest 就能把本文每个结论跑一遍(61 个单测 + 2 个压力测试,覆盖 Linux/GCC、Windows/MSVC、ThreadSanitizer 三套 CI)。 注:本项目为个人学习复现,与 vLLM 官方项目无关,不含其任何源代码。


0. 先看一眼要解决的问题

Transformer 在 decode 阶段有一个核心特征:生成第 ttt 个 token 时,前面 t−1t - 1t−1 个 token 的 Key 和 Value 向量完全不变。因此没有任何推理引擎会每步重算整个历史前缀——把算过的 K/V 缓存下来,每步只计算新增那一个 token 的 K/V,再与历史缓存做注意力计算(Attention)。

#learn-vllm-paged-kvcache#vLLM

Original · 19 min readRead the original
Group chat

No comments yet. Start the conversation.