C++17 Tutorial Explores Continuous Batching and Paged KV Caches
A personal C++17 learning repository teaches vLLM-style continuous batching and paged KV-cache scheduling without a GPU or third-party dependencies. The implementation covers request admission, preemption with recomputation, background-thread concurrency, copy-on-write, prefix caching, invariant checks, and differential testing. The author reports 61 unit tests and two stress tests, with CI configurations for Linux/GCC, Windows/MSVC, and ThreadSanitizer. The project is described as a personal learning reproduction unrelated to the official vLLM project and contains none of its source code.
The repository provides a concrete teaching example for studying LLM inference scheduling, KV-cache management, preemption, and concurrency. Its test results and CI coverage are author-reported, and it does not provide real GPU inference performance data.
手写 vLLM 分页 KV Cache(一):先从 Continuous Batching 调度器看全局
系列导读:本系列采用 Top-Down(自顶向下) 的顺序拆解 vLLM 核心机制——本篇先从最上层的调度器切入,搞清楚请求如何流转、显存不足时如何抢占重算、以及多线程下如何保证并发安全;后续两篇再向下深潜到底层的物理块切分(
BlockAllocator)与共享写时复制(CoW / Prefix Cache)。 全文代码来自我的学习仓库 learn-vllm-paged-kvcache:纯 C++17、零第三方依赖、无 GPU,cmake --build之后ctest就能把本文每个结论跑一遍(61 个单测 + 2 个压力测试,覆盖 Linux/GCC、Windows/MSVC、ThreadSanitizer 三套 CI)。 注:本项目为个人学习复现,与 vLLM 官方项目无关,不含其任何源代码。
0. 先说清楚要解决什么问题
假设显存池里只有 16 个物理块,4 个请求同时在跑。
静态批处理(Static Batching)的做法是:攒一个 batch,一起 prefill,一起 decode,直到最慢的那个生成结束才统一收工。这会带来三个直接浪费:
#learn-vllm-paged-kvcache#vLLM#Continuous Batching#BlockAllocator#Copy-on-Write#Prefix Cache