ai-infra-book: An open-source book that derives LLM inference and training system design through quantitative analysis of hardware constraints and model architectures.
What problem it solves
Developers often need to understand why model applications are slow or costly, but system decisions require reasoning across model architecture, hardware limits, memory, computation, communication, and scheduling. Easy-to-miss factors such as KV cache, bandwidth limits, and inter-device communication can shift estimates by orders of magnitude, so system design should begin from explicit physical constraints.
Key features
Constraint-driven system analysis — The book derives system designs by first defining task and quality requirements, then enumerating computation, storage, communication, and dependencies against hardware capacity, bandwidth, and compute limits.
Source: README
Broad AI infrastructure coverage — Twelve chapters connect model architecture and workloads with accelerator design, operators and runtimes, supernodes, data-center networking, inference and training systems, resource scheduling, and edge-cloud collaboration.
Source: README, manuscripts, case-studies
Reproducible calculation tools — A Python CLI can reproduce calculations from the book and estimate resource requirements with alternative models and inputs. Static calculations use only the Python 3.10+ standard library and require neither a GPU nor model weights.
Source: README, calculations
Experiments with documented conditions — Chapter-organized experiments include run instructions, input conditions, and result explanations. GPU-dependent work identifies its hardware and dependencies, while existing records can be reviewed without the required device.
Source: README, experiments
Multi-format and multilingual editions — The manuscript is available through an online site, PDF, and EPUB, with community translations in English, Traditional Chinese, and Russian. The repository also includes scripts for local website and book builds.
Source: README, website
Quick start
Copied verbatim from the project's README.
git clone https://github.com/bojieli/ai-infra-book.git
cd ai-infra-book
# 查看模型支持情况
python3 calculations/calc.py models
# 估算 Qwen3-8B 在 8192-token prefill 下的逐算子资源需求
python3 calculations/calc.py forward --model qwen3-8b --tokens 8192 --format mdpython3 -m venv .venv-site
source .venv-site/bin/activate
python -m pip install -r website/requirements.txt
python scripts/build_site.py
python scripts/check_site.py
python scripts/build_site.py --serveWho it may suit
This may suit developers who have called model APIs or run models locally and want to understand latency and cost, as well as systems, networking, and chip engineers, researchers, and students working on related problems.
Topics to discuss
Which hardware constraint most often invalidates quantitative estimates for LLM inference and training?
How should readers reproduce and document experiments when model versions, hardware, and inputs can affect results?
Which missing mechanism or tradeoff should receive deeper coverage in the manuscript?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/bojieli/ai-infra-book#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…