Karpathy's nanochat offers a low-cost LLM training harness
Andrej Karpathy's MIT-licensed nanochat repository provides a minimal, hackable experimental harness for training language models across tokenization, pretraining, fine-tuning, evaluation, and inference. It is designed to run on a single GPU node, with model size selected through a depth setting while other hyperparameters are calculated automatically. Karpathy says a GPT-2-capability model can be trained for about $48 in roughly two hours on an 8xH100 node, or for closer to $15 on a spot instance; the report does not independently verify these figures.
nanochat packages the full LLM training workflow into a readable, modifiable and directly usable research codebase. Its minimal design and claimed low training costs may make small-model experimentation more accessible, though the headline cost, speed and quality figures are developer claims rather than independently verified results.
The best ChatGPT that $100 can buy.
nanochat
nanochat is the simplest experimental harness for training LLMs. It is designed to run on a single GPU node, the code is minimal/hackable, and it covers all major LLM stages including tokenization, pretraining, finetuning, evaluation, and inference. For example, you can train your own GPT-2 capability LLM (which cost ~$43,000 to train in 2019) for only $48 (~2 hours of 8XH100 GPU node) and then talk to it over a simple CLI. On a spot instance, the total cost can be closer to ~$15. More generally, nanochat is configured out of the box to train an entire miniseries of compute-optimal models by setting one single complexity dial: --depth, the number of layers in the GPT transformer model (GPT-2 capability happens to be approximately depth 26). All other hyperparameters (the width of the transformer, number of heads, learning rate adjustments, training horizons, weight decays, ...) are calculated automatically in an optimal way.
#nanochat#Andrej Karpathy#GPT-2#H100#DCLM CORE#nanoGPT#modded-nanoGPT#Hugging Face#NVIDIA ClimbMix