minimind: An open-source project for training a small language model from scratch and understanding its complete training pipeline.
What problem it solves
Large language models can be difficult for individuals to train, deploy, or understand because of their scale and the abstractions used by many frameworks. The project addresses this by providing native PyTorch implementations, compact model structures, open datasets, and staged workflows for pretraining, fine-tuning, preference optimization, tool use, and agentic training.
Key features
Compact Dense and MoE model structures — The project provides Dense and MoE implementations of the MiniMind LLM structure, with model configuration maintained directly in the repository.
Source: README, model/model_minimind.py
End-to-end training pipeline — The workflow covers pretraining, supervised fine-tuning, LoRA, DPO, PPO, GRPO, CISPO, tool use, agentic RL, adaptive thinking, and model distillation.
Source: README, trainer
Native PyTorch algorithms — Core training algorithms and modules are implemented from scratch with native PyTorch rather than relying on high-level abstractions from third-party training libraries.
Source: README
Open datasets and tokenizer training — The repository includes code for tokenizer training and documents the pretraining, SFT, preference, and agentic RL dataset formats and training-stage use cases.
Source: README, dataset
Deployment and framework compatibility — The project includes command-line evaluation, a Streamlit chat interface, an OpenAI-compatible service, and compatibility with common inference and training frameworks.
Source: README, eval_llm.py, scripts
Quick start
Copied verbatim from the project's README.
# 克隆仓库、安装依赖
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple# 方式1
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
# 方式2
git clone https://huggingface.co/jingyaogong/minimind-3# 方式1:使用 Transformers 格式模型
python eval_llm.py --load_from ./minimind-3
# 方式2:基于 PyTorch 模型(确保./out目录下有对应权重)
python eval_llm.py --load_from ./model --weight full_sftWho it may suit
Inference: may suit developers who want to inspect native PyTorch language-model internals, reproduce a small-model training workflow, or extend the provided pretraining and post-training stages.
Topics to discuss
Which parts of the native PyTorch implementation are most useful for learning the complete model-training lifecycle?
How should the default dataset combinations be adapted for different compute budgets and target tasks?
What additional examples or evaluation workflows would help community members reproduce and extend the project?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/jingyaogong/minimind#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…