Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

minimind

minimind: An open-source project for training a small language model from scratch and understanding its complete training pipeline.

What problem it solves

Large language models can be difficult for individuals to train, deploy, or understand because of their scale and the abstractions used by many frameworks. The project addresses this by providing native PyTorch implementations, compact model structures, open datasets, and staged workflows for pretraining, fine-tuning, preference optimization, tool use, and agentic training.

Key features

  • Compact Dense and MoE model structures — The project provides Dense and MoE implementations of the MiniMind LLM structure, with model configuration maintained directly in the repository.

    Source: README, model/model_minimind.py

  • End-to-end training pipeline — The workflow covers pretraining, supervised fine-tuning, LoRA, DPO, PPO, GRPO, CISPO, tool use, agentic RL, adaptive thinking, and model distillation.

    Source: README, trainer

  • Native PyTorch algorithms — Core training algorithms and modules are implemented from scratch with native PyTorch rather than relying on high-level abstractions from third-party training libraries.

    Source: README

  • Open datasets and tokenizer training — The repository includes code for tokenizer training and documents the pretraining, SFT, preference, and agentic RL dataset formats and training-stage use cases.

    Source: README, dataset

  • Deployment and framework compatibility — The project includes command-line evaluation, a Streamlit chat interface, an OpenAI-compatible service, and compatibility with common inference and training frameworks.

    Source: README, eval_llm.py, scripts

Quick start

Copied verbatim from the project's README.

# 克隆仓库、安装依赖
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
# 方式1
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
# 方式2
git clone https://huggingface.co/jingyaogong/minimind-3
# 方式1:使用 Transformers 格式模型
python eval_llm.py --load_from ./minimind-3
# 方式2:基于 PyTorch 模型(确保./out目录下有对应权重)
python eval_llm.py --load_from ./model --weight full_sft

Who it may suit

Inference: may suit developers who want to inspect native PyTorch language-model internals, reproduce a small-model training workflow, or extend the provided pretraining and post-training stages.

Topics to discuss

  • Which parts of the native PyTorch implementation are most useful for learning the complete model-training lifecycle?

  • How should the default dataset combinations be adapted for different compute budgets and target tasks?

  • What additional examples or evaluation workflows would help community members reproduce and extend the project?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/jingyaogong/minimind#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/jingyaogong/minimind

Comments

…