开发者发布ZeRO-3自蒸馏仓库
开发者ifiaposto发布了GitHub仓库sdft_zero3，将自蒸馏、EMA教师模型、ZeRO-3与PEFT整合为即插即用项目。作者称其支持教师与学生轮次之间的LoRA适配器切换、EMA教师检查点保存与恢复，并提供用于缓解显存峰值的DeepSpeed配置；已在Qwen3.5和ToolAlpaca上进行测试。
该项目为分布式自蒸馏提供了具体实现，但相关功能与扩展性目前仅由作者陈述，尚无独立验证。
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.
We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).
Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:
-> robust SDFT implementation that runs under ZeRO-3
-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes
-> Saving and resuming EMA teacher checkpoint
#sdft_zero3#TRL#DeepSpeed ZeRO-3#PEFT#LoRA#DeepSpeed#Qwen3.5#ToolAlpaca#GitHub