Developer releases sdft_zero3 for self-distillation under ZeRO-3
Developer ifiaposto released sdft_zero3, a plug-and-play GitHub repository combining self-distillation with an EMA teacher, ZeRO-3, and PEFT. The author says it includes LoRA adapter switching for teacher and student passes, resumable EMA teacher checkpoints, and DeepSpeed configurations intended to address memory spikes. The author reports testing the method with Qwen3.5 and ToolAlpaca.
The repository offers a concrete implementation for distributed self-distillation, but its capabilities and scalability claims remain author-reported and have not been independently validated.
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.
We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).
Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:
-> robust SDFT implementation that runs under ZeRO-3
-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes
-> Saving and resuming EMA teacher checkpoint
#sdft_zero3#TRL#DeepSpeed ZeRO-3#PEFT#LoRA#DeepSpeed#Qwen3.5#ToolAlpaca#GitHub