学习式预测器改进大模型微调时长估算
Dharma-AI R&D 分析了 NVIDIA H200 集群上的272次生产级监督微调任务,总计约7,382个GPU小时。五折交叉验证显示,其CatBoost预测器的R²为0.98、平均绝对误差为1.71小时,而测试中的FLOP基线分别为0.31和12.41小时。该研究仅涵盖H200上的SFT任务,完整论文计划发布到arXiv。
Dharma-AI R&D 称,更准确的调度前时长估算可帮助共享集群运营方规划GPU分配与调度,但现有结果仅限于该机构在H200上开展的SFT工作负载。
Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.
A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.
GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.
Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.
We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched. So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total, and compared analytical estimates against learned predictors.
#Dharma-AI R&D#NVIDIA H200#CatBoost#Hugging Face Transformers#Unsloth#Calculon#XGBoost