Modal: mixed-precision, mixed-accelerator samplers enable stable async RL
In a technical blog post, Modal reports that running reinforcement-learning samplers on mixed precisions and mixed accelerators can keep large-model RL stable while cutting cost. Six kinds of accelerators feed a single BF16 trainer on B200: B200 and B300 serve NVFP4 weights, H100 and H200 serve FP8 and BF16 weights, and A100 and RTX PRO 6000 serve BF16 weights, each running its own kernels. The run trains Qwen3.6-35B-A3B on agentic coding with up to 256K tokens of context, using the 2.7K repository tasks Xiaomi released with MiMo-V2.6, and evaluates on SWE-bench Pro V2 and its HARD-51 subset. The authors add masked importance sampling, score centering and top-p sampling with mask replay as off-policy corrections; uncorrected GRPO collapses soon after training starts on the mixed samplers, while SC+MIS with or without top-p trains stably, with pass@4 rising over roughly 100–140 steps and levelling around 65–75% on HARD-51 and rising from 89% to about 94% on the full set, at about 15% lower sampler cost per update for most of training.
If mixed-precision, mixed-accelerator sampling holds up, large-scale RL post-training could use cheaper inference hardware for sampling without destabilizing training.