Ai2 replaces priority scheduler with fair-share GPU budgets and time-slicing contracts
Ai2's AI Infrastructure team says it replaced its priority-based cluster scheduler with GPU time budgets, hierarchical fair-share allocation and a time-slicing contract for dividing scarce compute among research projects. Ai2 manages thousands of NVIDIA H100, B200 and B300 GPUs in clusters ranging from 88 to 1024 GPUs, serving about 150 internal researchers, with submitted workloads requesting 2–3x the available capacity at any time. The post says the old setup led to GPU "squatting," priority inflation and manual negotiations over shutting down jobs, and that the new contract asks workloads to declare a minimum runtime during which they cannot be preempted before being requeued. Ai2 also reports a 74% reduction in repairs requiring human intervention, a self-reported figure, and says it is investigating whether capacity fragmentation could increase queue waits for the largest workloads.
Ai2 says the change shifted debates over how much GPU time each research project deserves from case-by-case operational work into a transparent administrative budgeting process.
Impactful scheduling for GPU clusters
Building a cluster scheduler to prioritize high-impact research while maintaining full occupancy
October 9, 2026
Jeremy Tryba - Ai2
On the AI Infrastructure team at Ai2, we’re responsible for providing the institute’s GPU compute capacity, specifically targeting large, distributed training workloads. We think about this task as a pyramid of four metrics that build on each other.