Ai2 published the results of replacing its GPU cluster scheduler in an October 9 report. A priority-based scheduler gave way to GPU-time budgets, hierarchical fair-share and a time-slicing contract.

The rollout went cluster by cluster from late July, and over the reported 30 days teams received 98% of the GPU time they were owed. Owed time is the allocation capped hour by hour at actual demand.

Cluster occupancy was 98% both before and after the change. These are operator observations, not an independent benchmark.

The starting point was a bottleneck in the old design. With demand at 2-3x supply, cost-free HIGH priority eventually covered 100% of workloads, and some users held GPUs with no-op jobs.

Hierarchical fair-share over time rather than machines

Ai2’s answer was to allocate a share of GPU time instead of handing teams GPUs. Managers divide budgets proportionally among projects and researchers, and the scheduler tracks occupancy over a sliding lookback window that defaults to 7 days.

Workloads from under-used allocations sort ahead of those from over-used ones. A time budget is a claim over a window, not a continuous reservation of the same devices.

Ai2 itself says the algorithm is not new. Slurm’s Fair Tree compares shares and usage among siblings under the same parent in an account tree.

Ai2 only cites this lineage; nothing shows that its scheduler runs Slurm’s implementation.

Minimum-runtime contract and the preemption boundary

At submission, a workload declares the minimum runtime it needs to make meaningful progress and is protected from preemption during it. Ai2 set the protection cap at 8 hours.

After that window, resumable workloads are requeued automatically. A minimum of zero marks the time as unallocated, uncharged to any budget and preemptible from the start.

The contract also let unhealthy hosts drain automatically as workloads reached their minimum runtime. Ai2 reports that repairs needing a human in the loop fell by 74%.

Volatile interactive sessions fared worse: once preempted after the 8-hour protection, their state had to be rebuilt by hand. Ai2 is also investigating capacity fragmentation that could lengthen waits for the largest jobs.

Trading protection length against restart cost

The design question is how large a unit of recoverable work to choose. A short window creates more chances to rebalance, but those chances are worth less if each restart discards progress.

A long window gives a job room to progress while delaying others waiting for the same hardware. An occupancy figure alone does not appear to decide between them.

A realistic pilot records checkpoint duration, restart time and work lost to interruption next to queue latency. The comparison holds when interactive and resumable workloads are counted separately.

Checking preemption reasons through operating metrics

Ai2’s report publishes no inspection commands or configuration keys. Checks therefore stay at the level of operating metrics.

For a similar policy, the first thing to check appears to be whether delivered versus allocated time per group, the moment a job became preemptible and the charged budget are visible in one view.

Who allocates budgets, and how an insufficient share can be challenged, weighs just as much. A transparent formula earns little operational trust if allocation decisions stay opaque.

What the 98% figures do not show

Three questions separate here. Was available hardware assigned, did each group receive the time it was owed, and did that time become useful computation?

Occupancy answers the first and demand-capped delivery the second. Neither proves the third, and this article does not reproduce Ai2’s scheduler or simulations.

Progress that survives preemption, more than a full cluster, is the real scorecard for this scheduler.