AI21 Cuts AI Workload Start Times 83% With Google Cloud AI Hypercomputer

AI21 reports major improvements in AI workload scheduling

AI21 says it has reduced the time high-priority AI workloads spend waiting to start by 83% after adopting Google Cloud AI Hypercomputer and changing how workloads are scheduled across its shared GPU infrastructure.

According to AI21, high-priority jobs that previously waited as long as 72 hours for enough contiguous compute capacity now wait about 12 hours.

The company also reports that manual scheduling interventions fell from approximately 20 per week to zero, while GPU fragmentation across its cluster declined from 15% to 8%.

AI21 attributes the changes to a combination of Google Kubernetes Engine (GKE), Kueue scheduling, Google Cloud AI Hypercomputer, and elastic capacity options.

The company says its total compute cost did not change as a result because its reserved GPU fleet was already intentionally operated at close to full utilization.

Instead, the primary change was how available compute was allocated and scheduled.

The core numbers reported by AI21

These figures are AI21's reported production results, as described in the Google Cloud customer story.

AI21's challenge: thousands of GPUs in a shared cluster

AI21 develops foundation models and agent optimization products, including the Jamba family of language models.

The company says its production workloads run on a shared Google Kubernetes Engine cluster that pools thousands of Google Cloud A3 instances powered by NVIDIA H100 Tensor Core GPUs and A3 Ultra instances powered by NVIDIA H200 Tensor Core GPUs.

Rather than assigning each team its own fixed slice of infrastructure, AI21 says the shared approach allows teams to draw from the available capacity across the fleet.

That approach can improve overall utilization, but it also makes scheduling more complicated.

When the cluster was highly utilized, simply finding free GPUs was not enough.

A large training job might require a substantial portion of the cluster at the same time. Even when individual GPUs were technically available, those resources could be distributed across different machines and therefore unusable for a workload requiring contiguous capacity.

This created two separate scheduling problems:

Contention: deciding which workload should receive available compute first.

Fragmentation: having free capacity scattered across machines in pieces too small to accommodate a larger job.

Before Kueue, scheduling depended on Slack

AI21 says its earlier scheduling process relied heavily on manual coordination.

Teams requesting capacity would post in a Slack channel called #gpu-resources and wait for other teams to respond.

That approach worked when the cluster had sufficient spare capacity.

As utilization approached the level AI21 wanted for its reserved fleet, however, every large workload could become a negotiation.

According to the company, team leads spent time resolving compute allocation disputes, while high-priority multi-node training jobs could remain blocked for up to 72 hours.

The problem was not always a lack of total capacity.

AI21 describes situations where enough GPUs were technically free but distributed across several nodes.

For example, the company illustrates a case where eight available GPUs were distributed as 1 + 1 + 4 + 2 across four nodes.

That represented eight free GPUs, but they could not accommodate an eight-GPU workload that required the resources to fit together appropriately.

Without an all-or-nothing admission mechanism, AI21 says the cluster could also enter a deadlock state in which machines held resources without allowing a larger workload to run.

AI21 evaluates different Kubernetes schedulers

AI21 says it evaluated several open-source batch scheduling options, including:

• Apache YuniKorn

• Volcano

• Kueue

The company says YuniKorn did not cover all of its use cases.

Volcano offered more features, according to AI21, but integrating it would have required replacing core Kubernetes scheduler components.

Kueue was selected because AI21 says it worked with standard Kubernetes, did not require replacing core components and did not require rewriting existing job specifications.

The scheduling layer was then paired with AI Hypercomputer's infrastructure capabilities through GKE.

AI21 says this gave it direct access to GPU hardware and drivers while avoiding the overhead of managing raw compute instances itself.

Kueue connects reserved and elastic capacity

AI21 says Kueue's integration with GKE also allowed its scheduling logic to work with Google Cloud capacity options including Spot VMs and Dynamic Workload Scheduler.

The company's reserved capacity remains the primary environment.

When that capacity is full, AI21 says the same scheduling logic can reach into elastic capacity rather than leaving workloads waiting solely because the reserved fleet is occupied.

This approach addresses both the scheduling and capacity-utilization sides of the problem.

The important distinction is that AI21 did not simply add more reserved GPUs to solve its scheduling challenge.

Instead, it changed how available capacity was admitted and allocated across workloads.

Admission Fair Sharing changes workload prioritization

Adopting Kueue also became an opportunity for AI21 to contribute requirements based on its production environment.

AI21 says it worked with the Kueue team as a design partner through its relationship with Google Cloud.

One requirement involved fair admission ordering across teams for multi-node jobs without preempting workloads that were already running.

The Kueue team subsequently built Admission Fair Sharing (AFS).

According to the source, AFS changes the admission queue to favor teams that have historically used less capacity without disrupting workloads already running.

This is different from simply allowing a new job to interrupt another workload.

Instead, the mechanism changes the ordering of pending workloads.

AI21 says this was one of the requirements that emerged from its production experience.

Topology Aware Scheduling tackles GPU fragmentation

AI21 also enabled Topology Aware Scheduling.

This feature makes Kueue aware of the physical layout of the cluster.

That matters for workloads that need GPUs to fit within a particular node configuration.

Instead of admitting a workload that technically has enough total GPUs available but cannot actually fit in the required topology, the scheduler can reject admission until the appropriate capacity becomes available.

This addresses the fragmentation problem directly.

For AI training workloads, total free capacity and usable free capacity are not necessarily the same thing.

AI21's experience illustrates why scheduling large GPU workloads requires awareness of how compute resources are physically arranged, not just how many resources remain unused.

The result: less waiting and lower fragmentation

AI21 says the Slack-based #gpu-resources process has now been archived.

Workloads, ranging from debug pods to multi-node training jobs and inference deployments, are queued, prioritized and scheduled automatically.

The company reports several changes:

High-priority job wait time

Previously, high-priority jobs could wait up to 72 hours.

AI21 says the wait is now 12 hours.

That represents an approximately 83% reduction in reported wait time.

Manual scheduling

Manual scheduling interventions fell from about 20 per week to zero, according to AI21.

GPU fragmentation

AI21 reports that cluster fragmentation declined from 15% to 8%.

Zombie jobs

The company says the previous problem of "zombie jobs", where workloads were partially admitted without having enough usable capacity to run, is now gone.

Compute cost

AI21 says these changes did not alter its total cost.

The company explains that its reserved fleet was already operated close to 100% utilization intentionally, meaning raw infrastructure spending was not the metric it was trying to optimize.

Instead, the focus was on improving how people and workloads interact with the existing compute fleet.

AI21's scheduling architecture

At a high level, the production workflow described by AI21 can be represented as:

AI Workloads

↓

Google Kubernetes Engine

↓

Kueue Scheduling

↓

Admission + Prioritization

↓

Topology-Aware Placement

↓

Reserved Capacity / Elastic Capacity

↓

GPU Workload Execution

AI21 says the environment supports different workload types, including debugging workloads, multi-node training runs and inference deployments.

The scheduler determines when and where workloads can be admitted based on available resources and scheduling requirements.

Why GPU fragmentation matters

One of the most useful details in AI21's account is the distinction between available capacity and usable capacity.

A cluster may report that several GPUs are free.

But if those GPUs are distributed across different nodes, a workload requiring a particular multi-GPU configuration may still be unable to start.

This creates an important scheduling constraint for large AI training workloads.

AI21's reported reduction from 15% fragmentation to 8% suggests that improving workload placement became an important part of increasing the practical usefulness of its existing fleet.

The company did not report that it increased the size of its reserved fleet as part of this change.

Instead, it focused on scheduling, admission and topology awareness.

Open source became part of the infrastructure feedback loop

AI21's experience also demonstrates how production users can influence open-source infrastructure projects.

The company says its requirements document for Kueue described scheduling behavior that it needed but that was not yet available.

That requirement contributed to the development of Admission Fair Sharing.

AI21 says it then used the resulting capability alongside Topology Aware Scheduling.

The company describes this as a production feedback loop in which operational requirements become input for the evolution of an open-source project.

This is particularly relevant for specialized AI infrastructure, where scheduling requirements can differ significantly from conventional application workloads.

What AI21's results mean for AI infrastructure

Source facts: AI21 reports that its Kueue-based scheduling system reduced high-priority workload wait times from up to 72 hours to 12 hours, eliminated approximately 20 weekly manual interventions, reduced fragmentation from 15% to 8%, and removed its reported zombie-job problem.

Analysis: The announcement highlights a broader industry shift toward treating GPU scheduling as an infrastructure optimization problem rather than simply a question of adding more compute.

For organizations operating shared AI clusters, the practical challenge can involve deciding how to allocate existing resources across competing workloads while accounting for multi-node requirements and physical topology.

AI21's experience also suggests that scheduling efficiency can affect engineering productivity even when the amount of compute remains unchanged.

In this case, the company says its goal was not to reduce raw infrastructure spending. Its reserved fleet was already intentionally highly utilized.

Instead, the reported benefit came from reducing the amount of human coordination required to use that infrastructure.

That distinction is important.

Better scheduling is not the same as more compute.

It can instead mean making existing compute easier to allocate and use.

Conclusion

AI21's deployment of Google Cloud AI Hypercomputer illustrates the scheduling challenges that emerge when AI workloads share a highly utilized GPU fleet.

The company says its earlier approach depended on manual coordination through Slack and could leave large multi-node jobs waiting for sufficient contiguous capacity.

By adopting Kueue with GKE, and using features such as Admission Fair Sharing and Topology Aware Scheduling, AI21 reports that it was able to automate workload admission and improve how its existing GPU capacity was allocated.

The reported results include an 83% reduction in high-priority workload wait time, a decline in manual interventions from approximately 20 per week to zero, and a reduction in GPU fragmentation from 15% to 8%.

AI21 also says the changes did not increase or decrease its total cost because the company was already intentionally operating its reserved fleet at close to full utilization.

The broader lesson from the source is that AI infrastructure efficiency is not only about acquiring more GPUs. Scheduling, topology awareness and workload prioritization can determine how effectively an existing fleet can serve competing AI workloads.

The results described in this article are reported by AI21 in the Google Cloud customer story and were not independently verified.