Kueue GPU Topology Aware Scheduling: Install, Configure, Run AI Jobs

If you run multi-GPU AI jobs on Kubernetes, you may have seen a job stay stuck in Pending even when your cluster has plenty of free GPUs. This usually happens because the free GPUs are spread across many nodes instead of sitting together on one node or rack. This guide shows you how to fix this with Kueue GPU topology-aware scheduling. 

The default Kubernetes scheduler places pods one by one. It does not know that an AI job's pods need to sit close together, so GPUs can talk to each other fast. A basic Kueue setup has the same problem. It only checks if there are enough free GPUs in total, not where those free GPUs are.

This is where Kueue GPU topology-aware scheduling helps. It teaches Kueue your real layout, such as which node sits in which rack. This way, Kueue waits until one node or rack has enough free GPUs together. It prevents pods from spreading across nodes and prevents jobs from getting stuck half-started.

Kueue is an open-source tool that adds a queue in front of Kubernetes. Instead of every job fighting for pods at once, Kueue holds jobs until there is enough quota, then lets them run in a fair order. 

Kueue solves "too many jobs, not enough GPUs" through quotas. But quotas only count numbers, not location. Topology-Aware Scheduling fixes that. This is why Kueue GPU topology-aware scheduling matters for real AI clusters with more than a few nodes.

This guide is for anyone running GPU servers on Kubernetes who wants queueing and smart pod placement for training or inference jobs. We assume you already have:

• A Kubernetes cluster, such as K3s, kubeadm, or managed.
• At least two or three nodes with NVIDIA GPUs, with the NVIDIA GPU Operator already installed so nvidia.com/gpu shows up as an allocatable resource. If you have not done this yet, follow our guide on how to install the NVIDIA GPU Operator on a bare-metal K3s cluster.
• kubectl and helm installed and pointed at your cluster.
• jq installed on your workstation.

Note: If you plan to build a real multi-GPU cluster, PerLod's GPU dedicated servers give you full root access and predictable hardware, which makes rack-level labeling and topology testing much easier than on shared cloud instances.

Confirm your nodes are visible and that GPUs show up as allocatable resources:

You should see each GPU node listed with a gpus count matching its physical cards. If any node shows null, its GPU Operator setup is not complete, and you should fix that before continuing.

At this point, you should install the latest stable release of Kueue. Always check the Kueue releases page for the newest tag before you install, and change the version number below if a newer one is available.

You can install Kueue with the following kubectl command:

Wait for the controller to become ready:

If you prefer Helm, you can use this Helm command to install Kueue:

Check that the controller pod is running:

You should see a kueue-controller-manager pod with a Running status.

The TopologyAwareScheduling feature is what makes Kueue GPU topology-aware scheduling work. If you installed v0.19.4 above, it is already on; no extra steps needed. You can check it with this command:

If you do not see TopologyAwareScheduling=false anywhere in the output, the feature is active by default, and you can move to the next step.

Cloud providers usually add rack or zone labels to nodes automatically. On a bare-metal or dedicated GPU cluster, you must add these labels yourself so Kueue knows which physical group each node belongs to.

For this guide, consider three GPU nodes split across two racks:

Replace gpu-node-1, gpu-node-2, and gpu-node-3 with your real node names from kubectl get nodes. The nvidia.com/gpu.present label is added automatically by GPU Feature Discovery once the GPU Operator is installed. So you only need to add the rack label. Confirm the labels are set up correctly:

The Topology object tells Kueue how your data center is organized, from the biggest group down to every single node. Create the file with your desired text editor:

Add this content into the file:

With this, group nodes first by rack, then by individual hostname. The kubernetes.io/hostname label must always be the last level in the list. Apply it:

A ResourceFlavor tells Kueue which nodes to use for a given quota, and links those nodes to the Topology object you just made. Create the file with:

Paste this content:

Apply it with:

This is the key link for Kueue GPU topology-aware scheduling. The topologyName field turns a normal ResourceFlavor into a topology-aware one. Without it, Kueue only tracks quota, not where pods land.

The ClusterQueue sets the total GPU quota that all teams or jobs share. Create the file with:

Paste this content and change nominalQuota to match your real total GPU count:

Apply it:

The LocalQueue is the entry point your team submits jobs to inside a namespace. Create a namespace for AI jobs first, then the queue:

Add this content:

Apply and check that everything is registered:

At this point, you have a complete queueing setup for Kueue GPU topology-aware scheduling.

Now create a job file that asks for 4 GPUs, all on the same host:

Paste this content:

Apply it with:

In the fragmented state, only 2 free GPUs per node; this job stays Pending instead of splitting across two nodes. That's not a failure; it's Kueue protecting your job from bad placement. Once gpu-node-3 with 4 free GPUs together is free, Kueue puts the whole job there automatically.

Now check the Workload object Kueue created for your job:

Get the exact node assignment:

The output lists the exact node Kueue chose. If your job is still pending, check the reason:

Check the Events section and the QuotaReserved message. It will tell you if no rack or node has enough free GPUs yet. This confirms Kueue GPU topology-aware scheduling is checking real placement, not just totals.

To try softer modes, you can just change the annotation in your job file and submit it again:

With preferred, Kueue first tries to fit all pods in one rack. It only spreads them across racks if it really has to. This mode is a good default for jobs that run faster together but can still finish if split.

For background jobs that don't need speed, use this instead:

This tells Kueue to fill any free GPU slots it can find. Over time, this makes the small gaps across your cluster smaller instead of leaving them spread out everywhere.

Kueue GPU topology-aware scheduling knows where your free GPUs sit, not just how many there are. You've now installed Kueue, labeled your GPU nodes, built the Topology object, and set up quota with a ClusterQueue and LocalQueue. Pick required for tight training jobs, preferred when locality helps, and unconstrained for background work. That's what keeps your GPU cluster fast and fair instead of stuck.

We hope you enjoy this guide. For more detailed information, you can check the official Kueue Topology-Aware Scheduling Docs.