GPU Scheduling on Kubernetes

Kubernetes doesn't know what a GPU is. It never has. What it schedules is an integer count of an opaque resource that something else, a device plugin, told it about. Every GPU platform you've used, from a managed EKS node group to a bare-metal cluster, is built on that one fact plus a handful of add-ons that make it usable in practice.

This article covers the mechanism itself: how a GPU becomes visible to the scheduler at all, the two competing APIs for requesting one, how to share a single card between pods, and why the scheduler that ships with Kubernetes still can't run a distributed training job on its own. If you want the applied version on a specific cloud, our guide to deploying vLLM on Amazon EKS with GPUs walks through the same concepts in a real cluster.

How the Scheduler Sees a GPU

A GPU becomes visible to Kubernetes through the device plugin framework, stable since Kubernetes 1.26. A device plugin runs as a DaemonSet on each GPU node, registers with the kubelet over a local gRPC socket, and reports what it finds under a vendor-namespaced extended resource name, nvidia.com/gpu being the one you'll see everywhere. The kubelet folds that into the node's Allocatable capacity, and from that point the scheduler treats it exactly like CPU or memory: a number to bin-pack pods against.

The catch is how narrow that number is. Extended resources are integer-only, can't be overcommitted, and can't be split across containers in the same pod. A request for nvidia.com/gpu: "1" reserves one whole card. There's no way to ask for a quarter of a GPU through this path. That limitation is exactly what MIG, time-slicing, MPS, and DRA each exist to work around, in different ways.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: app
      image: your-image
      resources:
        limits:
          nvidia.com/gpu: "1"   # integer only, no overcommit, no fractions

The toleration matters as much as the resource request. GPU nodes are almost always tainted on purpose, so an ordinary CPU-only pod doesn't accidentally land on hardware that costs ten times as much per hour. Without a matching toleration, your own GPU pod would be blocked by the same taint.

Extended Resources vs Dynamic Resource Allocation

Extended resources have one real weakness: a bare integer can't express "give me an H100, not an A10G" or "these two containers can share one GPU." Dynamic Resource Allocation is Kubernetes' answer to that.

DRA is now generally available, with the feature gate locked on, so you can't disable it even if you try. It first appeared as alpha in Kubernetes 1.30.

Instead of a plain number, DRA asks you to write what you actually need. A cluster admin defines a DeviceClass that filters available hardware using CEL expressions: by GPU model, memory size, whatever attributes the driver exposes. Workloads reference that class through a ResourceClaim or a ResourceClaimTemplate, and a DRA-aware driver publishes what's available on each node through a ResourceSlice.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: high-performance-gpu
spec:
  selectors:
    - cel:
        expression: "device.attributes['gpu.nvidia.com'].productName == 'H100'"
---
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod-dra
spec:
  containers:
    - name: app
      image: your-image
      resources:
        claims:
          - name: gpu
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: h100-claim-template

DRA also supports sharing a claimed device across multiple containers or even multiple pods, something extended resources flatly can't do. NVIDIA moved its GPU DRA driver into Kubernetes SIGs, dropping the Beta label, which is a reasonable signal that this is where GPU scheduling is heading rather than a side experiment.

The honest caveat: the scheduler still doesn't support preemption for DRA claims. A high-priority pod that needs a DRA-claimed device just waits if a lower-priority pod is holding it, even if it's otherwise entitled to bump it. This is a documented limitation, not a temporary bug.

On EKS specifically, Karpenter added DRA support in v1.14.0 (July 2026), and it correctly schedules DRA pods onto GPU nodes that already exist. What it still can't do is provision a brand new node for a pod whose GPU request is expressed purely as a ResourceClaim, with no matching node already sitting there. Since provisioning capacity on demand is one of Karpenter's main reasons for existing on GPU workloads, that gap is not a footnote, it can decide your whole architecture. A managed node group or self-managed nodes running the DRA driver directly still cover that case.

For most teams today, extended resources remain the simpler default. DRA is worth adopting once you specifically need its device filtering or cross-container sharing.

Sharing One GPU Between Pods

All three of the common options come from NVIDIA's stack, and they sit at genuinely different points on the isolation-versus-flexibility line, not just different configuration flags on the same thing.

A rough decision rule: the same model behind many replicas leans toward time-slicing, because you don't need to protect replicas from each other. Genuinely mixed, untrusted workloads on shared infrastructure lean toward MIG, because a bad actor or a crash in one slice can't reach another. MPS sits in between, for workloads you trust but want more real concurrency from than time-slicing gives you.

One detail worth knowing about MPS: the device plugin advertises MPS capacity under nvidia.com/gpu resources, and only with full GPUs. There's no fractional MPS resource you can request through the standard plugin path.

A fourth option: hardware-level isolation. If MIG's hardware partitioning isn't enough (say, you're in a regulated environment with hard multi-tenancy requirements), Kata Containers with GPU support runs each pod in a dedicated microVM, with the GPU exposed via VFIO passthrough. Isolation is enforced at the hardware virtualization boundary rather than the driver layer. This is heavier than the other three options and worth it only when the isolation guarantee itself is the requirement.

DRA can express any of these as a device attribute once your driver supports it, which is part of why it's the direction NVIDIA and the Kubernetes project are both investing in.

Queueing and Gang Scheduling for Multi-GPU Jobs

Plain kube-scheduler places pods one at a time, independently. That's fine for a stateless web service. It's a real problem for a distributed training job or a multi-node inference deployment that needs, say, 8 GPUs across 2 nodes to do anything useful: the scheduler can happily place half the pods, leave the rest pending, and let the job sit there holding GPUs it can't use while doing no useful work.

Kueue runs as CRDs on top of the existing scheduler, no second scheduler binary needed, and handles quota and admission: a job either gets all the resources it needs admitted together, or it waits in a queue. Kueue's native support for JobSet and LeaderWorkerSet means it can gang-admit the kind of multi-pod groups our vLLM on EKS guide covers for multi-node serving.

KAI Scheduler has been accepted as a CNCF Sandbox project, moving from an NVIDIA-governed tool toward community-developed standard. It handles DRA for GPUs, topology-aware placement, and hierarchical PodGroups for gang scheduling, using dominant resource fairness to share GPUs across teams. It's worth watching if Kueue and Volcano together feel like more than your cluster needs.

Autoscaling the Nodes Underneath

Everything above assumes the GPU nodes already exist. Getting them to exist, and disappear again when idle, is a separate job.

On EKS specifically, Karpenter is the autoscaler most teams reach for. Cluster Autoscaler struggles with instance heterogeneity, Spot diversification and consolidation at GenAI scale, which is exactly the territory GPU fleets live in. Karpenter provisions GPU nodes on demand from a NodePool you define, then removes them when idle. It works with the Kubernetes scheduler rather than replacing it: the scheduler places pods on nodes, and Karpenter provisions capacity for the pods the scheduler can't fit. GKE and AKS solve the same problem with their own tools, not Karpenter; our guide to open-weight LLMs on GKE covers GKE's version of this.

A few operational notes that matter more than they look. NodePool limits are your cost guardrail: always set spec.limits (e.g., nvidia.com/gpu) plus a per-namespace ResourceQuota, so a runaway Deployment or a misconfigured HPA can't scale GPUs without bound. Inference pools and training pools want different consolidation policies too: inference does well with consolidationPolicy: WhenEmptyOrUnderutilized and a short consolidateAfter, cheap to lose a replica, expensive to keep an idle node, while training usually wants a longer window or a WhenEmpty policy so Karpenter doesn't evict mid-run and destroy hours of work. And if you're mixing GPU and Neuron hardware, provision two NodePools from day one rather than one; future hardware migration becomes a cost experiment rather than a re-architecture.

None of this changes anything above: the device plugin and scheduler layer works identically no matter which autoscaler put the node there.

What This Costs

Scheduling decisions and cost decisions are more connected than they look. An idle GPU node costs the same whether kube-scheduler put a pod on it or not, so getting the scheduling layer right (tight autoscaling, sensible sharing where isolation allows it, gang admission so jobs don't sit half-started) is directly what keeps a GPU fleet from quietly overspending. We've worked through the actual GPU pricing and utilization math in our GPU costs for LLM inference article, and the enterprise buy-versus-build version of that question in self-hosted LLMs vs API costs.

Where I'd Start

Related Articles

Deploying vLLM on Amazon EKS with GPUs

Deploy vLLM on Amazon EKS with GPU nodes: the AL2023 vs Bottlerocket AMI choice, device plugin setup, a working Deployment, autoscaling and multi-node serving.

By Shalomi Umeshika • Meetrix.io

GPU Costs for LLM Inference: AWS vs GCP (2026)

GPU costs for LLM inference on AWS vs GCP in 2026: L4, A100 and H100 hourly prices, cost per million tokens, Spot and commitment discounts, and hidden costs.

By Shalomi Umeshika • Meetrix.io

Self-Hosted LLMs: When They Beat API Costs for Enterprises

Self-hosted LLMs vs API costs for enterprises: real September 2026 prices, a breakeven formula, hidden costs on both sides, and when hybrid wins.

By Shalomi Umeshika • Meetrix.io