AI Infrastructure on Amazon EKS

The problem

GPU capacity is expensive, scarce and easy to strand: nodes sit idle between jobs, a single workload can hold a whole accelerator, and standard Kubernetes gives you almost no visibility into what the GPU is actually doing.

Constraints

• GPU nodes are costly enough that always-on capacity is not an option
• Workloads are bursty — capacity has to appear and drain on demand
• Standard Kubernetes metrics say nothing about GPU utilisation or memory

Architecture

Key decisions

What was chosen, what it was chosen over, and why.

• Chose

  Karpenter NodePools keyed on nvidia.com/gpu

  Instead of

  Static GPU managed node groups

  Static groups pay for accelerators between jobs. Karpenter provisions on pending-pod demand across the g4dn and g6 spot families and consolidates when idle — which matters far more at GPU pricing than at CPU pricing.

• Chose

  GPU time slicing for sharing

  Instead of

  MIG or MPS

  All three were evaluated against VRAM limits. Time slicing needs no hardware partitioning support, so it works on the instance families in play. The cost is no memory isolation between tenants — acceptable for development and inference under one team, not for untrusted multi-tenancy.

• Chose

  GPU Operator owns the driver stack

  Instead of

  Baking drivers into a custom AMI

  Kernel modules, the container runtime hook, Node Feature Discovery and the device plugin are versioned and reconciled together, so a driver upgrade is a Helm value rather than an AMI rebuild and node roll.

• Chose

  DCGM exporter into the existing Prometheus and Grafana

  Instead of

  A separate GPU monitoring tool

  GPU telemetry becomes another Prometheus target on port 9400, so the same dashboards, alert rules and on-call paths apply. Utilisation and memory pressure are the two signals that say whether sharing is actually working.