AI Infrastructure on Amazon EKS
Production-style AI infrastructure on Amazon EKS for provisioning, sharing and observing NVIDIA GPUs — Karpenter, GPU Operator, CUDA workloads and DCGM observability.
- GPU Operator
- Karpenter
- CUDA
- Time Slicing
- Observability
The problem
GPU capacity is expensive, scarce and easy to strand: nodes sit idle between jobs, a single workload can hold a whole accelerator, and standard Kubernetes gives you almost no visibility into what the GPU is actually doing.
Constraints
- GPU nodes are costly enough that always-on capacity is not an option
- Workloads are bursty — capacity has to appear and drain on demand
- Standard Kubernetes metrics say nothing about GPU utilisation or memory
Architecture
Observability
Driver & device lifecycle
Capacity
Interactive Hardware Simulator
GPU Allocation & VRAM Sharing
Telemetry & Isolation Profile
g4dn.xlarge Spot pricing per the project README; dedicated mode is that range × 4 cards.
Key decisions
What was chosen, what it was chosen over, and why.
Chose
Karpenter NodePools keyed on nvidia.com/gpu
Instead of
Static GPU managed node groups
Static groups pay for accelerators between jobs. Karpenter provisions on pending-pod demand across the g4dn and g6 spot families and consolidates when idle — which matters far more at GPU pricing than at CPU pricing.
Chose
GPU time slicing for sharing
Instead of
MIG or MPS
All three were evaluated against VRAM limits. Time slicing needs no hardware partitioning support, so it works on the instance families in play. The cost is no memory isolation between tenants — acceptable for development and inference under one team, not for untrusted multi-tenancy.
Chose
GPU Operator owns the driver stack
Instead of
Baking drivers into a custom AMI
Kernel modules, the container runtime hook, Node Feature Discovery and the device plugin are versioned and reconciled together, so a driver upgrade is a Helm value rather than an AMI rebuild and node roll.
Chose
DCGM exporter into the existing Prometheus and Grafana
Instead of
A separate GPU monitoring tool
GPU telemetry becomes another Prometheus target on port 9400, so the same dashboards, alert rules and on-call paths apply. Utilisation and memory pressure are the two signals that say whether sharing is actually working.
Production resilience & failure modes
Failure mode this architecture has to account for
Time slicing shares compute, not memory — there is no per-pod VRAM limit, so one pod over-allocating pushes its neighbours on the same card into a CUDA OOM they did nothing to cause.
How it's handled
This isn't prevented, it's made visible and bounded by choice of tenant: DCGM exporter scraped per-pod (not through the Service VIP, which would silently sample one random GPU node) surfaces memory pressure before it saturates, so time slicing is scoped to workloads that can tolerate a noisy neighbour. Where hard isolation is a requirement rather than a nice-to-have, the answer is MIG, not a mitigation bolted onto time slicing.