FinOps Kubernetes Operator
Kubernetes operator that scales non-production workloads to zero during declared sleep windows, with per-workload exclusions.
- Kubernetes Operator
- Python
- Kopf
- FinOps
The problem
Non-production clusters run all night and all weekend at full replica count, paying for capacity nobody is using — and the obvious fix, scaling things down, is exactly the kind of automation that causes an outage when it gets one workload wrong.
Constraints
- Must never scale down a workload someone depends on
- Teams need an escape hatch they control themselves
- The operator's own failure must be safe — no action is better than a wrong action
Architecture
Trigger
Reconcile
Act
Interactive Operator Simulator
Reconciliation & Schedule State
web-frontend
Namespace: qa-env
cart-api
Namespace: qa-env
critical-worker
exclude: trueAnnotation Bypass
Key decisions
What was chosen, what it was chosen over, and why.
Chose
Sleep schedules annotated on the namespace
Instead of
Per-workload schedules or a central config file
A team owns its namespace, so the schedule sits at the boundary the team already controls and every workload inside it inherits one intent rather than drifting apart.
Chose
System namespaces skipped unconditionally
Instead of
Relying on operators to annotate correctly
kube-system and friends can never be scaled to zero by this operator regardless of configuration. The failure mode of a cost tool has to be inaction, never an outage.
Chose
Original replica count stored before scaling down
Instead of
Restoring to a fixed default
Waking a workload has to return it to the size it actually was, not to whatever the chart shipped, or the operator silently resizes production-shaped environments overnight.
Also decided
Per-workload exclusion annotationnotA central allow-list maintained by the platform team
Opting a critical workload out has to be possible without a platform-team round trip. Cost automation only survives contact with users if the escape hatch is trivial.
Server-side field_selector when listing podsnotListing everything and filtering client-side
The API server does the filtering, so the operator does not pull every pod in the cluster into memory to count a handful. It also skips terminating pods, which would otherwise read as rogue workloads.
Timer-based reconciliation every 60 secondsnotEvent-driven triggers
The trigger is wall-clock time, not a cluster event. A periodic loop matches the shape of the problem and converges after any missed tick, restart or reschedule.
Kopf and PythonnotThe Go operator SDK and controller-runtime
The reconciliation logic is small and schedule-shaped, so Kopf keeps it short and quick to iterate. The trade is a heavier runtime and a smaller ecosystem than controller-runtime.
Production resilience & failure modes
Failure mode this architecture has to account for
A workload under an active HPA or KEDA ScaledObject sits in a sleep-scheduled namespace — the operator scales it to zero, and the autoscaler, seeing load, scales it straight back up. The two controllers fight in a loop.
How it's handled
The operator doesn't try to out-reconcile another controller — that's a fight it can't win safely. Workloads under HPA/KEDA management opt out with a `finops-operator/exclude: "true"` annotation instead, so the two never touch the same replica count. Native coordination (pausing KEDA's `ScaledObject` for the sleep window instead of requiring opt-out) is on the roadmap, not yet built.