OpenTelemetry & LGTM Platform
Cloud-native observability stack using OpenTelemetry Collectors in agent and gateway patterns, feeding the LGTM stack for unified metrics, logs and distributed tracing.
- OpenTelemetry
- LGTM Stack
- Prometheus
- Grafana
- Loki
- Tempo
The problem
Telemetry ends up coupled to whichever vendor was chosen first — instrumentation is rewritten every time the backend changes, and metrics, logs and traces stay in three unconnected tools.
Constraints
- Instrumentation must outlive any single observability vendor
- Metrics, logs and traces need to correlate on the same request
- Collection overhead has to stay proportionate to the workload
Architecture
Source
Collect
Store
Interactive Pipeline Simulator
Ingest & Tail-Sampling Flow
1. Node DaemonSet
2. Collector Gateway
3. LGTM Destination
Key decisions
What was chosen, what it was chosen over, and why.
Chose
OpenTelemetry Collector as the single ingest point
Instead of
Vendor agents exporting straight from each app
Applications emit OTLP and know nothing about the backend, so changing or adding one becomes an exporter config change instead of a re-instrumentation project. That is the whole reason to accept the extra hop.
Chose
Agent DaemonSet feeding a central gateway
Instead of
Agent-only collection
Node-local agents enrich with Kubernetes metadata and buffer through brief network trouble. The gateway holds everything needing a global view — filtering, batching and sampling — in one place instead of on every node.
Chose
Tail-based sampling at the gateway
Instead of
Head-based sampling at the source
Keeps 100% of errors and latency outliers while shedding healthy high-volume traces. It cannot work at the agent, which only ever sees part of a trace — which is what forces the gateway tier.
Also decided
A dedicated observability clusternotCo-locating the backend with the workloads
The platform team owns routing, sampling, dashboards and cost controls on isolated node groups, so a workload cluster incident does not take down the tooling you need to debug it.
Specialised backends behind one GrafananotA single general-purpose store
Metrics, logs and traces have genuinely different retention and query shapes. Grafana unifies them at the point of use, so the split costs nothing where it would be felt.
Production resilience & failure modes
Failure mode this architecture has to account for
A traffic spike upstream sends the gateway's tail-sampling buffer (`num_traces: 10000`, 10s decision wait) more concurrent traces than it can hold in memory before deciding what to keep.
How it's handled
The `filter/budgeting` and `tail_sampling` processors run before export, not after — a misbehaving tenant gets dropped at the gateway instead of paying to ship its spans to Tempo first. The five sampling policies (errors, 5xx, slow requests, then 5% of everything healthy) are evaluated in that order, so the traces worth keeping are decided before the traces worth dropping ever get sized for export.