Skip to content
All writing
5 min read

Why the OpenTelemetry Collector earns its extra hop

Sending telemetry straight to your backend is simpler and cheaper. Here is what that simplicity costs you the second time you change vendors.

Every observability vendor ships an agent that takes telemetry directly from your application. It works, it is well documented, and it is one fewer component to run. Putting a collector in the middle is strictly more infrastructure. So it has to earn the hop.

What direct export actually couples

When applications talk to a backend directly, the vendor's assumptions end up distributed across every service you own. Its SDK is in your dependency tree. Its attribute conventions are in your instrumentation. Its sampling configuration is in your application config, deployed on your application's release cadence.

None of that hurts until the day the answer to "can we evaluate a different backend?" is "we would have to re-instrument everything first." At that point the cost of switching has quietly become larger than the saving from switching, and the decision makes itself. That is not a tooling problem, it is a coupling problem, and it was created years earlier by a choice nobody wrote down.

What the collector changes

With a collector, applications emit OTLP and know nothing else. The backend becomes an exporter configuration — a change to one component, reviewed on its own, rolled back on its own. Adding a second backend to run side by side during an evaluation stops being a migration and becomes a config block.

Processing also moves out of the application. Attribute scrubbing, PII redaction, sampling, batching — all of it happens in a component owned by the platform team rather than being reimplemented, slightly differently, in every service.

Agent, gateway, or both

Two topologies, and most real deployments want both.

  • Agent — a collector per node, usually a DaemonSet. Close to the workload, so it can enrich with node and pod metadata and survive brief network trouble. Cheap and horizontally scaled by definition.
  • Gateway — a central deployment every agent forwards to. This is where anything needing a global view belongs: tail-based sampling, which cannot work on a node that has only seen part of a trace, plus egress control and credential handling in one place instead of on every node.

The platform I built runs both: a DaemonSet in each workload cluster enriching with Kubernetes metadata, forwarding to a gateway fleet in a separate observability cluster that owns filtering, batching and sampling. Splitting the cluster as well as the tier matters more than it sounds — it means a workload-cluster incident cannot take down the tooling you need to debug it.

When to skip it

The collector is not free. It is another deployment to run, size, monitor and page on, and a component that sits between your services and the ability to see them — which makes it a dependency of your own debugging.

For one service, one team, and a vendor you have no intention of leaving, direct export is the honest answer. The collector earns its place when telemetry has to outlive a vendor decision, and it is worth being able to say which of those situations you are in rather than adopting it because it is the recommended architecture.