Kubernetes attributes processor: make telemetry cluster-aware

In Kubernetes, “which service emitted this?” is not enough.

When the page fires, you need to know which pod, which namespace, which node, which workload, and often which version produced the signal. A trace says checkout slowed down. A log says connection refused. A metric says CPU spiked. None of that is complete until it lands on the real cluster object that emitted it.

The Kubernetes attributes processor is the Collector component that makes that join happen. It watches Kubernetes metadata, keeps a local cache, matches incoming telemetry to a pod, and writes the result as OpenTelemetry resource attributes.

If the Collector can still identify the pod, telemetry becomes cluster-aware. If it cannot, the rest of the pipeline is guessing.

What it adds

The processor does not scrape pods, collect logs, or create spans. It enriches telemetry that already exists.

A signal may arrive with a resource like this:

{
  "resource": {
    "attributes": {
      "service.name": "checkout-api"
    }
  }
}

After enrichment, it can leave with Kubernetes context:

{
  "resource": {
    "attributes": {
      "service.name": "checkout-api",
      "k8s.namespace.name": "payments",
      "k8s.pod.name": "checkout-api-7b9f9554f8-q9vnr",
      "k8s.pod.uid": "d30d0a5e-4c3d-4e25-8a74-3b95c04b620a",
      "k8s.deployment.name": "checkout-api",
      "k8s.node.name": "ip-10-0-42-18"
    }
  }
}

That looks like metadata, but operationally it is a join key. It lets you ask useful questions:

• Which namespace is producing the errors?
• Did only one deployment version regress?
• Are failures tied to one node?
• Is this workload noisy across every signal or just one metric?

The application should not have to know all of this. The Collector sits close enough to the cluster to add it consistently.

The processor has two jobs

The implementation is easier to reason about if you split it in two:

1. Build a cache of Kubernetes metadata.
2. Match incoming telemetry to a pod and attach fields from that cache.

At startup, the processor creates Kubernetes clients and watches the resources it needs. Pods are the main object. Depending on extraction rules, it may also need namespaces, nodes, ReplicaSets, Deployments, StatefulSets, DaemonSets, Jobs, or CronJobs.

As Kubernetes sends add, update, and delete events, the processor keeps lookup maps in memory. A pod may be indexed by pod IP, pod UID, container ID, pod name plus namespace, or other configured association fields.

When telemetry passes through the processor, it tries to find a matching pod in that cache. If it finds one, it adds configured metadata as resource attributes.

That is the happy path. Most production issues happen when the matching signal is missing, ambiguous, or lost before the processor sees it.

Pod association is where correctness lives

Pod association tells the processor where to look for an identifier.

Two source types matter most:

Source	What it uses	Best fit
connection	The incoming connection IP	Agent Collector receiving directly from local workloads
resource_attribute	Attributes such as k8s.pod.uid, k8s.pod.ip, k8s.pod.name, k8s.namespace.name	Gateway Collector or telemetry that already carries pod identity

Rules are checked in order. Strong identifiers should come first.

processors:
  k8s_attributes:
    auth_type: serviceAccount
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: connection

If you run the Collector as an agent on each node and workloads send directly to it, connection-based matching can work well. If telemetry flows through sidecars, gateways, proxies, or load balancers, the connection IP may no longer be the pod IP. In that case, the safer path is to make the workload or earlier Collector attach pod identity as resource attributes and use those for association.

This is the part to test before you trust the dashboards. A pipeline can look correct and still attach the wrong pod if the association rule is weak.

Agent and gateway modes are different

The processor behaves differently depending on where the Collector runs.

Deployment shape	What works well	What to watch
Agent per node	Local pod IP association, node-local enrichment	Needs DaemonSet permissions and local routing
Central gateway	Consistent policy and export point	Connection IP may be the previous Collector, not the workload
Agent plus gateway	Enrich near source, centralize export later	Avoid duplicate or conflicting enrichment

For Kubernetes metadata, earlier is usually better. The closer the processor is to the emitting pod, the more trustworthy the association data is.

A common pattern is:

workload -> node Collector with k8s_attributes -> gateway Collector -> backend

The node Collector enriches telemetry while pod identity is still available. The gateway can batch, route, redact, and export without needing to rediscover which pod emitted each record.

What to extract

Do not extract every possible label and annotation because it is available. Kubernetes metadata can be high-cardinality and sometimes sensitive.

Start with resource identity:

processors:
  k8s_attributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.node.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.job.name
        - k8s.cronjob.name

Then add labels only when they have an owner and a query use case:

processors:
  k8s_attributes:
    extract:
      labels:
        - tag_name: app.team
          key: app.kubernetes.io/team
          from: pod
        - tag_name: app.version
          key: app.kubernetes.io/version
          from: pod

The question is not “can we attach this?” The question is “will we operate this field responsibly after it lands in the backend?”

RBAC is part of the feature

The processor can only enrich from resources it is allowed to watch. If it needs namespace or workload metadata but the service account cannot read those objects, the config may start and still produce incomplete enrichment.

Treat RBAC as part of the pipeline config:

rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]

If enrichment is missing, check permissions before you blame the application. The Collector cannot attach metadata it was never allowed to see.

Debug the join, not only the output

When enrichment fails, keep the investigation narrow:

1. Does the incoming telemetry contain any pod identifier?
2. Is the processor running in the pipeline for that signal?
3. Does the Collector have RBAC to watch the needed resources?
4. Is the association rule looking at the right field?
5. Did another processor drop or rename the field before enrichment?
6. Are you in gateway mode where connection IP is no longer useful?

Use the debug exporter temporarily to compare records before and after enrichment. Do not start by changing every processor. Prove where the context disappears.

Conclusion

The Kubernetes attributes processor turns telemetry from “a service emitted this” into “this exact workload in this exact cluster shape emitted this.”

That context is what makes logs, metrics, and traces useful in Kubernetes. Without it, every incident becomes a manual join between telemetry and cluster state. With it, namespace, pod, node, deployment, and workload become normal dimensions of debugging.

The processor is simple to enable, but production quality depends on association rules, RBAC, deployment shape, and restraint around labels. Get those right and your telemetry starts speaking Kubernetes instead of merely passing through it.