Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Agent Flow mode Reference Components reference otelcol.processor.k8sattributes

otelcol.processor.k8sattributes

otelcol.processor.k8sattributes accepts telemetry data from other otelcol components and adds Kubernetes metadata to the resource attributes of spans, logs, or metrics.

You can specify multiple otelcol.processor.k8sattributes components by giving them different labels.

Usage

river Copy

otelcol.processor.k8sattributes "LABEL" {
  output {
    metrics = [...]
    logs    = [...]
    traces  = [...]
  }
}

Arguments

The following arguments are supported:

The supported values for auth_type are:

• none: No authentication is required.
• serviceAccount: Use the built-in service account that Kubernetes automatically provisions for each pod.
• kubeConfig: Use local credentials like those used by kubectl.
• tls: Use client TLS authentication.

Setting passthrough to true enables the “passthrough mode” of otelcol.processor.k8sattributes:

• Only a k8s.pod.ip resource attribute will be added.
• No other metadata will be added.
• The Kubernetes API will not be accessed.
• To correctly detect the pod IPs, Grafana Agent must receive spans directly from services.
• The passthrough setting is useful when configuring the Agent as a Kubernetes Deployment. A Grafana Agent running as a Deployment cannot detect the IP addresses of pods generating telemetry data without any of the well-known IP attributes. If the Deployment Grafana Agent receives telemetry from Grafana Agents deployed as DaemonSet, then some of those attributes might be missing. As a workaround, you can configure the DaemonSet Grafana Agents with passthrough set to true.

Blocks

The following blocks are supported inside the definition of otelcol.processor.k8sattributes:

The > symbol indicates deeper levels of nesting. For example, extract > annotation refers to an annotation block defined inside an extract block.

extract block

The extract block configures which metadata, annotations, and labels to extract from the pod.

The following attributes are supported:

The currently supported metadata keys are:

• k8s.pod.name
• k8s.pod.uid
• k8s.deployment.name
• k8s.node.name
• k8s.namespace.name
• k8s.pod.start_time
• k8s.replicaset.name
• k8s.replicaset.uid
• k8s.daemonset.name
• k8s.daemonset.uid
• k8s.job.name
• k8s.job.uid
• k8s.cronjob.name
• k8s.statefulset.name
• k8s.statefulset.uid
• k8s.container.name
• container.image.name
• container.image.tag
• container.id

By default, if metadata is not specified, the following fields are extracted and added to spans, metrics, and logs as resource attributes:

• k8s.pod.name
• k8s.pod.uid
• k8s.pod.start_time
• k8s.namespace.name
• k8s.node.name
• k8s.deployment.name (if the pod is controlled by a deployment)
• k8s.container.name (requires an additional attribute to be set: container.id)
• container.image.name (requires one of the following additional attributes to be set: container.id or k8s.container.name)
• container.image.tag (requires one of the following additional attributes to be set: container.id or k8s.container.name)

annotation block

The annotation block configures how to extract Kubernetes annotations.

The following attributes are supported:

When you don’t specify the tag_name, a default tag name is used with the format:

• k8s.pod.annotations.<annotation key>
• k8s.pod.labels.<label key>

For example, if tag_name isn’t specified and the key is git_sha, the attribute name will be k8s.pod.annotations.git_sha.

You can set either the key attribute or the key_regex attribute, but not both. When key_regex is present, tag_name supports back reference to both named capturing and positioned capturing.

For example, assume your pod spec contains the following labels:

• app.kubernetes.io/component: mysql
• app.kubernetes.io/version: 5.7.21

If you’d like to add tags for all labels with the prefix app.kubernetes.io/ and trim the prefix, then you can specify the following extraction rules:

river Copy

extract {
	label {
	    from = "pod"
		key_regex = "kubernetes.io/(.*)"
		tag_name  = "$1"
	}
}

These rules add the component and version tags to the spans or metrics.

You can set the from attribute to either "pod" or "namespace".

label block

The label block configures how to extract Kubernetes labels.

The following attributes are supported:

When you don’t specify the tag_name, a default tag name is used with the format:

• k8s.pod.annotations.<annotation key>
• k8s.pod.labels.<label key>

For example, if tag_name isn’t specified and the key is git_sha, the attribute name will be k8s.pod.annotations.git_sha.

You can set either the key attribute or the key_regex attribute, but not both. When key_regex is present, tag_name supports back reference to both named capturing and positioned capturing.

For example, assume your pod spec contains the following labels:

• app.kubernetes.io/component: mysql
• app.kubernetes.io/version: 5.7.21

If you’d like to add tags for all labels with the prefix app.kubernetes.io/ and trim the prefix, then you can specify the following extraction rules:

river Copy

extract {
	label {
	    from = "pod"
		key_regex = "kubernetes.io/(.*)"
		tag_name  = "$1"
	}
}

These rules add the component and version tags to the spans or metrics.

You can set the from attribute to either "pod" or "namespace".

filter block

The filter block configures which nodes to get data from and which fields and labels to fetch.

The following attributes are supported:

If node is specified, then any pods not running on the specified node will be ignored by otelcol.processor.k8sattributes.

field block

The field block allows you to filter pods by generic Kubernetes fields.

The following attributes are supported:

For op, the following values are allowed:

• equals: The field value must equal the provided value.
• not-equals: The field value must not be equal to the provided value.
• exists: The field value must exist. Only applicable to annotation fields.
• does-not-exist: The field value must not exist. Only applicable to annotation fields.

label block

The label block allows you to filter pods by generic Kubernetes labels.

The following attributes are supported:

For op, the following values are allowed:

• equals: The field value must equal the provided value.
• not-equals: The field value must not be equal to the provided value.
• exists: The field value must exist. Only applicable to annotation fields.
• does-not-exist: The field value must not exist. Only applicable to annotation fields.

pod_association block

The pod_association block configures rules on how to associate logs/traces/metrics to pods.

The pod_association block does not support any arguments and is configured fully through child blocks.

The pod_association block can be repeated multiple times, to configure additional rules.

Example:

river Copy

pod_association {
    source {
        from = "resource_attribute"
        name = "k8s.pod.ip"
    }
}

pod_association {
    source {
        from = "resource_attribute"
        name = "k8s.pod.uid"
    }
    source {
        from = "connection"
    }
}

source block

The source block configures a pod association rule. This is used by the k8sattributes processor to determine the pod associated with a telemetry signal.

When multiple source blocks are specified inside a pod_association block, both source blocks has to match for the pod to be associated with the telemetry signal.

The following attributes are supported:

exclude block

The exclude block configures which pods to exclude from the processor.

pod block

The pod block configures a pod to be excluded from the processor.

The following attributes are supported:

output block

The output block configures a set of components to forward resulting telemetry data to.

The following arguments are supported:

You must specify the output block, but all its arguments are optional. By default, telemetry data is dropped. Configure the metrics, logs, and traces arguments accordingly to send telemetry data to other components.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.processor.k8sattributes is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.processor.k8sattributes does not expose any component-specific debug information.

Examples

Basic usage

In most cases, this is enough to get started. It’ll add these resource attributes to all logs, metrics, and traces:

• k8s.namespace.name
• k8s.pod.name
• k8s.pod.uid
• k8s.pod.start_time
• k8s.deployment.name
• k8s.node.name

Example:

river Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
    logs    = [otelcol.processor.k8sattributes.default.input]
    traces  = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

Add additional metadata and labels

river Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
    logs    = [otelcol.processor.k8sattributes.default.input]
    traces  = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    label {
      from      = "pod"
      key_regex = "(.*)/(.*)"
      tag_name  = "$1.$2"
    }

    metadata = [
      "k8s.namespace.name",
      "k8s.deployment.name",
      "k8s.statefulset.name",
      "k8s.daemonset.name",
      "k8s.cronjob.name",
      "k8s.job.name",
      "k8s.node.name",
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.pod.start_time",
    ]
  }

  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

Adding Kubernetes metadata to Prometheus metrics

otelcol.processor.k8sattributes adds metadata to metrics signals in the form of resource attributes. To display the metadata as labels of Prometheus metrics, the OTLP attributes must be converted from resource attributes to datapoint attributes. One way to do this is by using an otelcol.processor.transform component.

river Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    label {
      from = "pod"
    }

    metadata = [
      "k8s.namespace.name",
      "k8s.pod.name",
    ]
  }

  output {
    metrics = [otelcol.processor.transform.add_kube_attrs.input]
  }
}

otelcol.processor.transform "add_kube_attrs" {
  error_mode = "ignore"

  metric_statements {
    context = "datapoint"
    statements = [
      "set(attributes[\"k8s.pod.name\"], resource.attributes[\"k8s.pod.name\"])",
      "set(attributes[\"k8s.namespace.name\"], resource.attributes[\"k8s.namespace.name\"])",
    ]
  }

  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}