This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Manage Monitor Tempo Set up monitoring

Set up monitoring for Tempo

You can set up monitoring for Tempo using an existing or new cluster. If you don’t have a cluster available, you can use the linked documentation to set up the Tempo, Mimir, and Grafana using Helm or you can use Grafana Cloud.

To set up monitoring, you need to:

• Use Grafana Alloy to remote-write to Tempo and set up Grafana to visualize the tracing data by following Set up a test app.
• Update your Alloy configuration to scrape metrics to monitor for your Tempo data.

This procedure assumes that you have set up Tempo using the Helm chart with Grafana Alloy.

The steps outlined below use the Alloy configurations described in Set up a test application for a Tempo cluster.

Before you begin

To configure monitoring using the examples on this page, you’ll need the following running in your Kubernetes environment:

• Tempo instance - For storing traces and emitting metrics (install using the tempo-distributed Helm chart)
• Mimir - For storing metrics emitted from Tempo (install using the mimir-distributed Helm chart)
• Grafana - For visualizing traces and metrics (install on Kubernetes)

You can use Grafana Alloy or the OpenTelemetry Collector. This procedure provides examples only for Grafana Alloy.

The rest of this documentation assumes that the Tempo, Grafana, and Mimir instances use the same Kubernetes cluster.

If you are using Grafana Cloud, you can skip the installation sections and set up the Mimir (Prometheus) and Tempo data sources in your Grafana instance.

Use a test app for Tempo to send data to Grafana

Before you can monitor Tempo data, you need to configure Grafana Alloy to send traces to Tempo.

Use these instructions to create a test application in your Tempo cluster. These steps configure Grafana Alloy to remote-write to Tempo. In addition, the test app instructions explain how to configure a Tempo data source in Grafana and view the tracing data.

In these examples, Tempo is installed in a namespace called tempo. Change this namespace name in the examples as needed to fit your own environment.

Configure Grafana

In your Grafana instance, you’ll need:

• A Tempo data source (created in the previous section)
• A Mimir (Prometheus) data source

Enable Tempo metrics scraping

Tempo exposes Prometheus metrics from all of its components to allow meta-monitoring. Alloy can both collect traces and act as a Prometheus scraper to retrieve metrics from Tempo. To use this capability, you need to configure Alloy to scrape from all of the components.

Alloy lets you discover targets to scrape in a cluster via a variety of ways. Usually for Prometheus metrics scraping, you would annotate the pods, services, and others, to signal that Alloy should scrape metrics from those objects using annotations such as prometheus/scrape: true as well as a port and path.

However, the Tempo objects already have some convenient annotations supplied under the app.kubernetes.io prefixes. The Helm deployment includes these annotations. For example, the label annotations for Tempo’s distributor component contain:

yaml Copy

app.kubernetes.io/component=distributor
app.kubernetes.io/instance=tempo
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=tempo
app.kubernetes.io/part-of=memberlist
app.kubernetes.io/version=2.6.0

Because of this, you can use Kubernetes service discovery in Grafana Alloy using these annotations, ensuring Alloy scrapes metrics from each Tempo component. Using the discovery.kubernetes Alloy component, you can include selectors to scrape from targets based on labels, for example. As there are Tempo-specific component labels, you can specify a rule that covers all of the Tempo components.

yaml Copy

discovery.kubernetes "k8s_pods" {
  role = "pod"
  selectors {
    // Only scrape pods with a particular selector.
    role = "pod"
    // The selector is any component that belongs to Tempo.
    label = "app.kubernetes.io/name=tempo"
  }
}

This rule lets Alloy know that only pods that include an annotation of app.kubernetes.io/name=tempo should have their metrics scraped.

Something to note is that the dashboards for Tempo expect the specific namespace, cluster name, and job name for labels included with the scraped Tempo metrics, where the job nomenclature is <namespace>/<app.kubernetes.io/component>. The latter expands to distributor, compactor, ingester, and so on. For example: tempo/compactor, where tempo is the namespace and compactor replaces <app.kubernetes.io/component>.

You can acquire all three labels from the annotations that are included as Kubernetes metadata from each components scrape data. Because this metadata is a string with a prefix of __meta_kubernetes, Alloy must be configured not to discard these labels during the metrics pipeline. To do this, you can use the discover.relabel component to add rules to keep any required metadata labels.

In the following example, the specific Tempo component name (for example, __meta_kubernetes_app_kubernetes_io_component) is replaced by a more useful label name and kept for future pipeline operations.

yaml Copy

rule {
  source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_component"]
  action = "replace"
  regex = "(.*)"
  replacement = "$1"
  target_label = "k8s_component_name"
}

By adding rules to keep the __meta_kubernetes_namespace label for namespace and __meta_kubernetes_app_kubernetes_io_instance label for cluster name, you can also attach these labels to written metrics for dashboard use later.

Finally, including a Prometheus relabel component to the Alloy configuration allows the use of the kept metadata labels to create the correct nomenclature for the job:

alloy Copy

rule {
  source_labels = ["namespace", "k8s_component_name"]
  action = "replace"
  regex = "(.*?);(.*?)"
  replacement = "$1/$2"
  target_label = "job"
}

This lets you create a configuration that scrapes metrics from Tempo components and writes the data to a Mimir instance of your choice.

This example provides a Helm values.yaml file that you can use for Alloy deployed on Kubernetes. The file configures the options Alloy uses to scrap a running instance of Tempo. Refer to the comments in the example for details.

yaml Copy

alloy:
  configMap:
  create: true
  content: |-
    // Scrape Prometheus metrics for Tempo.
    prometheus.scrape "tempo" {
      // Use Kubernetes discovery to find the relevant pods to scrape.
      targets    = discovery.relabel.k8s_pods.output
      // Forward to the Prometheus relabeling component.
      forward_to = [prometheus.relabel.tempo.receiver]
    }

    // Determine how to select pods to scrape.
    discovery.kubernetes "k8s_pods" {
      // Only scrape pods.
      role = "pod"
      selectors {
        // Only scrape pods with a particular selector.
        role = "pod"
        // The selector is any component that belongs to Tempo.
        label = "app.kubernetes.io/name=tempo"
      }
    }

    // Relabel data from Kubernetes discovery.
    discovery.relabel "k8s_pods" {
      // Relabel from targets scraped by the discovery selection.
      targets = discovery.kubernetes.k8s_pods.targets

      // Create new namespace label based on the discovered kubernetes namespace.
      rule {
        source_labels = ["__meta_kubernetes_namespace"]
        action = "replace"
        regex = "(.*)"
        replacement = "$1"
        target_label = "namespace"
      }

      // Create new component label based on the discovered kubernetes component.
      rule {
        source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_component"]
        action = "replace"
        regex = "(.*)"
        replacement = "$1"
        target_label = "k8s_component_name"
      }

      // Create new cluster label based on the discovered kubernetes instance.
      rule {
        source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_instance"]
        action = "replace"
        regex = "(.*)"
        replacement = "$1"
        target_label = "cluster"
      }
    }

    // Relabel data from Prometheus scraping.
    prometheus.relabel "tempo" {
      // Replace the existing job label with one comprised of the namespace and component.
      rule {
        source_labels = ["namespace", "k8s_component_name"]
        action = "replace"
        regex = "(.*?);(.*?)"
        replacement = "$1/$2"
        target_label = "job"
      }


      // Send the metrics to the Prometheus remote write component.
      forward_to = [prometheus.remote_write.tempo.receiver]
    }

    // Remote write the metrics to a Prometheus compatible endpoint (in this case Mimir).
    prometheus.remote_write "tempo" {
      endpoint {
        url = "https://mimir-cluster.distributor.mimir.svc.cluster.local:9001/api/v1/push"
      }
    }

This example doesn’t include ingestion for any other data such as traces for sending to Tempo, but can be included with some configuration updates. Refer to Configure Alloy to remote-write to Tempo for more information.

Install Tempo dashboards in Grafana

Alloy scrapes metrics from Tempo and sends them to Mimir or another Prometheus compatible time-series database. You can then monitor Tempo using the mixin.

Tempo ships with a mixin that includes:

• Relevant dashboards for overseeing the health of Tempo as a whole, as well as its individual components
• Recording rules that simplify the generation of metrics for dashboards and free-form queries
• Alerts that trigger when Tempo falls out of operational parameters

To install the mixins in Grafana, you need to:

1. Download the mixin dashboards from the Tempo repository.

2. Import the dashboards in your Grafana instance.

3. Upload alerts.yaml and rules.yaml files for Mimir or Prometheus

Download the tempo-mixin dashboards

1. First, clone the Tempo repository from Github:

   bash Copy

   git clone git+ssh://github.com/grafana/tempo

2. Once you have a local copy of the repository, navigate to the operations/tempo-mixin-compiled directory.

   bash Copy

   cd operations/tempo-mixin-compiled

This contains a compiled version of the alert and recording rules, as well as the dashboards.

Import the dashboards to Grafana

The dashboards directory includes the six monitoring dashboards that can be installed into your Grafana instance. Refer to Import a dashboard in the Grafana documentation.

To create a folder:

1. Open your Grafana instance and select Dashboards.
2. Select New in the right corner.
3. Select New folder from the New drop-down.
4. Name your folder, for example, “Tempo Monitoring”.
5. Select Create.

To import a dashboard:

1. Open your Grafana instance and select Dashboards.
2. Select New in the right corner.
3. Select Import.
4. On the Import dashboard screen, select Upload.
5. Browse to operations/tempo-mixin-compiled/dashboards and select the dashboard to import.
6. Drag the dashboard file, for example, tempo-operational.json, onto the Upload area of the Import dashboard screen. Alternatively, you can browse to and select a file.
7. Select a folder in the Folder drop-down where you want to save the imported dashboard. For example, select Tempo Monitoring created in the earlier steps.
8. Select Import.

The imported files are listed in the Tempo Monitoring dashboard folder.

To view the dashboards in Grafana:

1. Select Dashboards in your Grafana instance.
2. Select Tempo Monitoring, or the folder where you uploaded the imported dashboards.
3. Select any files in the folder to view it.

The ‘Tempo Operational’ dashboard shows read (query) information:

Add alerts and rules to Prometheus or Mimir

The rules and alerts need to be installed into your Mimir or Prometheus instance. To do this in Prometheus, refer to the recording rules and alerting rules documentation.

For Mimir, you can use [mimirtool](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/) to upload rule and alert configuration. Using a default installation of Mimir used as the metrics store for the Alloy configuration, you might run the following:

bash Copy

mimirtool rules load operations/tempo-mixin-compiles/rules.yml --address=https://mimir-cluster.distributor.mimir.svc.cluster.local:9001

mimirtool alertmanager load operations/tempo-mixin-compiles/alerts.yml --address=https://mimir-cluster.distributor.mimir.svc.cluster.local:9001

For Grafana Cloud, you need to add the username and API key as well. Refer to the mimirtool documentation for more information.