This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.

Documentation Grafana Tempo API Metrics summary API

Metrics summary API

This document explains how to use the metrics summary API in Tempo. This API returns RED metrics (span count, erroring span count, and latency information) for kind=server spans sent to Tempo in the last hour, grouped by a user-specified attribute.

Deprecation in favor of TraceQL metrics

The metrics summary API is now redundant given the release of TraceQL metrics. This API is therefore being deprecated to reduce our maintenance burden.

TraceQL metrics queries are significantly more powerful than what metrics summary API provides. TraceQL metrics can look at arbitrary time windows (not just the last hour), return time series information (rather than just a single instant value over the past hour), and can look at all spans (not just kind=server).

To provide an example, if you were to aggregate by the attribute resource.cloud.region with the metrics summary API, you could get the same results with a couple TraceQL queries:

Rate of requests by resource.cloud.region

Copy

{ } | rate() by (resource.cloud.region)

Error rate by resource.cloud.region

Copy

{ status=error } | rate() by (resource.cloud.region)

The 50th, 90th, and 99th percentile latencies (for example, p99, p90, and p50) by resource.cloud.region:

Copy

{ } | quantile_over_time(duration, .99, .9, .5) by (resource.cloud.region)

If you want something faster than typing these queries out in Explore’s code mode, use Explore Traces, a queryless experience for navigating your trace data stored in Tempo and powered by TraceQL metrics queries under the hood.

Activate metrics summary

To enable the (deprecated) metrics summary API, you must turn on the local blocks processor in the metrics generator. Be aware that the generator uses considerably more resources, including disk space, if it’s enabled:

yaml Copy

overrides:
  defaults:
    metrics_generator:
      processors: [..., 'local-blocks']

In Grafana and Grafana Cloud, the Metrics summary API is disabled by default. To enable it in Grafana Cloud, contact Grafana Support.

Request

To make a request to this API, use the following endpoint on the query-frontend:

Copy

GET http://<tempo>/api/metrics/summary

Query Parameters

All query parameters must be URL-encoded to preserve non-URL-safe characters in the query such as &.

Example:

bash Copy

curl "$URL/api/metrics/summary" --data-urlencode 'q={resource.service.name="checkout-service"}' --data-urlencode 'groupBy=name'

Response

The Tempo response is a SpanMetricsSummary object defined in tempo.proto, relevant section pasted below:

Copy

message SpanMetricsSummaryResponse {
  repeated SpanMetricsSummary summaries = 1;
}

message SpanMetricsSummary {
  uint64 spanCount = 1;
  uint64 errorSpanCount = 2;
  TraceQLStatic static = 3;
  uint64 p99 = 4;
  uint64 p95 = 5;
  uint64 p90 = 6;
  uint64 p50 = 7;
}

message TraceQLStatic {
  int32 type = 1;
  int64 n = 2;
  double f = 3;
  string s = 4;
  bool b = 5;
  uint64 d = 6;
  int32 status = 7;
  int32 kind = 8;
}

The response is returned as JSON following standard protobuf->JSON mapping rules.

Example:

JavaScript Copy

{
   "summaries": [
       {
           "spanCount": "20",
           "series" : [
               {
                   "key": ".attr1",
                   "value": {
                       "type": 5,
                       "s": "foo"
                   },
               },
               ...
           ],
           "p99": "68719476736",
           "p95": "1073741824",
           "p90": "1017990479",
           "p50": "664499239"
       },