Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Grafana Cloud Enterprise Open source

Tempo query editor

The Tempo data source’s query editor helps you query and display traces from Tempo in Explore.

This topic explains configuration and queries specific to the Tempo data source. For general documentation on querying data sources in Grafana, see Query and transform data.

Use this to search for traces by service name, span name, duration range, or process-level attributes that are included in your application’s instrumentation, such as HTTP status code and customer ID.

To configure Tempo and the Tempo data source for search, refer to Configure the data source.

To search for traces:

  1. Select Search from the Query type selector.
  2. Fill out the search form:
NameDescription
Service NameReturns a list of services.
Span NameReturns a list of span names.
TagsSets tags with values in the logfmt format, such as error=true db.statement="select * from User".
Min DurationFilters all traces with a duration higher than the set value. Possible values are 1.2s, 100ms, 500us.
Max DurationFilters all traces with a duration lower than the set value. Possible values are 1.2s, 100ms, 500us.
LimitLimits the number of traces returned.
Screenshot of the Tempo search feature with a trace rendered in the right panel
Screenshot of the Tempo search feature with a trace rendered in the right panel

Search recent traces

You can search recent traces held in Tempo’s ingesters. By default, ingesters store the last 15 minutes of tracing data.

To configure your Tempo data source to use this feature, refer to the Tempo documentation.

Search the backend datastore

Tempo includes the ability to search the entire backend datastore.

To configure your Tempo data source to use this feature, refer to the Tempo documentation.

Query by TraceID

To query a particular trace:

  1. Select the TraceQL query type.
  2. Enter the trace’s ID into the query field.
Screenshot of the Tempo TraceID query type
Screenshot of the Tempo TraceID query type

Query by TraceQL

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what is searched

To learn more about how to query by TraceQL, refer to the TraceQL documentation.

You can create TraceQL queries using the Query editor or using Search query tab (preview feature).

Note

This new Search query type is available as an experimental feature. This feature is under active development and is not production ready. To try this feature, enable the traceqlSearch feature flag in Grafana (read documentation). Grafana Cloud users should contact Grafana Support to enable this feature.

Using the Search tab in Explore, you can use the query builder’s drop-downs to compose TraceQL queries. The selections you make automatically generate a TraceQL query.

To access Search, select your Tempo data source, and then choose Explore and select Query type > Search. You can use the query builder to search trace data by resource service name, span name, duration, and one or more tags. The examples on this page use the default filters.

In addition, you can add query builder blocks, view the query history, and use the Inspector to see details.

Screenshot of the Tempo Search query type
Screenshot of the Tempo Search query type

To perform a search, you need to select filters and/or tags and then run the query. The results appear underneath the query builder. The screenshot below identifies the areas used to perform a search.

Parts of Tempo Search query type
Parts of Tempo Search query type
NumberNameActionComment
1Data sourceUse the data source drop-down to select a Tempo data source.Each data source has its own version of search. This Search is specific to the Tempo data source.
2Query type tabSelect Search.
3Choose filterChoose whether to filter using Resource Service Name, Span Name, and/or Duration.Optional. You can execute an empty query in the Search tab. In TraceQL, {} is a valid query and is the default query until you choose options.
4Filters conditionsSelect options for one or more filters. For example, you can define a filter where Resource Service Name (resource.service.name) equals (=) cloud-logs-archiver.Optional. At least one tag or filter must be defined.
5TagsAdd tags for span, resource, or unscoped and define their conditions.Optional.
6TraceQL queryDisplays the TraceQL query constructed by your selections.This TraceQL query is executed when you select Run query.

Every query searches the data for the selected time frame. By default, queries run against data from the last hour. Select Time range to the left of Run query to choose the time range for the data your query runs against. Read the dashboard time range documentation to learn more.

To access Search, use the following steps:

  1. Sign into Grafana.
  2. Select your Tempo data source.
  3. From the menu, choose Explore and select Query type > Search.

Define filters

Using filters, you refine the data returned from the query by selecting Resource Service Name, Span Name, or Duration. The Duration represents span time, calculated by subtracting the end time from the start time of the span.

Grafana administrators can change the default filters using the Tempo data source configuration. Filters can be limited by the operators. The available operators are determined by the field type. For example, Span Name and Resource Service Name are string fields so the comparison operators are equals (=), not equal (!=), or regular expressions (=~). Duration is a duration field type and uses range selections (>, >=, <, <=).

When you select multiple values for the same filter, Grafana automatically changes the operator to the regex operator =~ and concatenates the values with a |. This capability only applies to fields with drop-down value selection.

For example, if you choose Span Name = get and then Span Name = log_results_cache, operator drop-down changes from = to =~ and both get and log_results_cache are listed in the Span Name field. The resulting query is updated with this:

{duration>5ms && duration<10ms && name=~"get|log_results_cache"}

To define filters, follow these steps:

  1. Choose one of the filters.
  2. Select a comparison operator from the drop-down.
  3. Resource Service Name and Span Name only: Select one or more values from the drop-down.
  4. Duration only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (ns), milliseconds (ms), seconds (s), minutes (m), and hours (h).

You can either select Run query to execute the query or define tags and then run the query.

Define tags

You can add any tags to your query. Tags can be selected by scoped (span or resource) or unscoped. If you select unscoped, then all tags are searched for matches.

To add a tag, follow these steps:

  1. Select span, resource, or unscoped.
  2. Select a tag from the Select tag drop-down.
  3. Select a comparison operator.
  4. Select a value from the Select value drop-down. This field is populated based upon the tag.
  5. Optional: Select + to add an additional tag.

Optional: Add queries

Using Add query, you can have successive queries that run in sequential order. For example, query A runs and then query B. You can reorder the queries by dragging and dropping them above or below other queries. Select + Add query to add another query block.

Run queries and view results

Select Run query to run the TraceQL query (1 in the screenshot).

Queries can take a little while to return results. The results appear in a table underneath the query builder. Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).

Tempo Search query type results
Tempo Search query type results

Query Loki for traces

To find traces to visualize, you can use the Loki query editor. For results, you must configure derived fields in the Loki data source that point to this data source.

Screenshot of the Tempo query editor showing the Loki Search tab
Screenshot of the Tempo query editor showing the Loki Search tab