Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Tempo query editor
The Tempo data source’s query editor helps you query and display traces from Tempo in Explore.
This topic explains configuration and queries specific to the Tempo data source. For general documentation on querying data sources in Grafana, see Query and transform data.
Query by search
Use this to search for traces by service name, span name, duration range, or process-level attributes that are included in your application’s instrumentation, such as HTTP status code and customer ID.
To configure Tempo and the Tempo data source for search, refer to Configure the data source.
To search for traces:
- Select Search from the Query type selector.
- Fill out the search form:
Name | Description |
---|---|
Service Name | Returns a list of services. |
Span Name | Returns a list of span names. |
Tags | Sets tags with values in the logfmt format, such as error=true db.statement="select * from User" . |
Min Duration | Filters all traces with a duration higher than the set value. Possible values are 1.2s , 100ms , 500us . |
Max Duration | Filters all traces with a duration lower than the set value. Possible values are 1.2s , 100ms , 500us . |
Limit | Limits the number of traces returned. |
Search recent traces
You can search recent traces held in Tempo’s ingesters. By default, ingesters store the last 15 minutes of tracing data.
To configure your Tempo data source to use this feature, refer to the Tempo documentation.
Search the backend datastore
Tempo includes the ability to search the entire backend datastore.
To configure your Tempo data source to use this feature, refer to the Tempo documentation.
Query by TraceID
To query a particular trace:
- Select the TraceQL query type.
- Enter the trace’s ID into the query field.
Query by TraceQL
Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what is searched
To learn more about how to query by TraceQL, refer to the TraceQL documentation.
You can create TraceQL queries using the Query editor or using Search query tab (preview feature).
Create TraceQL queries using Search
Note
This new Search query type is available as an experimental feature. This feature is under active development and is not production ready. To try this feature, enable thetraceqlSearch
feature flag in Grafana (read documentation). Grafana Cloud users should contact Grafana Support to enable this feature.
Using the Search tab in Explore, you can use the query builder’s drop-downs to compose TraceQL queries. The selections you make automatically generate a TraceQL query.
To access Search, select your Tempo data source, and then choose Explore and select Query type > Search. You can use the query builder to search trace data by resource service name, span name, duration, and one or more tags. The examples on this page use the default filters.
In addition, you can add query builder blocks, view the query history, and use the Inspector to see details.
Perform a search
To perform a search, you need to select filters and/or tags and then run the query. The results appear underneath the query builder. The screenshot below identifies the areas used to perform a search.
Number | Name | Action | Comment |
---|---|---|---|
1 | Data source | Use the data source drop-down to select a Tempo data source. | Each data source has its own version of search. This Search is specific to the Tempo data source. |
2 | Query type tab | Select Search. | |
3 | Choose filter | Choose whether to filter using Resource Service Name, Span Name, and/or Duration. | Optional. You can execute an empty query in the Search tab. In TraceQL, {} is a valid query and is the default query until you choose options. |
4 | Filters conditions | Select options for one or more filters. For example, you can define a filter where Resource Service Name (resource.service.name ) equals (= ) cloud-logs-archiver . | Optional. At least one tag or filter must be defined. |
5 | Tags | Add tags for span, resource, or unscoped and define their conditions. | Optional. |
6 | TraceQL query | Displays the TraceQL query constructed by your selections. | This TraceQL query is executed when you select Run query. |
Every query searches the data for the selected time frame. By default, queries run against data from the last hour. Select Time range to the left of Run query to choose the time range for the data your query runs against. Read the dashboard time range documentation to learn more.
To access Search, use the following steps:
- Sign into Grafana.
- Select your Tempo data source.
- From the menu, choose Explore and select Query type > Search.
Define filters
Using filters, you refine the data returned from the query by selecting Resource Service Name, Span Name, or Duration. The Duration represents span time, calculated by subtracting the end time from the start time of the span.
Grafana administrators can change the default filters using the Tempo data source configuration.
Filters can be limited by the operators. The available operators are determined by the field type.
For example, Span Name and Resource Service Name are string fields so the comparison operators are equals (=
), not equal (!=
), or regular expressions (=~
).
Duration is a duration field type and uses range selections (>
, >=
, <
, <=
).
When you select multiple values for the same filter, Grafana automatically changes the operator to the regex operator =~
and concatenates the values with a |
. This capability only applies to fields with drop-down value selection.
For example, if you choose Span Name = get
and then Span Name = log_results_cache,
operator drop-down changes from =
to =~
and both get
and log_results_cache
are listed in the Span Name field. The resulting query is updated with this:
{duration>5ms && duration<10ms && name=~"get|log_results_cache"}
To define filters, follow these steps:
- Choose one of the filters.
- Select a comparison operator from the drop-down.
- Resource Service Name and Span Name only: Select one or more values from the drop-down.
- Duration only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (
ns
), milliseconds (ms
), seconds (s
), minutes (m
), and hours (h
).
You can either select Run query to execute the query or define tags and then run the query.
Define tags
You can add any tags to your query. Tags can be selected by scoped (span or resource) or unscoped. If you select unscoped, then all tags are searched for matches.
To add a tag, follow these steps:
- Select span, resource, or unscoped.
- Select a tag from the Select tag drop-down.
- Select a comparison operator.
- Select a value from the Select value drop-down. This field is populated based upon the tag.
- Optional: Select + to add an additional tag.
Optional: Add queries
Using Add query, you can have successive queries that run in sequential order. For example, query A runs and then query B. You can reorder the queries by dragging and dropping them above or below other queries. Select + Add query to add another query block.
Run queries and view results
Select Run query to run the TraceQL query (1 in the screenshot).
Queries can take a little while to return results. The results appear in a table underneath the query builder. Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).
Query Loki for traces
To find traces to visualize, you can use the Loki query editor. For results, you must configure derived fields in the Loki data source that point to this data source.