Grafana Tempo 2.3 release: faster trace queries, TraceQL upgrades
Grafana Tempo 2.3 has been unleashed upon the world, bringing with it the latest iteration of the vParquet backend! Tempo 2.3 has a little bit of everything, but the headline item here is vParquet3 and new features that improve search speeds.
Watch the video above for all the details, or continue reading to get a quick overview of the latest updates in Tempo. If you’re looking for something more in-depth, don’t hesitate to jump into the changelog or our Grafana Tempo 2.3 release notes.
vParquet3
vParquet3 is the latest Parquet backend version for Tempo. This new format has a number of small improvements, but the big feature is the ability to choose your own dedicated string attribute columns, which allows for much more performant TraceQL queries.
Parquet backends will, by default, put the majority of the span attributes into one large column. This means that for some TraceQL queries you are pulling a lot more data than necessary. With vParquet3, you can use the dedicated columns configuration to reduce the size of this main attributes column. We’ve found the most useful columns to pull out are simply the ones that contribute the most to the column’s size, so we recommend using the CLI tool to help discover these attributes.
Check out this search latency reduction as we rolled out vParquet3 in one of our clusters:
Note: vParquet3 is included in Tempo 2.3 but is not default, and we encourage everyone to enable and configure it. vParquet3 is officially GA and we have been running it in production for months. We have decided to leave vParquet2 default to allow for a cleaner upgrade and downgrade path.
New TraceQL features
It will be a sad day when I write a Tempo release blog post and can’t include a TraceQL section. Thankfully, today is not that day!
Structural operators
Tempo 2.3 brings a few new structural operators to the table. First, we have the parent and ancestor operators. Recall that all structural operators return results from the right-hand side of the operator! Parent and ancestor are the inverse of child and descendant.
Find, spans of service foo that have children from service bar:
{ resource.service.name = "bar" } < { resource.service.name = "foo" }Ancestor can be used to find HTTP request spans that have an error somewhere beneath them.
{ status = error } << { span.http.route = "/api/failing" }Compare this to a descendant query that finds errors beneath a failing API.
{ span.http.route = "/api/failing" } >> { status = error }The next set of structural operators are experimental only. We are aware of cases in which they will return false positives so we are hesitant to call them GA. These false positives tend to occur on larger traces that are written to Tempo across longer periods of time. However, we do feel they will be quite useful in some cases and have decided to include them.
In Tempo 2.3, we have added negated structural operators. These operators return spans from the right hand side if the left hand side can NOT be found.
This query finds an errored span that has no child error. This is useful for tracking down the final error in a series of propagated errors.
{ status = error } !< { status = error }This simple query finds all leaf spans:
{ } !< { }Negated regex
Negated regex is a new binary operator that does exactly what it says. This operator returns true if the regex does NOT match and mirrors similar operators in PromQL and LogQL.
{ span.http.route !~ ".*v1.*" }New intrinsic
We’ve added a new intrinsic called statusMessage. This field is often filled with a useful human-readable error message. Just give this query a shot to see what errors or messages are occurring in your cluster:
{ statusMessage != "" }Complete support for OpenTelemetry attributes
With this latest release, Tempo finally supports the full scope of OpenTelemetry attribute names. There are no character restrictions on OTel attribute names and the specification states that “Every name MUST be a valid Unicode sequence.” Unfortunately this means that attribute names can contain difficult to parse characters like whitespace, pipes, and curly brackets.
Tempo 2.3 adds quoted attribute names, which gives developers the freedom to create terrible attribute names and still query them.
{ span."name with spaces" = "foo" }Previously, we terminated attribute names on a set of characters that were simply not allowed in TraceQL, but quoted attribute names allow any valid unicode character to be supported.
What’s next in Grafana Tempo?
We have two large initiatives going on in Tempo at the moment. First, we are looking to extend the TraceQL language to allow querying of links, events, and lists. Second, we are working on querying metrics directly from traces dynamically with TraceQL. We have been building towards this feature for over a year and are excited that it’s finally getting some heads down attention. I know everyone is excited for it and so are we!
Expect some more news on both of these fronts soon! We will post a design doc for both of these updates to the repo, which will give the community an opportunity to give feedback and be involved before we finalize the design. If you’d like to have your voice heard on these features please connect to the community through any of the options below. Happy tracing!
If you are interested in hearing more about Grafana Tempo news or search progress, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, reach out on X (formerly Twitter), or join our monthly Tempo community call. See you there!
And if you want to get even closer to where the magic happens, why not have a look at our open positions at Grafana Labs?
The easiest way to get started with Grafana Tempo is with Grafana Cloud, and our free forever tier now includes 50GB of traces along with 50GB of logs and 10K series of metrics. You can sign up for free!
