Grafana Tempo 2.2 release: TraceQL structural operators are here!

Get excited about Grafana Tempo 2.2! Not only is this release on time, but it is also chock full of TraceQL features and performance improvements. I was honestly a little shocked by how much we have accomplished in the last three months when summarizing the changelog.

The headline item here is definitely structural operators. Since we penned the first iterations of TraceQL, Tempo’s query language for distributed tracing, I have been looking forward to this feature more than any other. Structural operators allow a user to very clearly and succinctly search for complex structural conditions in a trace. This is a powerful capability and a unique offering in the open source distributed tracing world. Within three days of merging these operators, we were already using them to debug production issues. After one week, I can’t imagine distributed tracing without them. See below for details!

Also, read on to get a quick overview of the latest updates in Tempo. If you’re looking for something more in-depth, don’t hesitate to jump into the changelog or our Grafana Tempo 2.2 release notes.

New TraceQL features and updates

We have added some really nice features to TraceQL in this release. Honestly, we intended to focus more on performance, but we’ve just built such a powerful foundation on top of Parquet that the features keep flowing.

Structural operators

Structural operators are a standout feature in Tempo 2.2. These simple operators can ask powerful questions about the structure of a trace in very simple ways. This release adds descendant (>>), child (>) and sibling (~) operators. Below are some examples. 

Find any erroring or latent spans beneath a problematic endpoint:

{ span.http.route = "/api/failing" } >> { status = error }

{ span.http.route = "/api/slow" } >> { duration > 1s }

Learn if any calls pass through a specific set of services: 

{ resource.service.name = "foo" } >> { resource.service.name = "bar" } >> { resource.service.name = "baz" }

Learn if service foo ever calls anything else:

{ resource.service.name = "foo" } > { resource.service.name != "foo" }

Learn if database and cache access are ever siblings:

{ span.db.system = "postgres" } ~ { span.db.system = "redis" }

Negated regex

Community member kousikmitra took some time out of their day to contribute the negated regex operator. This allows a quick and easy way to search for fields that do not match a regex. Here’s an easy query to find all spans with SQL statements that aren’t SELECT:

{ span.db.statement !~ "SELECT.*" }

New trace-level intrinsics

Three new trace-level intrinsics were added to the TraceQL query language.

1. traceDuration
   The total duration of the trace. The maximum span end time minus the minimum span start time.
2. rootName
   The name of the root span in the trace.
3. rootServiceName
   The resource.service.name of the root span in the trace.

These intrinsics are significantly more efficient to search than similar options because they exist at the trace level only. rootName and rootServiceName also open up the ability to ask new questions that TraceQL could not answer before.

For example, let’s find errors in any trace that originates from a specific service:

{ rootServiceName = "foo" && status = error }

Use these when possible to improve query times!

select()

Select is a new pipeline operator that allows you to surface fields from spans with little overhead. Select is performed only after all other search criteria have been satisfied so it has little impact on performance.

{ status = error && span.db.name = "postgres" } | select(span.db.statement)

by()

The by operator allows you to split a spanset “by” a specific field for downstream pipelines. It is quite useful if you want to look for something like a trace that has too many errors in a single process:

{ status = error } | by(resource.service.name) | count() > 2

Heads up that Grafana does not currently correctly render all results returned from by() but it’s coming soon!

But wait, there’s more!

It’s amazing that after listing all those TraceQL features, we still have more really nice updates worth mentioning. And yet we do!

For space, I’ll just briefly mention all the performance improvements in this release. Ok, done.

How about this experimental metrics endpoint that lets you dynamically generate metrics from your tracing data?

Or results streaming for TraceQL?

Also, we added support for ipv6, a new API for configuring limits, span filtering in metrics generator, and more! Go check the release notes for all the details.

What’s next in Grafana Tempo?

Tempo is currently firing on all cylinders. The future is bright, and will definitely include additional TraceQL features. Most importantly, we are looking to support some of the fields you cannot currently query like events, links, kvmaps, and lists.

Next, we need to figure out what to do with metrics. As mentioned above, Tempo 2.2 has an experimental dynamic metrics API. We wanted to start getting a feel for dynamic metrics from traces without committing to an API. In the coming months, we intend to take a step back and really flesh out what we want out of dynamic metrics from traces. If you’re interested in participating in the discussion, come join the Tempo community call!

Finally, performance is always top of mind. We have an upcoming revision to the Parquet schema that will give operators even more control over their columns, which will allow for queries to pull significantly less data. Expect great news in Tempo 2.3!

If you are interested in hearing more about Grafana Tempo news or search progress, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, reach out on Twitter, or join our monthly Tempo community call. See you there!

And if you want to get even closer to where the magic happens, why not have a look at our open positions at Grafana Labs? 

The easiest way to get started with Grafana Tempo is with Grafana Cloud, and our free forever tier now includes 50GB of traces along with 50GB of logs and 10K series of metrics. You can sign up for free!