TraceQL: a first-of-its-kind query language to accelerate trace analysis in Tempo 2.0

The much-anticipated release of Grafana Tempo 2.0, which we previewed at ObservabilityCON 2022, will represent a huge step forward for the distributed tracing backend. Among the biggest highlights will be TraceQL, a first-of-its-kind query language that makes it easier than ever to find the exact trace you’re looking for.

Grafana Tempo is an open source, easy-to-use, and highly scalable means for storing queries in a cost-effective manner. It’s deeply integrated with Grafana, Prometheus, and Grafana Loki, and it works with any of the open source tracing protocols, including Jaeger, Zipkin, OpenCensus, and OpenTelemetry.

Tempo has proven to be a highly effective tool for distributed tracing across modern architectures, and we’re excited for where it’s headed. As VP of Technology Tom Wilkie explained at the conference, this release is reminiscent  of the 2.0 release of Grafana Loki, which added the LogQL query language. LogQL searches logs more intelligently so you could better extract metrics and manipulate and analyze data via dashboards. That addition helped transform Loki into what it is today — our second most popular OSS project behind Grafana.

Which is why we’re excited to introduce TraceQL in the upcoming Tempo 2.0 release. With our new trace querying language, you can interactively extract and search through your traces so you can diagnose and respond to root causes faster.

TraceQL: Why Grafana Labs added a query language for traces

Distributed traces contain a wealth of information that can help you track down bugs, identify root cause, analyze performance, and more. And while tools like auto-instrumentation make it easy to start capturing data, it can be much harder to extract value from that data. Today, the Tempo metrics-generator can aggregate traces into service graphs and span metrics, and you can use exemplars to navigate from a spike in API latency to a trace that contributed to that spike — but traces can do so much more.  

Traces are the flow of events throughout your components. They have a tree structure with a root, branches, and leaves; arbitrary key-value data at any location; and, of course, timestamps. With existing tracing solutions, including Tempo 1.0, it can be challenging to find what you’re looking for if you don’t know the exact trace, or if you need broader context about a chain of events. Also, what if we want to use this structure to go beyond just isolating individual events? What if we want to find entire sequences of events?

Unfortunately, none of the existing query languages make it easy to express these complex questions, so we are adding a new query language designed from the ground up to work well with traces. Taking inspiration from PromQL and LogQL, with their familiar and flexible syntax, TraceQL fits perfectly in the Grafana LGTM stack. For example, look at this query shown at ObservabilityCON:

{ .db.statement =~ "INSERT.*"} | avg(duration) > 1s

This query will find traces that made inserts into the database, and where the average duration of those inserts was greater than one second. This type of query would be much harder, if even possible, in other systems.

In the current Tempo search, there’s only basic functionality to look for exact or partial text matches. But TraceQL is data-type aware, so you get flexible, powerful operations for dealing with text, integers, and other data types.

Plus, Tempo 2.0 will incorporate a new storage format based on the open source columnar Apache Parquet project. Tempo will write trace data in Parquet, which is supported by a huge range of databases, data pipelines, and analytics tools. Combining this with TraceQL will help you unlock your data better and build the tooling and analysis that’s important to your organization. 

Grafana Tempo 2.0 release date and what’s next

Grafana Tempo 2.0 will be available in the coming weeks, but you will be able to preview the new search experience in Grafana 9.3.1. We’re excited to see what users do with TraceQL and how it improves their tracing journey. We’ll continue to iterate, including adding the ability to find and analyze traces based on their structure, more advanced processing pipelines, and aggregations. But for now, check out Grafana Tempo 2.0 in the coming weeks and let us know what you think!

Grafana Cloud is the easiest way to get started with metrics, logs, traces, and dashboards. We have a generous free forever tier and plans for every use case. Sign up for free now!