Menu
Documentationbreadcrumb arrow Grafana Enterprise Tracesbreadcrumb arrow Configurebreadcrumb arrow Federation-frontend
Enterprise RSS

Using cross-tenant and cross-cluster query federation

Grafana Enterprise Traces (GET) can perform cross-tenant and cross-cluster query federation using the optional federation-frontend service. This service aggregates the data from multiple GET clusters and from multiple tenants in a single trace lookup or search query. It works by reverse proxying the read path of GET clusters and merging the results into a single response.

The federation-frontend target itself requires a configured GET installation installed using the tempo-distributed Helm chart. No other GET components are required.

You can run the federation-frontend service on its own.

Cross-tenant query federation

GET supports creating access policies that can span multiple tenants. Viewers in Grafana Enterprise can see data coming from more than one tenant simultaneously.

To read more about tenant federation, see the Cross-tenant query federation page.

Cross-cluster query federation

Cross-cluster query federation allows a team that is globally responsible for all observability to make a single query across the GET clusters for all regions to retrieve trace results.

To read more about cluster federation, see the Cross-cluster query federation page.

Limitations of the federation-frontend

This feature has one known limitation:

  • No result caching in the federation-frontend

If your use-case is blocked by a limitation, please reach out through our support channels with a feature request.

Cross-tenant and cross-cluster TraceQL queries

If you have configured multi-stack Tempo data source, you can perform TraceQL queries across those stacks and tenants. Refer to Set up a multi-stack Tempo data source in Grafana for information about configuring the Tempo data source.

Queries performed using the cross-tenant configured data source, in either Explore or inside of dashboards, are performed across all the tenants that you specified in the X-Scope-OrgID header.

TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,

{ span.attr1 = "bar" } && { span.attr2 = "foo" }

TraceQL evaluates a contiguously stored trace. If these two conditions are satisfied in separate tenants, then Tempo doesn’t return the trace.