Cross-cluster query federation

NOTE: Cross-cluster query federation is an experimental feature. As such, the configuration settings, command line flags, and specifics of the implementation are subject to change.

Overview

Starting with version 1.4, Grafana Enterprise Metrics (GEM) includes the optional federation-frontend component. The goal of this component is to provide the ability to combine data from multiple GEM clusters into a single PromQL query. The federation-frontend component queries underlying target clusters using the following methods:

Mimir query sharding adapted for the target clusters. This is also known as cluster sharding.
The Prometheus remote_read API
The Metadata APIs

You can run this component on its own, as it doesn’t require any other components of Grafana Mimir. A common use case is combining the data from two GEM clusters that are running in different regions, as shown in the following diagram:

Configure the federation frontend

To configure the federation-frontend component, you need to disable authentication. The federation frontend forwards the Basic authentication and Bearer token supplied by its clients to the underlying target clusters.

Additionally, to start the federation frontend, you must configure the target to be federation-frontend.

You need to configure a list of target clusters within the federation.proxy_targets block; currently, there are no equivalent CLI flags available. Each entry requires a name that contains an identifier that’s exposed using the __cluster__ label in the query results and a url that points to a GEM instance. For GEM, use the following URL: http://<gem-host>/prometheus.

Optionally, you can configure each proxy_target to have Basic auth credentials, which override the user-supplied ones.

Warning: When you configure Basic auth via the proxy_target configuration, its credentials there take precedence over the client-supplied ones. Without other preventive action, any client that can reach the federation frontend can perform queries by using those credentials.

In the following example, two clusters in two different regions are queried via the federation frontend:

multitenancy_enabled: false # The federation frontend does not do any authentication itself
target: federation-frontend # Run the federation frontend only

federation:
  proxy_targets:
    - name: us-west
      url: http://gem-us-west/prometheus
    - name: us-east
      url: http://gem-us-east/prometheus

Configure cross-cluster sharding

Cross-cluster sharding shards queries into different clusters and then runs the queries on those individual clusters before combining them on the federation frontend. This approach improves the performance of cross-cluster queries through providing more distributed computation and less network transfer compared to using remote read for all queries.

The federation-frontend component enables cross-cluster sharding by default. You can configure cross-cluster sharding using the cluster_sharding_enabled setting and the -federation.cluster-sharding-enabled CLI flag. For example:

federation:
  cluster_sharding_enabled: true

Combine metrics from a local GEM cluster and Grafana Cloud Metric stack

The federation frontend allows you to get a combined view of metrics stored in a local GEM cluster and a hosted Grafana Cloud Metrics stack. With the following configuration, you can query both of the clusters as though they were one:

federation:
  proxy_targets:
    - name: own-data-center
      url: http://gem/prometheus
    - name: grafana-cloud
      url: https://prometheus-us-central1.grafana.net/api/prom
      basic_auth:
        username: <tenant-id>
        password: <token>

Warning: This gives any client that can reach the federation frontend access to your metrics data in Grafana Cloud Metrics without further authentication.

By using the authentication credentials of the local GEM cluster, you can execute a query against both clusters. To do so, set the access policy’s token as a variable for subsequent commands:

$ export API_TOKEN="the long token string you copied"

$ curl -s -u "<tenant-id>:$API_TOKEN" -G --data-urlencode "query=count(up) by (__cluster__)" http://federation-frontend/prometheus/api/v1/query | jq

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__cluster__": "own-data-center"
        },
        "value": [1623344524.382, "10"]
      },
      {
        "metric": {
          "__cluster__": "grafana-cloud"
        },
        "value": [1623344524.382, "4"]
      }
    ]
  }
}

Limitations of cross-cluster query federation

This experimental feature has the following limitations:

No result caching in the federation frontend
No support for alerting/ruler on a federation level
No support for endpoints other than range queries, instant queries, and metadata queries. Metadata queries consist of label names queries, labels values queries, and series queries.
No support for exemplars

If your use case is blocked by one of those limitations, then feel free to reach out through our support channels with a feature request.