Documentation Grafana Enterprise Metrics Manage a tenant Using tenant federation

Overview

GEM supports creating access policies that can span multiple tenants. Doing so enables viewers in Grafana Enterprise to view data coming from more than one tenant simultaneously.

This page will cover the ability to query data from multiple tenants at once, as well as the ability to create alerting and recording rules that use data from multiple tenants at once.

Prerequisites

• A configured Grafana Enterprise Metrics cluster. To create a GEM cluster, refer to Set up GEM.

• This guide will assume there are two tenants: team-engineering and team-finance. To create a tenant, refer to Set up a GEM tenant.

Cross-tenant queries

Set up an access policy with tenant federation and a token

To allow queries to span both GEM tenants, which are for demonstration purposes named team-engineering and team-finance, create a new access policy called leadership. The necessary steps are:

1. Create a new access policy leadership.

2. Enable the metrics:read scope.

3. Add the tenants team-engineering and team-finance. Alternatively, you can add the special tenant name * to create an access policy that has access to all tenants in the cluster.

4. Create a new token for the access-policy and store the token in your clipboard:

Set up a Grafana data source using the access policy

1. Create a new Prometheus data source from the Grafana configuration menu.

2. Enter the URL of your GEM cluster, for example http://enterprise-metrics/prometheus.

3. From the Auth section, enable Basic auth.

4. In the User field, enter: team-engineering|team-finance where all the names of the tenants that you want to query across are separated by the | pipe character.

5. In the Password field, paste the token created in the token creation process.

Queries that are performed using this data source in either Explore or inside of dashboards are performed across all of the tenants that you specified in the User field, and are processed as if all of the data were in a single tenant.

To submit a query across all tenants that your access policy has access to, you can either:

1. Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across tenant1, tenant2, and tenant3 you would enter tenant1|tenant2|tenant3.
2. Set the username to a wildcard character *. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.

When using an access policy that has a wildcard (*) as the username, you can query all tenants for that cluster by also specifying * as the username in your data source URL.

Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.

Cross-tenant alerting and recording rules

GEM supports alerting and recording rules that operate on data from multiple tenants.

To create a federated alerting or recording rule, use the source_tenants field when defining your rule group. This field defines which tenants the rule will read data from. Any rule in a group with a non-empty source_tenants field is considered a federated rule.

Using the same example tenants we defined above, we’ll create a federated rule group that evaluates data coming from tenants team-engineering and team-finance, and stores the resulting series in team-engineering.

First, enable query tenant federation and rules tenant federation. The configuration parameters for each are as follows:

If you are using the mimir-distributed Helm chart to deploy GEM, then you can enable these parameters by including the following configuration in your values file:

yaml Copy

mimir:
  structuredConfig:
    tenant_federation:
      enabled: true
    ruler:
      tenant_federation:
        enabled: true

Next, create your rule and rule group in a file named groups.yaml.

yaml Copy

namespace: fed-rule-group
groups:
  - name: example
    interval: 5m
    source_tenants: [team-engineering, team-finance]
    rules:
      - record: job:http_inprogress_requests:sum
        expr: sum by (job) (http_inprogress_requests)

This example recording rule is calculating a sum of http_inprogress_requests by job every 5 minutes. Because the source tenants are team-engineering and team-finance, GEM will use all series with metric name http_inprogress_requests in both of these tenants when calculating the sum.

To create the cross-tenant rule group, use the following mimirtool command:

sh Copy

mimirtool rules load --address http://enterprise-metrics/ --user team-engineering --key $API_TOKEN groups.yaml

The username used in the mimirtool command call is what dictates where the output of the rule will be stored. In our example above, the output of the rule will be stored in team-engineering because we have --user team-engineering.

The access policy to which the API token belongs must have metrics:read and rules:write scopes and apply to all the tenants in source_tenants (team-engineering and team-finance in our example) and the owning tenant (team-engineering in our example). Otherwise, an HTTP unauthorized status (401) is returned and the rule group is not created.

Note: Creating federated rules is not supported with OIDC-based tokens that reference more than one GEM access policy. To create federated rules, either use a GEM token or an OIDC-based token with only a single GEM access policy that has metrics:read and rules:write scopes and applies to the owning tenant and the source tenants.

When the access policy used to create the federated rule group is deleted or updated to remove the metrics:read scope or to remove any of the source_tenants, that rule group will stop evaluating at the next rules synchronization. The synchronization interval is set by the -ruler.poll-interval configuration parameter. By default, this is 1 minute.

You can find more details about cross-tenant rules in the Grafana Mimir API documentation.