Grafana Enterprise Metrics Alerting

Both the GEM alertmanager and the ruler work nearly identically to their Mimir counterparts: the Grafana Mimir ruler, and Grafana Mimir Alertmanager. The only difference between GEM and Mimir is the configuration of the communication between the GEM ruler and GEM alertmanager. Additionally, this difference only applies when you are running GEM with the token-based authentication model.

Context

When the ruler evaluates alerting rules for a tenant, it sends the alertmanager the notifications for every firing alert.

In GEM, the requests between the ruler and the alertmanager must be authenticated and authorized for each tenant. The requests must have basic authentication with the tenant ID and an access token from an access policy, which has the alerts:write scope for that tenant. The ruler has configuration options that allow you to set the basic authentication credentials for communicating with the alertmanager.

The problem is that these credentials can only be configured per ruler, not per tenant. For example, if the configuration uses tenant-A as the tenant ID, then the alerts from all tenants in the ruler are sent to the tenant-A tenant in the alertmanager. This means that alerts will use the alertmanager tenant configuration, such as alert receivers, alert routes, and inhibition rules, of tenant-A.

Configure communication between the ruler and the alertmanager with enterprise authentication

The ruler configuration options can take a special value for the basic authentication user. This value allows Alertmanager to detect the tenant from the ruler and use the appropriate Alertmanager tenant configuration for routing the alerts.

Create an access policy with the alerts:write scope to all tenants in the cluster.
For detailed instructions, see Create an access policy.
Create an access token for that policy.
Create an access policy also shows you how to create a token for the access policy.
Add the following to your GEM configuration:
yaml
```
ruler:
  alertmanager_url: http://alertmanager
  alertmanager_client:
    basic_auth_username: __alertmanager__
    basic_auth_password: <TOKEN>
```
Alternatively, you can set the CLI arguments -ruler.alertmanager-url, -ruler.alertmanager-client.basic-auth-username, and -ruler.alertmanager-client.basic-auth-password:
1. Replace http://alertmanager with the address of the alertmanager component or components in your GEM cluster.
  The alertmanager URL is already configured by the mimir-distributed Helm chart, so you can omit it from your values file.
2. Replace <TOKEN> with the token that you created in the previous step.
  You can also inject the token in the configuration via an environment variable. For details, see Use environment variables in the configuration.
(Optional) Configure the tenants in Alertmanager.
See Tenant configurations.
(Optional) Configure the rules in the ruler.
See Rules in the mimirtool documentation.