Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Grafana Enterprise Metrics Alerting
Both the GEM alertmanager and the ruler work nearly identically to their Mimir counterparts: the Grafana Mimir ruler, and Grafana Mimir Alertmanager. The only difference between GEM and Mimir is the configuration of the communication between the GEM ruler and GEM alertmanager. Additionally, this difference only applies when you are running GEM with the token-based authentication model.
Context
When the ruler evaluates alerting rules for a tenant, it sends the alertmanager the notifications for every firing alert.
In GEM, the requests between the ruler and the alertmanager must be authenticated and authorized for each tenant.
The requests must have basic authentication with the tenant ID and an access token from an access policy,
which has the alerts:write
scope for that tenant.
The ruler has configuration options that allow you to set the basic authentication credentials for communicating with the alertmanager.
The problem is that these credentials can only be configured per ruler, not per tenant.
For example, if the configuration uses tenant-A
as the tenant ID, then the alerts from all tenants in the ruler are
sent to the tenant-A
tenant in the alertmanager. This means that alerts will use the
alertmanager tenant configuration, such as alert receivers, alert routes, and inhibition rules, of tenant-A
.
Configure communication between the ruler and the alertmanager with enterprise authentication
The ruler configuration options can take a special value for the basic authentication user. This value allows Alertmanager to detect the tenant from the ruler and use the appropriate Alertmanager tenant configuration for routing the alerts.
Create an access policy with the
alerts:write
scope to all tenants in the cluster.For detailed instructions, see Create an access policy.
Create an access token for that policy.
Create an access policy also shows you how to create a token for the access policy.
Add the following to your GEM configuration:
ruler: alertmanager_url: http://alertmanager alertmanager_client: basic_auth_username: __alertmanager__ basic_auth_password: <TOKEN>
Alternatively, you can set the CLI arguments
-ruler.alertmanager-url
,-ruler.alertmanager-client.basic-auth-username
, and-ruler.alertmanager-client.basic-auth-password
:Replace
http://alertmanager
with the address of the alertmanager component or components in your GEM cluster.The alertmanager URL is already configured by the
mimir-distributed
Helm chart, so you can omit it from your values file.Replace
<TOKEN>
with the token that you created in the previous step.You can also inject the token in the configuration via an environment variable. For details, see Use environment variables in the configuration.
(Optional) Configure the tenants in Alertmanager.
(Optional) Configure the rules in the ruler.
See Rules in the
mimirtool
documentation.