Documentation Grafana Cloud Alerts and IRM IRM Configure Integrations Integration reference OnCall integrations Grafana Alerting

Grafana Alerting integration for Grafana IRM

Grafana Alerting for Grafana IRM can be set up using two methods:

• Grafana IRM is connected to the same Grafana instance being used to manage Grafana IRM.
• Grafana IRM is connected to one or more Grafana instances, separate from the one being used to manage Grafana IRM.

Configure Grafana Alerting in same Grafana instance

Use the following method if you are connecting Grafana IRM with alerts coming from the same Grafana instance from which Grafana IRM is being managed.

1. In Grafana IRM, navigate to IRM > Integrations > Monitoring Systems

2. Click Quick connect in the Grafana Alerting tile. This will open a New Grafana Alerting integration configuration window

3. In the opened window, enter a name and description for the integration and choose existing or create a new contact point. This contact point will send alerts to the created integration

   Note

   You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana IRM. For more information, see Contact points in Grafana Alerting

4. Determine the escalation chain for the new integration by either selecting an existing one or by creating a new escalation chain

5. In Grafana Cloud Alerting, navigate to Alerting > Contact Points and find a contact point with a name matching the integration you created in Grafana IRM

6. Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana IRM

Configure external Grafana Alerting from other Grafana Instance

Connect Grafana IRM with alerts coming from a Grafana instance that is different from the instance that Grafana IRM is being managed:

1. In Grafana IRM, navigate to IRM > Integrations > Monitoring Systems

2. Select the Alertmanager tile

3. Enter a name and description for the integration, click Create

4. A new page will open with the integration details. Copy the IRM Integration URL from HTTP Endpoint section

5. Go to the other Grafana instance to connect to Grafana IRM and navigate to Alerting > Contact Points

6. Select New Contact Point

7. Choose the contact point type webhook, then paste the URL generated in step 3 into the URL field

   Note

   You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana IRM. For more information, see Contact points in Grafana Alerting

8. Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana IRM

Note about grouping and autoresolution

Grafana IRM relies on the Grafana Alerting grouping and autoresolution mechanism to ensure consistency between alert state in IRM and AlertManager. It’s recommended to configure grouping on the Grafana Alerting side and use default grouping and autoresolution templates on the IRM side. Changing this templates might lead to incorrect grouping and autoresolution behavior.

Note about legacy integration

Before we were using each alert from Grafana Alerting group as a separate payload:

json Copy

{
  "labels": {
    "severity": "critical",
    "alertname": "InstanceDown"
  },
  "annotations": {
    "title": "Instance localhost:8081 down",
    "description": "Node has been down for more than 1 minute"
  },
  ...
}

This behavior was leading to mismatch in alert state between IRM and Grafana Alerting and draining of rate-limits, since each Grafana Alerting alert was counted separately.

We decided to change this behavior to respect Grafana Alerting grouping by using AlertManager group as one payload.

json Copy

{
  "alerts": [...],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "job": "node",
    "alertname": "InstanceDown"
  },
  "commonAnnotations": {
    "description": "Node has been down for more than 1 minute"
  },
  "groupKey": "{}:{alertname=\"InstanceDown\"}",
  ...
}

You can read more about AlertManager Data model here.

After-migration checklist

1. Send a new demo alert to the migrated integration
2. Adjust routes to the new shape of payload. You can use payload of the demo alert from previous step as an example
3. If outgoing webhooks utilized the alerts payload from the migrated integration, it’s needed to adjust them as well

Related topics

• Grafana IRM integrations overview
• Alert escalation and routing
• Grafana Alerting documentation