Alerts and IRM

IRM

Configure

Alert escalation and routing

Alert routes

Grafana Cloud

Configure alert routes

Alert routes in Grafana IRM determine how incoming alerts are processed based on their content. By creating routes with specific conditions, you can direct different alerts to the appropriate escalation chains, ensuring the right teams are notified for each type of alert.

About alert routes

Routes act as filters that evaluate incoming alerts and direct them to specific escalation chains. When an alert arrives, Grafana IRM evaluates it against each route in order, and the first matching route determines how the alert will be handled.

Key features of alert routes:

Routes use Jinja2 templates to create flexible matching conditions
Routes are evaluated in order (top to bottom), with the first match taking precedence
Each route connects to a specific escalation chain
Routes can publish alerts to communication channels (Slack, Microsoft Teams)
Routes can be assigned to specific teams

How routing works

The alert routing process follows these steps:

An alert arrives through an integration (like Prometheus, Grafana Alerting, or a webhook)
The alert’s payload is evaluated against each route’s routing template in sequence
The first route with a template that evaluates to True is selected
The alert is sent to the escalation chain specified by the matching route
If configured, the alert is also published to the specified communication channels

Create and manage routes

To create a new alert route:

In Grafana IRM, navigate to OnCall > Integrations.
Select the integration where you want to add a route.
Click Add route.
In the Routing Template section, define conditions for alert matching (see Routing templates).
Select the appropriate escalation chain from the Escalation Chain dropdown.
- If needed, click Add new escalation chain to create a new chain in a separate tab.
- After creating the chain, return to the routes page and click Reload list.
Optional: In the Publish to ChatOps section, select communication channels for alert notifications.
Click Save to create the route.

To manage existing routes:

Change route order: Use the up/down arrows to reorder routes. Remember that routes are evaluated in order, and the first match wins.
Edit a route: Click the route to open its settings, make changes, and save.
Delete a route: Click the three dots menu on the route and select Delete Route.

Routing templates

Routing templates use Jinja2 syntax to create conditions for matching alerts. A template must evaluate to True for the route to be selected.

Basic routing examples

Match alerts with “database” in the title:

{{ "database" in payload.title | lower }}

Match critical severity alerts:

{{ payload.severity == "critical" }}

Match alerts from a specific service:

{{ payload.service == "payment-processor" }}

Match alerts with specific labels:

{{ payload.labels.environment == "production" }}

Combined conditions

You can combine multiple conditions using logical operators:

Match critical database alerts:

{{ "database" in payload.title | lower and payload.severity == "critical" }}

Match either production or staging environment alerts:

{{ payload.labels.environment == "production" or payload.labels.environment == "staging" }}

Match non-development environment alerts:

{{ payload.labels.environment != "development" }}

Accessing payload data

The routing template has access to the complete alert payload through the payload variable. The exact structure depends on the integration and incoming alert format.

Common payload fields include:

payload.title - Alert title
payload.message - Alert message
payload.severity - Alert severity level
payload.labels - Alert labels (as a dictionary)
payload.annotations - Alert annotations (as a dictionary)

Use {{ payload }} in a template to print the entire alert structure for debugging.

Label-based routing

For integrations that support structured labels (like Prometheus, Grafana Alerting), you can route based on label values:

{{ labels.team == "database" }}

This provides additional flexibility compared to payload-based routing, especially for alerts with well-structured metadata.

Note: Label-based routing is available exclusively on Grafana Cloud.

Example routing strategies

Team-based routing

Direct alerts to teams based on labels or service names:

# Database team route
{{ labels.team == "database" or "database" in payload.title | lower }}

# Frontend team route
{{ labels.team == "frontend" or "frontend" in payload.title | lower }}

# Default route (catches all remaining alerts)
{{ True }}

Severity-based routing

Create different escalation chains based on alert severity:

# Critical alerts route
{{ payload.severity == "critical" or labels.severity == "critical" }}

# Warning alerts route
{{ payload.severity == "warning" or labels.severity == "warning" }}

# Info alerts route
{{ payload.severity == "info" or labels.severity == "info" }}

Environment-based routing

Route differently based on the environment:

# Production route (high priority escalation)
{{ labels.environment == "production" }}

# Staging route (medium priority escalation)
{{ labels.environment == "staging" }}

# Development route (low priority escalation)
{{ labels.environment == "development" }}

Best practices

Order matters: Place more specific routes above more general ones
Default route: Create a catch-all route at the bottom that matches all alerts ({{ True }})
Test templates: Use the template preview feature to test your conditions with sample payloads
Use descriptive names: Give routes clear names that describe their purpose
Documentation: Add comments in complex templates to explain the logic
Simplify conditions: Break complex conditions into multiple routes when possible
Review regularly: Periodically review routes to ensure they’re still appropriate