Documentation Grafana Cloud Alerts and IRM IRM Configure Alert escalation and routing Escalation chains

Configure escalation chains

Escalation chains define the sequence of actions taken when an alert is triggered in Grafana IRM. They automate your incident response workflow by executing ordered steps until an alert is acknowledged, resolved, or all steps complete.

About escalation chains

An effective escalation chain:

• Ensures alerts reach the right people at the right time
• Implements tiered response procedures based on severity
• Automates notification and escalation processes
• Prevents alerts from being missed

Create and manage escalation chains

Create a new chain

1. Navigate to Escalation Chains in the left sidebar
2. Click New escalation chain
3. Enter a unique name and optional team assignment
4. Click Add escalation step to add steps to your chain
5. Configure steps and arrange them using drag-and-drop
6. Click Save

Edit or delete a chain

• To edit: Select a chain and click Edit, then make changes and save
• To delete: Select a chain, click Delete, and confirm

Types of escalation steps

Notification steps

• Notify users: Send notifications to specific users or groups
• Notify from on-call schedule: Alert currently on-call users
• Notify all team members: Alert everyone in a specified team
• Notify Slack channel/user group: Send notifications to Slack users
• Round robin notifications: Rotate through a list of users sequentially

Timing and control steps

• Wait: Pause for a specified duration before proceeding
• Repeat escalation: Loop the chain up to five times
• Time-based escalation: Continue only during specified time periods
• Threshold-based escalation: Escalate only after a threshold of alerts

Action steps

• Resolve incident automatically: Mark the alert group as “Resolved automatically” without user intervention
• Trigger outgoing webhook: Send data to an external system using a configured outgoing webhook
• Declare incident: Create a new incident with specified severity. Limited to one incident per route at a time; additional alerts are grouped into the active incident

Notification types

When configuring notification steps, you can specify which type of notification to use:

• Default notifications: Standard alert notifications as configured in the user’s profile
• Important notifications: High-priority notifications that may use different channels or frequencies

Each user can customize their notification preferences in their profile settings, including different channels (Slack, email, SMS, phone, mobile push), frequencies, and escalation conditions.

Example escalation chains

Basic notification chain

1. Notify primary on-call person (important)
2. Wait 5 minutes
3. Notify primary again (important)
4. Wait 10 minutes
5. Notify backup on-call person (important)

Business hours vs. after hours chain

1. Time-based escalation (continue if 9 AM - 5 PM, Monday-Friday)
   • If true: Notify business hours team
   • If false: Proceed to step 2
2. Notify 24/7 on-call engineer (important)

Critical system chain

1. Notify all team members (important)
2. Notify Slack channel #critical-alerts
3. Wait 2 minutes
4. Declare incident with severity “critical” if not acknowledged

Best practices

• Start simple: Begin with basic notification steps before adding complexity
• Test thoroughly: Verify chains with non-production alerts first
• Document your chains: Maintain explanations of each chain’s purpose
• Include wait steps: Add appropriate delays between notifications
• Use important notifications sparingly: Reserve for truly critical alerts
• Consider time zones: Create chains respecting global team distribution

Next steps

• Configure alert routes to direct alerts to your chains
• Configure alert templates to customize alert appearance
• Set up on-call schedules to automate rotation