Notifications in Grafana IRM

Grafana IRM provides multiple ways to notify users about alerts and incidents, ensuring teams stay informed and can respond quickly. This section explains the core notification concepts, how notification rules work, and how to set up notifications effectively.

Notification channels and methods

Grafana IRM supports multiple notification channels, listed here in their typical order of priority:

Mobile push notifications (Recommended for primary notifications)
Messaging apps (Slack, Microsoft Teams, Telegram, etc.)
Phone calls & SMS (Use as a backup for critical notifications)
Email (Use for non-urgent alerts or summary notifications)

For applications that aren’t directly supported, you can use outgoing webhooks.

Understanding notification rules

Notification rules determine how and when users receive alerts about incidents. These rules allow each team member to customize their notification preferences based on their role, availability, and preferred communication channels.

Types of notification rules

Grafana IRM supports two main types of notification rules:

Default notification rules: These rules specify how a user is notified for standard escalation events.
Important notification rules: These rules are triggered when an escalation event is marked as important, bypassing the default notification settings to ensure critical alerts are properly delivered.

Default notification rules

Default notification rules determine how users are notified for most incidents and alerts. These rules apply to all standard notifications that aren’t specifically marked as important.

Default notification rules provide a baseline notification strategy for each user. They specify:

Which notification channels to use (email, Slack, SMS, phone calls, mobile app)
The order in which different notification methods are used
The timing and frequency of notifications

Example default notification rule sequence:

Slack mention (immediate)
Wait 5 minutes
Mobile push notification
Wait 10 minutes
Phone call or SMS

Important notification rules

Important notification rules are designed for critical incidents that require immediate attention. When an escalation step is marked as “important,” it bypasses the default notification rules and uses these special rules instead.

Important notification rules provide an alternative notification strategy for high-priority alerts. Compared to default rules, these typically:

Use more immediate and intrusive communication methods
Have more aggressive notification patterns
May bypass quiet hours settings
Are designed to ensure critical incidents are not missed

Example important notification rule sequence:

Slack + Mobile push notification (immediate)
Wait 5 minutes
Phone call

How notifications work with escalation chains

Grafana IRM’s notification system has three key components that work together:

Notification channels: The methods used to send alerts (mobile app, Slack, phone, email)
Notification rules: User-defined sequences of channels and timing for alerts
Escalation chains: Who to notify and when to escalate

When an incident occurs:

The incident activates an escalation chain
The escalation chain identifies which users to notify
Each user receives notifications according to their personal notification rules:
- Regular steps use their default notification rules
- Steps marked as “important” use their important notification rules
If a user doesn’t acknowledge the alert within the specified time, the escalation continues to the next step

Setting up the connection

To connect notification rules to escalation chains:

First, ensure all users have configured their notification channels
Each user should set up their default and important notification rules
Create escalation chains that include the appropriate users or schedules
Mark critical escalation steps as “important” to trigger important notification rules
Test the complete notification flow to verify it works as expected

Notification quick setup guide

Tip
This is a simplified overview. For detailed instructions, refer to the specific sections on notification channels and notification rules.

Set up channels: Connect your mobile app, Slack, phone number, and other notification channels
Configure rules: Set up default and important notification rules with your preferred notification sequence
Test notifications: Verify that all notification methods work correctly before relying on them
Link to escalation chains: Connect notification rules to your escalation chains for incident handling

Best practices for effective notifications

For the most reliable notification setup:

Use multiple channels: Don’t rely on a single notification method
Layer notifications: Start with non-intrusive methods (Slack) and escalate to more intrusive ones (phone calls)
Test regularly: Ensure all notification paths work as expected