Get started with Grafana IRM
Grafana IRM (Incident Response & Management) is a unified platform that brings together on-call scheduling, alert escalation, and incident resolution in Grafana Cloud. This guide walks you through the initial setup and key concepts to help you get started quickly.
Before you begin
Before you begin, ensure that you have:
- A Grafana Cloud account and access to IRM.
- Admin or Editor permissions in your Grafana Cloud environment.
Step 1: Access Grafana IRM
- Log in to your Grafana Cloud account.
- In the Grafana main menu, click on Alerts & IRM → IRM.
- If this is your first time accessing IRM, you’ll be directed to the Get started tab of the IRM app.
Step 2: Configure integrations
Integrating your monitoring systems with Grafana IRM allows you to receive alerts and trigger incident response workflows.
- In the Grafana Cloud main menu, navigate to Alerts & IRM → IRM → Integrations.
- Click + Add integration and choose a monitoring tool (e.g., Grafana Alerting, Prometheus, Datadog).
- Configure the integration settings, including API keys or webhook URLs as required.
- For each integration, you can configure:
- Routes: Define which escalation chains handle different alerts
- Alert Templates: Control how alerts are displayed and processed
- Labels: Categorize and route your alerts based on labels
- Click Save to store your configuration and verify connectivity.
For Grafana Alerting integration:
- Navigate to Alerts & IRM → Alerting → Contact points.
- Create a new contact point of type Grafana OnCall.
- Configure your notification policies to use this contact point.
For more information, refer to Configure integrations.
Step 3: Set up on-call schedules
Setting up on-call schedules ensures the right people receive alerts when issues arise.
- Navigate to Alerts & IRM → IRM → Schedules.
- Click + Create schedule.
- Configure your schedule:
- Give it a descriptive name
- Configure rotation intervals
- Add team members to the rotation
- Configure notification preferences
- Click Save to create your schedule.
You can create multiple schedule layers (primary, secondary, etc.) for tiered response and assign different team members to each layer.
For more details, refer to On-call schedules.
Step 4: Create escalation chains
Escalation chains define what happens when an alert is triggered and ensure that the right responders are notified.
- Navigate to Alerts & IRM → IRM → Escalation chains.
- Click + Add escalation chain and give it a descriptive name (e.g., “Critical Alerts Chain”).
- Define the steps for notification escalation:
- Add on-call schedules or specific users as escalation steps
- Set wait times between escalation steps
- Add backup contacts for critical alerts
- Click Save to create your escalation chain and use it with your alert integrations.
Tip
Once an alert group is acknowledged, new alerts joining that group won’t trigger new escalation actions.
For more details, refer to Escalation chains.
Step 5: Configure alert routing
If you want different types of alerts to follow different escalation paths:
- Navigate to Alerts & IRM → IRM → Integrations and select your integration.
- Click on the Routes tab.
- Click + Add route and specify:
- Matching conditions based on alert content (e.g.,
severity: critical
) - The escalation chain to use for matching alerts
- ChatOps notification settings
- Matching conditions based on alert content (e.g.,
- Click Save and order your routes with the most specific rules first.
Routes are evaluated in order, and the first matching route is used to determine which escalation chain handles the alert.
For more details, refer to Alert routing.
Step 6: Configure user notification rules
Set up how team members are notified of alerts and incidents:
- Navigate to Alerts & IRM → IRM → Users.
- Each user should configure their personal notification policies:
- Add notification channels to enable different contact methods (email, SMS, phone, push notifications)
- Set up notification rules and escalation timing
- To verify setup, users can send a test notification to ensure delivery.
Choose between notification methods such as Slack, Microsoft Teams, Telegram, SMS, phone calls, or push notifications via the IRM mobile app.
For more information, refer to Notifications.
Step 7: Test your setup
Verify your IRM configuration is working correctly:
- Navigate to Alerts & IRM → IRM → Integrations.
- Find the integration you set up and click Send test alert.
- Confirm that:
- The alert appears in the Alert groups section
- The correct on-call responder is notified based on the escalation chain
- Alert routing logic (e.g., severity-based escalations) is functioning as expected
- Respond to the test alert and verify the response is registered.
Step 8: Declare and manage incidents
When an issue arises, you can declare an incident and mobilize responders:
- In the main menu of Grafana Cloud, click Alerts & IRM → IRM → Incidents.
- Click Declare incident.
- Provide a title, select a severity, and add labels as needed.
- Click More options to select channel prefix and status.
- Click Declare incident to create the incident.
- Assign responders and manage the incident through its lifecycle.
- Conduct a post-incident review (PIR) to document learnings.
You can also declare incidents from various locations in Grafana Cloud, including panels, firing alerts, and IRM alert groups.
For more details, refer to Incident management.
Next steps
Now that you have the basics set up, explore these additional capabilities:
- Install the Grafana IRM mobile app to manage incidents on the go
- Set up the IRM Slack integration for chat-first incident response
- Configure additional integrations for your monitoring tools
- Customize incident management settings to fit your team workflows
- Configure alert grouping and routing for more efficient alert handling
- Explore insights and reporting for historical analysis and improvements
- Automate workflows using Terraform and IRM APIs to streamline alerting and incident management