Incident settings in Grafana IRM
Grafana IRM provides customizable incident settings that allow you to tailor your incident response workflow to match your organization’s processes and terminology. These settings help standardize your incident management process and ensure consistency across teams.
This section explains how to configure key incident settings including labels, severities, statuses, roles, and Slack channel prefixes.
Before you begin
To configure incident settings in Grafana IRM, you need:
- A Grafana Cloud account with admin privileges
- Access to the Grafana IRM application
Access incident settings
To access incident settings:
- Navigate to IRM > Settings in the Grafana Cloud main menu
- Select the Incident tab
Incident labels
Annotating incidents with labels helps you filter and better understand incident trends over time. Use labels to group incidents and tag them with important metadata such as which teams were involved, what services were impacted, and if customers were affected.
You can apply labels to both active and resolved incidents using the + Add Label option in the incident card.
Add incident labels
- In the main menu, navigate to IRM > Settings
- Select the Incident tab
- Locate the Labels section, click + Add new label
- Provide a name and description for the label
- Optional: Assign a color from the hue wheel or leave the default color
- Click Add and repeat steps 2-4 as needed
Edit incident labels
- Navigate to IRM > Settings > Incident
- On an existing label, click the pencil icon
- Edit the name, description, or color as needed
- Click Update
Incident severities
Incident severities provide a defined measurement of the impact of an incident. Consistent and well-defined severities help others in your organization quickly understand the urgency of an incident.
Incident severities may influence your response process, such as how many people are engaged and who to inform. To get the most value out of incident severities, establish clear definitions of each severity level and the expected response within your organization.
The following recommended incident severities are pre-configured in Grafana IRM:
Severity | Description |
---|---|
Critical | Urgently requires immediate attention |
Major | Significant blocking problem that requires help |
Minor | May be affecting customers, but no one is blocked |
Pending | Severity to be decided and should be determined as soon as possible |
Add incident severities
Customize severities to match the severity levels you use in your organization.
- Navigate to IRM > Settings > Incident
- Under Severities, click Add new Severity
- Enter the
label
andlevel
and optionally theicon
anddescription
- Click Save
Edit incident severities
You can customize the severity label
and description
fields to match the severity levels your team uses.
- Navigate to IRM > Settings > Incident
- Under Severities, click the pencil icon on an existing severity
- Edit the label and description fields as needed
- Click Save
Note
To ensure your custom severities levels are mapped to the corresponding severity level in Grafana IRM, define your highest severity with the pre-configured critical severity.
For example, suppose your severity levels are defined as P1-P4:
Custom severity | Pre-configured severity |
---|---|
Priority 1 | Critical |
Priority 2 | Major |
Priority 3 | Minor |
Priority 4 | Pending |
Archive and unarchive incident severities
If needed, you can archive severities that don’t align with the severity levels your team uses.
- Navigate to IRM > Settings > Incident
- Under Severities, click archive on an existing severity to archive it
- To unarchive, find the severity in the archived list and click unarchive
Delete incident severities
If needed, you can delete severities that don’t align with the severity levels your team uses.
- Navigate to IRM > Settings > Incident
- Under Severities, click the trash icon on the desired severity
- Click Confirm remove to delete the severity
Incident statuses
The status of an incident indicates whether the incident is ongoing or if the issue has been resolved. The incident status should immediately indicate whether or not incident response is still in progress.
The following incident statuses are pre-configured in Grafana IRM:
Status | Description |
---|---|
Active | The incident is happening now |
Resolved | The incident is resolved |
Edit incident statuses
Customize what you call active and resolved incidents:
- Navigate to IRM > Settings > Incident
- Under Statuses, click the pencil icon for an existing status
- Edit the label and description as needed
- Click Save
Slack channel prefixes
Having a central and dedicated place for incident response to take place can be helpful to keep communication organized during an incident and improve analysis for post-incident reviews.
Prefixes determine the name of automatically created incident channels. For example, if the channel prefix is security
then the channel name will be #security-date-title
.
Tip
To maintain searchable and easy to manage incident channels, consider customizing your channel prefixes for different teams or services.
Edit channel prefixes
To customize Slack channel prefixes:
- Navigate to IRM > Settings > Incident
- Scroll to the Prefixes section
- To add a prefix, click + Add Prefix, provide a name and description, and click Add
- To edit a prefix, click the pencil icon on an existing prefix and click Update after making changes
Once your prefixes are defined, you can specify which prefix to use when you declare an incident in Grafana IRM.
Incident roles
Incident roles help to identify who’s involved and what they’re responsible for. Grafana IRM is pre-configured with two recommended key roles:
- Commander: Oversees the incident by managing communication, tasks, and necessary updates
- Investigator: Responsible for diagnosing and resolving the incident
You also have the flexibility to customize these roles to fit your organization’s specific needs. This includes the ability to create, edit, delete, archive, and define Key Roles.
Add a new role
- Navigate to IRM > Settings > Incident
- Scroll to the Roles section
- Click + Add new role
- Enter a name and description for the role
- If this role is critical to your incident process, check the box labeled Important to designate it as a Key Role
- Click Add to create the role
Edit, archive, or delete roles
- To edit a role, click the pencil icon, make your changes, and click Update
- To archive a role, click the lock icon and click Confirm archive (archived roles are inactive but can be restored if needed)
- To delete a role permanently, click the trash icon and click Confirm remove (this action is irreversible)