Documentation Grafana Cloud Alerts and IRM IRM Configure Integrations Webhooks Outgoing webhooks Incident outgoing webhooks

Configure Incident outgoing webhooks

Incident outgoing webhooks allow Grafana IRM to send real-time incident updates to external systems. These webhooks can notify specified URLs when incidents are created, updated, or closed, enabling you to create custom workflows based on incident events.

About Incident outgoing webhooks

Outgoing webhooks send HTTP POST requests to your specified endpoints when incident events occur. You can use these webhooks to:

• Integrate incident data with external systems
• Trigger automated workflows based on incident events
• Customize incident response processes
• Build connections with other tools in your infrastructure

Note: Outgoing webhooks work well in conjunction with the Grafana IRM APIs.

Example use case

A common use case for outgoing webhooks is automating Jira issue creation:

Multiple teams at your organization use Grafana IRM, so you use labels to identify which incidents belong to which teams. Your current incident response workflow includes a manual step to create an issue in the related teams’ Jira project when an incident is declared.

To automate this process, you can configure an outgoing webhook to:

1. Trigger when an incident is declared
2. Read the incident event labels
3. Use those labels to create an issue in the correct Jira project for the right team

Set up Incident outgoing webhooks

To install and configure the outgoing webhook integration:

1. From the Grafana IRM Integrations tab, click Apps, and select Outgoing webhooks
2. Click Install Integration
3. Once installed, a signing secret is automatically generated for verifying webhook requests

Configure webhook triggers

After installation, you need to specify when the webhook should fire:

1. Click Run when an event fires
2. Enter a target endpoint URL for the outgoing webhook POST request
3. Click Add event action to save your webhook configuration

Trigger webhooks based on events

You can configure webhooks to trigger on the following actions:

Available event types

Define filters for event triggers

You can create specific filters to control when webhooks are triggered. Each filter condition uses key-value pairs with logical operators.

Supported filter conditions

Logical operators

• and(): Combines conditions with logical AND (default behavior if no operator is specified)
• or(): Combines conditions with logical OR
• -: Negates a condition

Filter examples

• -severity:critical: Matches incidents with severity other than critical
• and(label:customer severity:critical): Matches critical incidents with the customer label
• (label:customer severity:critical): Simplified form of the above (AND is default)
• and(severity:critical or(label:"Customer A" label:"Customer B")): Matches critical incidents with either Customer A or Customer B label

Note: Event actions trigger at most once per incident. For example, if a label:customer filter is applied, adding, removing, and re-adding a customer label will result in only one webhook call.

Verify webhook requests

All webhook requests include a GI-Signature header that you can use to verify the authenticity of the request. This signature contains a timestamp and HMAC-SHA256 signature.

Signature format

Copy

GI-Signature:t=1677589543,v1=12d6e3e06e66f2a32b4027827fbb95ce139ee8381a1cec1c40a02ec5f877797c

The v1 signature is calculated using HMAC-SHA256 with the following signing string:

Copy

signing-string = body-hash + ":" + timestamp + ":" + signature-version

Signature verification example

Given:

• Payload: { 'somejson': true }
• Secret: some-secret-value
• Target URL: https://myendpoint.com/some/path
• Timestamp: 1677589543

The signature is calculated as:

Copy

signature-version = "v1"
body-hash = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM="
signing-string = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1"
hmac = a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

GI-Signature:t=1677589543,v1=a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

Command line verification

You can verify signatures using OpenSSL:

bash Copy

# Calculate the body hash
echo -n "{ 'somejson': true }" | openssl dgst -sha256 -binary | openssl enc -base64
Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=

# Calculate the HMAC signature
echo -n "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1" | openssl dgst -sha256 -hmac "some-secret-value"
a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

Sample outgoing webhook

Below is a sample webhook payload for an incident with an updated severity (grafana.incident.updated.severity).

Request headers

The webhook includes the following headers:

To conform with the CloudEvents specification, the following headers are also included:

Webhook payload

The webhook payload is a JSON object with the following structure:

json Copy

{
  "version": "v1.0.0",
  "id": "webhook-out-6400aeb030670e95",
  "source": "/grafana/incident",
  "time": "2023-03-02T14:12:00Z",
  "event": "grafana.incident.updated.severity",
  "incident": {
    "incidentID": "5",
    "severityID": 1,
    "labels": [],
    "isDrill": false,
    "createdTime": "2023-02-27T15:06:25.243788Z",
    "modifiedTime": "2023-03-02T14:12:00.730047Z",
    "createdByUser": {
      "userID": "grafana-incident:user-63f8b6204887f793",
      "name": "admin",
      "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
    },
    "closedTime": "",
    "durationSeconds": 860734,
    "status": "active",
    "title": "55",
    "overviewURL": "/a/grafana-incident-app/incidents/5/55",
    "roles": [
      {
        "role": "investigator",
        "description": "Leads the investigation (has their full-time attention)",
        "maxPeople": 1,
        "mandatory": true,
        "important": true,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      },
      {
        "role": "observer",
        "description": "Watching the incident",
        "maxPeople": 0,
        "mandatory": false,
        "important": false,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      },
      {
        "role": "commander",
        "description": "Owns the incident (has their full-time attention)",
        "maxPeople": 1,
        "mandatory": true,
        "important": true,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      }
    ],
    "taskList": {
      "tasks": [
        {
          "taskID": "task-63ff1bf465deb38c",
          "immutable": false,
          "createdTime": "2023-03-01T09:33:40.759215968Z",
          "modifiedTime": "2023-03-01T09:33:55.286168752Z",
          "text": "111",
          "status": "done",
          "authorUser": {
            "userID": "grafana-incident:user-63f8b6204887f793",
            "name": "admin",
            "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
          },
          "assignedUser": {
            "userID": "grafana-incident:user-63f8b6204887f793",
            "name": "admin",
            "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
          }
        }
      ],
      "todoCount": 0,
      "doneCount": 1
    },
    "summary": "Resolved with no comment",
    "heroImagePath": "/api/hero-images/default_org/mC0B9YIgrBH3Hf9jvnKsvalHT90QtlODeiXDd2aLgRsflEZrRHGS5ld2KfIAbndglMIHiKSR04FPRPBpS34DA182rpeBxF9MSypELoU7nyOzXS09YenaCUtBZEzCi1xd/v1128/5.png",
    "incidentStart": "2023-02-20T15:06:26Z",
    "incidentEnd": ""
  }
}

For the complete API reference, see the OutgoingWebhookPayload documentation.