Documentation Grafana Cloud Alerts and IRM IRM Configure Integrations Webhooks Incoming webhooks Incident webhooks

Configure Incident incoming webhooks

Incident incoming webhooks allow you to trigger incidents in Grafana IRM from any third-party system.

About Incident incoming webhooks

Incoming webhooks act as a bridge between external systems and Grafana IRM’s incident management capabilities. When a webhook is triggered:

• An incident is automatically created in Grafana IRM
• Details from the webhook request are attached to the incident
• Incident workflows and notifications are initiated

Set up a webhook

Create an Incident incoming webhook integration

1. In Grafana IRM, go to the Integrations tab
2. Click Apps and select Incoming Webhook
3. Click Install integration
4. Copy the Token and URL provided on the integration page for configuration in your external tool

Authenticate webhook requests

There are two methods to authorize webhook requests:

Configure webhook parameters

Customize incidents by adding parameters to the webhook URL:

Configure the third-party system

1. In your external system, configure a webhook to send a POST request to:
   Copy

   api/v1/incoming-webhooks/grafana.incident.create

2. Add your specific URL parameters
3. Set the Authorization header to Bearer {your_token}
4. Configure the request body as needed (JSON format recommended)

Example webhook configuration

Copy

POST .../api/v1/incoming-webhooks/grafana.incident.create?drill=true&title=json(message.shortMessage)&severity=minor&labels=autogenerated
Content-Type: application/json
Authorization: Bearer token_goes_here

{
  "message": {
    "shortMessage": "A short description might appear here"
  }
}

Extract data from JSON payloads

For webhook requests with JSON bodies, use the json() function in URL parameters to extract specific values:

Syntax

Copy

parameter=json(path.to.field)

Example

For a JSON body:

json Copy

Content-Type: "application/json"
{
  "reportID": "abc123",
  "report": {
    "title": "Unable to access public website"
  }
}

Add this parameter to extract the title:

Copy

title=json(report.title)

Copy

.../api/v1/incoming-webhooks/grafana.incident.create?title=json(report.title)

Note: JSON payloads must be smaller than 1MB to be processed correctly.

Adding request metadata

You can add context to webhook requests using these HTTP headers:

• User-Agent
• Origin
• Referer
• Link

These values are included in the incident details to help identify the webhook source.

Test your webhook

The integration details provides you with an example curl command to test the endpoint.

Using cURL

bash Copy

curl ".../api/v1/incoming-webhook/grafana.incident.create?drill=true&title=json(report.title)" \
  --request POST \
  --header 'Authorization: Bearer your_token_here' \
  --data '{"report":{"title":"testing the new incoming webhooks integration"}}'

Always test with drill=true before implementing in production to avoid creating unwanted incidents.

Response format

The webhook responds with a 200 OK status code and a JSON body:

json Copy

{
  "incident": {
    "incidentID": "incident-1",
    "title": "title of the incident"
    // other incident fields
  },
  "processingErrors": ["error messages if any"]
}

Note: The handler prioritizes creating incidents even if there are processing errors. Check the processingErrors field during testing and resolve all issues before using in production.

Manage webhook security

Protect your webhook token. If compromised, anyone could create incidents in your system. If security is compromised, reinstall the integration to generate a new token.

Disabling or resetting webhooks

To invalidate an existing webhook token:

1. In Grafana IRM, go to the Integrations tab
2. Click Apps and select Incoming Webhook
3. Click Uninstall integration to disable all incoming webhook requests
4. Click Install integration to generate a new token

Reset your webhook token if:

• The token has been compromised
• You notice suspicious incident creation
• You’re no longer using the webhook
• You need to update third-party system configurations

Limitations

• Maximum webhook body size: 1MB
• Request rate limit: 1 incident per minute (returns 429 Too Many Requests if exceeded)