Zabbix integration for Grafana OnCall

Zabbix is an open-source monitoring software tool for diverse IT components, including networks, servers, virtual machines, and cloud services. Zabbix provides monitoring for metrics such as network utilization, CPU load, and disk space consumption.

Configure Zabbix integration for Grafana OnCall

This integration is available for Grafana Cloud OnCall. You must have an Admin role to create integrations in Grafana OnCall.

In the Integrations tab, click + New integration to receive alerts.
Select Zabbix from the list of available integrations
Follow the instructions in the How to connect window to get your unique integration URL and review next steps.

Configure the Zabbix server

Deploy a Zabbix playground if you don’t have one set up:

 docker run --name zabbix-appliance -t \
      -p 10051:10051 \
      -p 80:80 \
      -d zabbix/zabbix-appliance:latest

Establish an ssh connection to a Zabbix server.
bash
```
docker exec -it zabbix-appliance bash
```
Place the grafana_oncall.sh script in the AlertScriptsPath directory specified within the Zabbix server configuration file (zabbix_server.conf).
bash
```
grep AlertScriptsPath /etc/zabbix/zabbix_server.conf
```
Note: The script must be executable by the user running the zabbix_server binary (usually “zabbix”) on the Zabbix server. For example, chmod +x grafana_oncall.sh
bash
```
ls -lh /usr/lib/zabbix/alertscripts/grafana_oncall.sh
-rw-r--r--    1 root     root        1.5K Jun  6 07:52 /usr/lib/zabbix/alertscripts/grafana_oncall.sh
```

Configure Zabbix alerts

Within Zabbix web interface, do the following:

In a browser, open localhost:80.
Navigate to Adminitstration > Media Types > Create Media Type.
Create a Media Type with the following fields.
- Name: Grafana OnCall
- Type: script
- Script parameters:
  - {ALERT.SENDTO}
  - {ALERT.SUBJECT}
  - {ALERT.MESSAGE}

Set the {ALERT.SEND_TO} value

To send alerts to Grafana OnCall, the {ALERT.SEND_TO} value must be set in the user media configuration.

In the web UI, navigate to Administration > Users and open the user properties form.
In the Media tab, click Add and copy the link from Grafana OnCall in the Send to field.
Click Test in the last column to send a test alert to Grafana OnCall.
Specify Send to OnCall using the unique integration URL from the above step in the testing window that opens.
Create a test message with a body and optional subject and click Test.

Grouping and auto-resolve of Zabbix notifications

Grafana OnCall provides grouping and auto-resolve of Zabbix notifications. Use the following procedure to configure grouping and auto-resolve.

Provide a parameter as an identifier for group differentiation to Grafana OnCall.
Append that variable to the subject of the action as ONCALL_GROUP: ID, where ID is any of the Zabbix macros. For example, {EVENT.ID}. The Grafana OnCall script grafana_oncall.sh extracts this event and passes the alert_uid to Grafana OnCall.
To enable auto-resolve within Grafana Oncall, the “Resolved” keyword is required in the Default subject field in Recovered operations.

grafana_oncall.sh script

#!/bin/bash
# This is the modification of original ericos's shell script.

# Get the url ($1), subject ($2), and message ($3)
url="$1"
subject="${2//$'\r\n'/'\n'}"
message="${3//$'\r\n'/'\n'}"

# Alert state depending on the subject indicating whether it is a trigger going in to problem state or recovering
recoversub='^RECOVER(Y|ED)?$|^OK$|^Resolved.*'

if [[ "$subject" =~ $recoversub ]]; then
    state='ok'
else
    state='alerting'
fi

payload='{
    "title": "'${subject}'",
    "state": "'${state}'",
    "message": "'${message}'"
}'

# Alert group identifier from the subject of action. Grouping will not work without ONCALL_GROUP in the action subject
regex='ONCALL_GROUP: ([a-zA-Z0-9_\"]*)'
if [[ "$subject" =~ $regex ]]; then
    alert_uid=${BASH_REMATCH[1]}
    payload='{
        "alert_uid": "'${alert_uid}'",
        "title": "'${subject}'",
        "state": "'${state}'",
        "message": "'${message}'"
    }'
fi

return=$(curl $url -d "${payload}" -H "Content-Type: application/json" -X POST)