A guide to Grafana OnCall SMS and call routing

Many organizations use incident response setups that enable them to page on-call personnel via calling or sending a message to a phone number. In this guide, you will learn how to configure such a system by using Grafana OnCall. 

For practical purposes, we’ll pair it with Twilio, though the same basic workflow should be applicable to other platforms. We will start with a basic setup that uses a phone number in Twilio to both call and send SMS messages to a webhook integration in Grafana OnCall. Then, we will add a route and an escalation chain to demonstrate how we can send different alerts from SMS messages and voice calls to different escalation chains. Let’s dive in.

Prerequisites

In order to implement the steps outlined in this guide in your own system, you’ll need access to the following:

• Grafana Cloud. If you don’t already have one, you can sign up for a forever-free account today.
• Grafana OnCall, our on-call management system. To find out more, check out our docs on getting started with Grafana OnCall
• You’ll also need to be a Grafana OnCall user with administrator privileges and notification settings configured.
• Twilio. You can sign up for an account here.

Basic setup

To get started, we will configure a phone number in Twilio and an integration in Grafana OnCall that will allow us to receive alerts from an SMS message or phone call made/sent to a phone number. We will expand this setup as we go.

Grafana OnCall setup

Again, be sure to have a Grafana OnCall user account set up with administrator privileges and some notification settings configured for your user to test functionality along the way. If you need to set these up first refer to the Grafana OnCall documentation.

Set up your integration

An integration in Grafana OnCall is an endpoint that receives alerts and connects them to routes and escalation chains. Here we will use a generic webhook integration that can accept any payload. We can then customize how this payload is handled in Grafana OnCall. First let’s create the integration:

1. In Grafana, navigate to Alerts & IRM -> OnCall -> Integrations.
2. Press + New Integration.
3. Select “Webhook (Generic)” as the integration type.
4. Give the integration a name and description and assign it to a team (optional).
5. You will now be on the screen for your integration. Take note of the URL. This is the address we will use to send alerts to this integration from Twilio.

Set up the escalation chain

An escalation chain lets us create a sequence of steps to handle notifications and other actions once an alert has been received. In this case we will create the simplest escalation chain for testing consisting of directly notifying your user. Later this can be customized to interface with schedules or other actions. Let’s create the escalation chain:

1. In Grafana, navigate to Alerts & IRM -> OnCall -> Escalation chains.
2. Press + New Escalation chain.
3. Give the escalation chain a name and press Create Escalation Chain.
4. For the first step of the escalation, choose “Notify users” and choose your user as the recipient.
5. Later, you can customize the escalation chain to suit your needs.

Connect escalation chain and test

Before we set things up in Twilio, let’s connect the escalation chain to the integration and then test that the setup is working correctly.

1. Navigate back to the webhook integration we created earlier.
2. In the “Integrations” list, click the integration name you created previously to go to the integration details.
3. In the “Routes” section, expand the section for the “Default” route.
4. At the third step in the default route, use the dropdown to select the escalation chain you created.
5. Press the Send demo alert button in the top right of the screen.
6. You should receive a notification by the method you have configured for your user.
7. Resolve your demo alert.

Now let’s switch to Twilio to set up the other side of the integration. 

Twilio setup

We will set up a phone number and some Studio flows that Twilio will trigger when it receives a phone call or SMS message. These Studio flows will be used to send the information as an alert to Grafana OnCall.

Set up Studio flow

We will set up a Studio flow to handle SMS messages and handle phone calls. The goal of each is to capture alert information and send it to Grafana OnCall in a format that we can make use of.

1. On the “Develop” tab, navigate to Studio -> Flows. (If you do not see “Studio,” select Explore Products and select Studio under the “Developer Tools” section. Also, pin it for easier access later.)

2. On the Studio -> Flows page select Create new Flow.

3. Enter a flow name and press Next.

4. Select “Import from JSON”’ and press Next.

5. Paste the following JSON into the dialog and replace <YOUR_INTEGRATION_URL> at Lines 54 and 156 with the URL of the webhook integration we created earlier in Grafana OnCall. Then press Next.

   json Copy

   {
      "description": "Basic SMS and Call escalation",
      "states": [
        {
          "name": "Trigger",
          "type": "trigger",
          "transitions": [
            {
              "next": "send_alert_from_sms",
              "event": "incomingMessage"
            },
            {
              "next": "describe_alert_from_call",
              "event": "incomingCall"
            },
            {
              "event": "incomingConversationMessage"
            },
            {
              "event": "incomingRequest"
            },
            {
              "event": "incomingParent"
            }
          ],
          "properties": {
            "offset": {
              "x": 0,
              "y": 0
            }
          }
        },
        {
          "name": "send_alert_from_sms",
          "type": "make-http-request",
          "transitions": [
            {
              "next": "send_alert_from_sms_success",
              "event": "success"
            },
            {
              "next": "send_alert_from_sms_fail",
              "event": "failed"
            }
          ],
          "properties": {
            "offset": {
              "x": -180,
              "y": 250
            },
            "method": "POST",
            "content_type": "application/json;charset=utf-8",
            "body": "{\"from\":\"{{trigger.message.From}}\",\"message\":\"{{trigger.message.Body}}\"}",
            "url": "<YOUR_INTEGRATION_URL>"
          }
        },
        {
          "name": "send_alert_from_sms_success",
          "type": "send-message",
          "transitions": [
            {
              "event": "sent"
            },
            {
              "event": "failed"
            }
          ],
          "properties": {
            "offset": {
              "x": -410,
              "y": 590
            },
            "service": "{{trigger.message.InstanceSid}}",
            "channel": "{{trigger.message.ChannelSid}}",
            "from": "{{flow.channel.address}}",
            "message_type": "custom",
            "to": "{{contact.channel.address}}",
            "body": "Alert sent successfully"
          }
        },
        {
          "name": "send_alert_from_sms_fail",
          "type": "send-message",
          "transitions": [
            {
              "event": "sent"
            },
            {
              "event": "failed"
            }
          ],
          "properties": {
            "offset": {
              "x": -60,
              "y": 590
            },
            "service": "{{trigger.message.InstanceSid}}",
            "channel": "{{trigger.message.ChannelSid}}",
            "from": "{{flow.channel.address}}",
            "message_type": "custom",
            "to": "{{contact.channel.address}}",
            "body": "Failed to send alert: Status({{widgets.send_escalation.status_code}})"
          }
        },
        {
          "name": "describe_alert_from_call",
          "type": "gather-input-on-call",
          "transitions": [
            {
              "event": "keypress"
            },
            {
              "next": "send_alert_from_call",
              "event": "speech"
            },
            {
              "event": "timeout"
            }
          ],
          "properties": {
            "speech_timeout": "auto",
            "offset": {
              "x": 350,
              "y": 240
            },
            "loop": 1,
            "finish_on_key": "#",
            "say": "Describe the alert to send. Press pound when finished.",
            "stop_gather": true,
            "gather_language": "en",
            "profanity_filter": "true",
            "timeout": 60
          }
        },
        {
          "name": "send_alert_from_call",
          "type": "make-http-request",
          "transitions": [
            {
              "next": "send_alert_from_call_success",
              "event": "success"
            },
            {
              "next": "send_alert_from_call_fail",
              "event": "failed"
            }
          ],
          "properties": {
            "offset": {
              "x": 360,
              "y": 590
            },
            "method": "POST",
            "content_type": "application/json;charset=utf-8",
            "body": "{\"from\":\"{{trigger.call.From}}\", \"message\":\"{{widgets.describe_alert_from_call.SpeechResult}} \"}",
            "url": "<YOUR_INTEGRATION_URL>"
          }
        },
        {
          "name": "send_alert_from_call_success",
          "type": "say-play",
          "transitions": [
            {
              "event": "audioComplete"
            }
          ],
          "properties": {
            "offset": {
              "x": 90,
              "y": 900
            },
            "loop": 1,
            "say": "Alert sent successfully"
          }
        },
        {
          "name": "send_alert_from_call_fail",
          "type": "say-play",
          "transitions": [
            {
              "event": "audioComplete"
            }
          ],
          "properties": {
            "offset": {
              "x": 520,
              "y": 900
            },
            "loop": 1,
            "say": "Failed to send alert: Status   ({{widgets.send_alert_from_call.status_code}})"
          }
        }
      ],
      "initial_state": "Trigger",
      "flags": {
        "allow_concurrent_calls": true
      }
     }

6. After importing you should see something like this:

This flow has two paths. The first path accepts the SMS message and forwards its contents to Grafana OnCall on the webhook integration we created. The content of the message will be passed as the message field along with the phone number the message was sent from. When the request is made to Grafana OnCall, whether it succeeds or fails, it will be communicated back to the sender. 

The voice call path is similar — the caller describes the alert and it will be converted to text and sent to the integration in Grafana OnCall.

7. Now press Publish to make the flow available to be connected to a phone number.

Buying a phone number

In Twilio’s console we will need to set up a phone number that we will use to receive calls and messages that we’ll forward to Grafana OnCall.

1. On the “Develop” tab navigate to # Phone Numbers -> Manage -> Buy a number. (If you don’t see “#Phone Numbers” select Explore Products and select Phone Numbers under “Super Network” section; also, pin it for easier access later.)
2. On the “Buy a Number” screen, search for a phone number in the country code you want, select one and choose Buy.
3. Depending on the country or region, you may need to fill out some additional information (address, contact person, etc.).
4. Once purchased, the number will be available in “Active Numbers.”

Configuring a phone number

Here we will connect our phone number with the flow we created.

1. On the “Develop” tab, navigate to # Phone Numbers -> Manage -> Active Numbers.
2. Select the purchased phone number from the list.
3. In the “Voice Configuration” section, use the “A call comes in” dropdown and select “Studio Flow.” Set the flow to the flow we created.
4. In the “Messaging Configuration” section, use the “A message comes in” dropdown and select “Studio Flow.” Set the flow to the flow we created.
5. Press Save configuration.

Testing and troubleshooting

First we will test our setup by sending an alert from an SMS message.

1. Using your mobile phone, send an SMS message to the phone number we purchased in Twilio.
2. You should get a response message of “Alert sent successfully.”
3. Shortly after that you should receive a notification from Grafana OnCall by whichever method you configured for your user.

• If you did not receive the “Alert sent successfully” message, you should have a message with a status code. Check to make sure you have the correct URL configured in your flow.
• Check that “A message comes in” is configured correctly for the phone number.
• In Studio -> Flows in the right side menu for your flow, you can view “Execution Logs to troubleshoot.”
• If you haven’t been notified in Grafana OnCall, check the alert groups to see if one was created. From the alert group you can view the escalation log to see what happened.

Next, test sending an alert by calling.

1. Using your phone to call the number we purchased in Twilio
2. You should hear instructions to describe your alert and press #.
3. Next you should hear a message that the alert was sent successfully.
4. Shortly after that you should receive a notification from Grafana OnCall by whichever method you configured for your user

For troubleshooting, follow the same steps outlined in the previous section.

Basic setup complete

We now have a basic setup through which we can send an alert to Grafana OnCall by SMS message or voice call. Next, we can build on this to add the ability to route alerts to different escalation chains.

Adding routes

In this section we will set up a list of options so we can select which route we want to send an alert to and route it appropriately in Grafana OnCall. To accomplish this in Grafana OnCall we will set up another route and escalation chain and attach them both to the integration. Then we will expand the Studio flow in Twilio to present the option to the caller. This setup can be easily expanded upon to handle more routes.

Grafana OnCall setup

Adding another escalation chain

1. In Grafana, navigate to Alerts & IRM -> OnCall -> Escalation chains.
2. Press + New Escalation chain.
3. Give the escalation chain a name and press Create Escalation Chain.
4. For the first step of the escalation, choose “Notify users” and choose your user as the recipient. You can choose “Important” instead of “Default” so that it is different from the previous one.
5. Later you can customize the escalation chain to suit your needs.

Adding a route

A route is part of an integration. Routes are sequentially matched rules defined as Jinja2. The first rule that evaluates to true is the selected route for an incoming alert payload. We will leave the existing escalation chain we created as our default route and add our newly created escalation chain to a new route.

1. Navigate back to the webhook integration we created.
2. Select the integration by clicking on its name.
3. Press the Add Route button.
4. At the first step of the route, press the Edit template button to open the template editor.
5. Enter {{ "abc" in payload.target.lower()}} as the template and press Save. This template means that if the alert payload sent from Twilio contains the target field and the value is “abc,” this route will be selected. Later this can be customized to better represent the logic of how you want to route your alerts (team, service, region, etc.).
6. At the third step of the route, choose the escalation chain you created in the previous step from the dropdown.

Now we have what we need so Grafana OnCall can send an alert to different escalation chains based on the content of the payload from Twilio.

Twilio setup

Adding another Studio flow

Here we need to set up a more complex Studio flow than the one we had in the “Basic setup” section.

1. In the “Develop” tab, navigate to Studio -> Flows. (If you do not see Studio select Explore Products and select Studio under “Developer Tools” section; Also, pin it for easier access later.)
2. On the Studio -> Flows page select Create new Flow.
3. Enter a flow name and press Next.
4. Select “Import from JSON” and press Next.
5. Paste the following JSON into the dialog and replace <YOUR_INTEGRATION_URL> at Lines 54 and 156 with the URL of the webhook integration we created earlier in Grafana OnCall. Then press Next.

json Copy

{
 "description": "Added Routes SMS and Call escalation",
 "states": [
   {
     "name": "Trigger",
     "type": "trigger",
     "transitions": [
       {
         "next": "sms_select_target",
         "event": "incomingMessage"
       },
       {
         "next": "call_select_target",
         "event": "incomingCall"
       },
       {
         "event": "incomingConversationMessage"
       },
       {
         "event": "incomingRequest"
       },
       {
         "event": "incomingParent"
       }
     ],
     "properties": {
       "offset": {
         "x": 80,
         "y": -200
       }
     }
   },
   {
     "name": "send_alert_from_sms",
     "type": "make-http-request",
     "transitions": [
       {
         "next": "send_alert_from_sms_success",
         "event": "success"
       },
       {
         "next": "send_alert_from_sms_fail",
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": -370,
         "y": 500
       },
       "method": "POST",
       "content_type": "application/json;charset=utf-8",
       "body": "{\"from\":\"{{trigger.message.From}}\",\"message\":\"{{trigger.message.Body}}\",\"target\":\"{{widgets.sms_select_target.inbound.Body}}\"}",
       "url": "<YOUR_INTEGRATION_URL>"
     }
   },
   {
     "name": "send_alert_from_sms_success",
     "type": "send-message",
     "transitions": [
       {
         "event": "sent"
       },
       {
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": -700,
         "y": 780
       },
       "service": "{{trigger.message.InstanceSid}}",
       "channel": "{{trigger.message.ChannelSid}}",
       "from": "{{flow.channel.address}}",
       "message_type": "custom",
       "to": "{{contact.channel.address}}",
       "body": "Alert sent successfully"
     }
   },
   {
     "name": "send_alert_from_sms_fail",
     "type": "send-message",
     "transitions": [
       {
         "event": "sent"
       },
       {
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": -340,
         "y": 780
       },
       "service": "{{trigger.message.InstanceSid}}",
       "channel": "{{trigger.message.ChannelSid}}",
       "from": "{{flow.channel.address}}",
       "message_type": "custom",
       "to": "{{contact.channel.address}}",
       "body": "Failed to send alert: Status({{widgets.send_escalation.status_code}})"
     }
   },
   {
     "name": "describe_alert_from_call",
     "type": "gather-input-on-call",
     "transitions": [
       {
         "event": "keypress"
       },
       {
         "next": "send_alert_from_call",
         "event": "speech"
       },
       {
         "event": "timeout"
       }
     ],
     "properties": {
       "speech_timeout": "auto",
       "offset": {
         "x": 350,
         "y": 310
       },
       "loop": 1,
       "finish_on_key": "#",
       "say": "Describe the alert to send. Press pound when finished.",
       "stop_gather": true,
       "gather_language": "en",
       "profanity_filter": "true",
       "timeout": 60
     }
   },
   {
     "name": "send_alert_from_call",
     "type": "make-http-request",
     "transitions": [
       {
         "next": "send_alert_from_call_success",
         "event": "success"
       },
       {
         "next": "send_alert_from_call_fail",
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": 350,
         "y": 580
       },
       "method": "POST",
       "content_type": "application/json;charset=utf-8",
       "body": "{\"from\":\"{{trigger.call.From}}\", \"message\":\"{{widgets.describe_alert_from_call.SpeechResult}} \",\"target\":\"{{widgets.call_set_target.target}}\"}",
       "url": "<YOUR_INTEGRATION_URL>"
     }
   },
   {
     "name": "send_alert_from_call_success",
     "type": "say-play",
     "transitions": [
       {
         "event": "audioComplete"
       }
     ],
     "properties": {
       "offset": {
         "x": 200,
         "y": 950
       },
       "loop": 1,
       "say": "Alert sent successfully"
     }
   },
   {
     "name": "send_alert_from_call_fail",
     "type": "say-play",
     "transitions": [
       {
         "event": "audioComplete"
       }
     ],
     "properties": {
       "offset": {
         "x": 630,
         "y": 950
       },
       "loop": 1,
       "say": "Failed to send alert: Status   ({{widgets.send_alert_from_call.status_code}})"
     }
   },
   {
     "name": "sms_select_target",
     "type": "send-and-wait-for-reply",
     "transitions": [
       {
         "next": "sms_validate_target",
         "event": "incomingMessage"
       },
       {
         "next": "sms_select_target_timeout",
         "event": "timeout"
       },
       {
         "event": "deliveryFailure"
       }
     ],
     "properties": {
       "offset": {
         "x": -330,
         "y": -50
       },
       "service": "{{trigger.message.InstanceSid}}",
       "channel": "{{trigger.message.ChannelSid}}",
       "from": "{{flow.channel.address}}",
       "message_type": "custom",
       "body": "Which target do you want to send the alert to?\nabc \ndefault",
       "timeout": "300"
     }
   },
   {
     "name": "sms_select_target_timeout",
     "type": "send-message",
     "transitions": [
       {
         "event": "sent"
       },
       {
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": -80,
         "y": 210
       },
       "service": "{{trigger.message.InstanceSid}}",
       "channel": "{{trigger.message.ChannelSid}}",
       "from": "{{flow.channel.address}}",
       "message_type": "custom",
       "to": "{{contact.channel.address}}",
       "body": "Target select timed out, send the alert again to start over."
     }
   },
   {
     "name": "sms_validate_target",
     "type": "split-based-on",
     "transitions": [
       {
         "next": "sms_validate_target_fail",
         "event": "noMatch"
       },
       {
         "next": "send_alert_from_sms",
         "event": "match",
         "conditions": [
           {
             "friendly_name": "If value equal_to abc",
             "arguments": [
               "{{widgets.sms_select_target.inbound.Body}}"
             ],
             "type": "matches_any_of",
             "value": "abc,default"
           }
         ]
       }
     ],
     "properties": {
       "input": "{{widgets.sms_select_target.inbound.Body}}",
       "offset": {
         "x": -590,
         "y": 210
       }
     }
   },
   {
     "name": "sms_validate_target_fail",
     "type": "send-message",
     "transitions": [
       {
         "event": "sent"
       },
       {
         "event": "failed"
       }
     ],
     "properties": {
       "offset": {
         "x": -700,
         "y": 500
       },
       "service": "{{trigger.message.InstanceSid}}",
       "channel": "{{trigger.message.ChannelSid}}",
       "from": "{{flow.channel.address}}",
       "message_type": "custom",
       "to": "{{contact.channel.address}}",
       "body": "{{widgets.sms_select_target.inbound.Body}} is not a valid target."
     }
   },
   {
     "name": "call_select_target",
     "type": "gather-input-on-call",
     "transitions": [
       {
         "next": "call_select_validate",
         "event": "keypress"
       },
       {
         "event": "speech"
       },
       {
         "event": "timeout"
       }
     ],
     "properties": {
       "number_of_digits": 1,
       "speech_timeout": "auto",
       "offset": {
         "x": 350,
         "y": 50
       },
       "loop": 1,
       "finish_on_key": "#",
       "say": "Which target do you want to send to? Press 1 for ABC. \nPress 2 for default.",
       "stop_gather": true,
       "gather_language": "en",
       "profanity_filter": "true",
       "timeout": 5
     }
   },
   {
     "name": "call_select_validate",
     "type": "split-based-on",
     "transitions": [
       {
         "next": "call_select_target",
         "event": "noMatch"
       },
       {
         "next": "call_set_target",
         "event": "match",
         "conditions": [
           {
             "friendly_name": "If value matches_any_of 1,2",
             "arguments": [
               "{{widgets.call_select_target.Digits}}"
             ],
             "type": "matches_any_of",
             "value": "1,2"
           }
         ]
       }
     ],
     "properties": {
       "input": "{{widgets.call_select_target.Digits}}",
       "offset": {
         "x": 760,
         "y": 50
       }
     }
   },
   {
     "name": "call_set_target",
     "type": "set-variables",
     "transitions": [
       {
         "next": "describe_alert_from_call",
         "event": "next"
       }
     ],
     "properties": {
       "variables": [
         {
           "value": "{% if widgets.call_select_target.Digits == \"1\" %}abc{% elsif widgets.call_select_target.Digits == \"2\" %}default{% endif %}",
           "key": "target"
         }
       ],
       "offset": {
         "x": 760,
         "y": 300
       }
     }
   }
 ],
 "initial_state": "Trigger",
 "flags": {
   "allow_concurrent_calls": true
 }
}

6. After importing you should see something like this:

Here we have added steps that prompt the user to decide which target they want to send the alert to as well as validation steps to ensure the values are correct.

Reconfiguring the phone number

We can now change which flow our phone is configured to use.

1. On the “Develop” tab, navigate to # Phone Numbers -> Manage -> Active Numbers.
2. Select the purchased phone number from the list.
3. In the “Voice Configuration” section, use the “A call comes in” dropdown and select “Studio Flow.” Set the flow to the new flow we created.
4. In the “Messaging Configuration” section, use the “A message comes in” dropdown and select “Studio Flow.” Set the flow to the new flow we created.
5. Press Save configuration,

Testing and troubleshooting

Testing and troubleshooting follows the same process as before. Exercise the SMS and voice call paths to test that you receive notifications from Grafana OnCall.

Next steps

Now that you’ve got everything connected, you can further edit and customize the Grafana OnCall routes and escalation chains. And by using the graphical editor for Studio flow, you can make changes to how SMS and voice calls are handled. Using these tools you can set up Grafana OnCall to send notifications to users in a way that matches your team structure using SMS and voice calls as a source of alerts.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!