Using Grafana to Monitor EMS Ambulance Service Operations

The Emergency Services team at Trapeze Group provides 24/7/365 support for ambulances in Australia. Each fleet can contain as many as 1,000 vehicles, with more than 60 telemetry channels and 120 million messages going in and out to paramedics every day.

The information ranges from “the 911 call that comes into the control center to dispatching ambulances; to monitoring them on scene, looking at vehicle performance, making sure they’re on scene quick enough, getting them to the right location, getting them out of scene, getting them into hospital,” James Wetherall, Group Manager, Emergency Services, said during his talk at GrafanaCon L.A.

These operations can be, quite literally, a matter of life or death, so observability is critical. To make the information more accessible, Wetherall said, “we’re trying to ingrain Grafana in the customers that we’re working with, get it exposed, and get everything visual.”

Trapeze’s data environment is based around SQL Server and Postgres. Because Trapeze’s customers operate in large geographical areas where there isn’t always high-speed cellular coverage – “I’m talking like 9KB a second,” he said – Wetherall’s team has in the past year done some work with the spatial side of Postgres, building custom protocols.

The data covers location, connectivity, safety, communication, awareness, and integration. There’s information about where the equipment is and where the people are, and communication with the hospitals. There are many integrated data sources around vehicle telematics and the internet of lifesaving things, such as defibrillators and specialist equipment, as well as those around smart cities, like traffic lights. “We’re monitoring all this in real time, making sure they have connectivity as we do things like run on multiple networks,” said Wetherall. “If one service is out, we need to use others and keep retrying.”

The Grafana Journey

When Trapeze started building out an observability solution for its customers three years ago, it was fully bespoke. “We’ve tried a whole bunch of stuff: different frameworks around Angular and React, some of the web bootstraps,” Wetherall said.

The team soon found that it was impossible to get everything they wanted quickly. “The software development to get something out in the hands of people was just slowing us down,” he said. “Adding multiple data sources was just impossible; it was like building something new every time. Most things weren’t operating in real time… The whole technicality of stitching these things hindered our progress. We really wanted to deliver in minutes. We wanted to be out of rapid prototype in front of users in real time. We wanted it to be synced to those data sources, remove dependency on developers.”

Wetherall’s goal was to focus development on building plugins and enhancing the solution, not architecting something from the ground up. “And we really want it to be an everyday tool,” he said. “Everyone in these organizations should have access, they should be able to see what they need to see quickly, and there shouldn’t be a steep learning curve.”

Grafana enabled all of this. “The key takeaway for me was to get it out of the back room and into our customers’ hands,” he said. “That’s just been a game-changer for us.”

During the talk, Wetherall showed a number of Trapeze’s key Grafana dashboards.

This dashboard monitors messages:

Using Grafana’s built-in maps to track vehicles, this dashboard shows information like where they are, what status they are in, how fast they’re going, and what the current response time is:

Other dashboards even drill down to things like the batteries in the vehicles, juxtaposed with what priority jobs they’re on and how long they’ve been responding.

Wetherall’s team keeps finding new and better ways to present all of the information that emergency services need. “The current world map plugin will allow us to plot points and scale points,” he said. In their first foray into customizing plugins, “we’ve been doing things like stitching together points to make lines, overlaying things like critical points within that response to an incident, when the crew got the job, when they arrived at scene. So it’s an easy way to drill back into jobs, go back in history, and replay how long the job took.”

Other dashboards hooked into the vehicle telemetry – battery life, lamp status, oil pressure, temperature – are integrated into the fleet management system, so vehicles can get repaired when necessary. Field techs commissioning vehicles can get a quick view of whether everything is working properly. “We can do some sort of predictive modeling on this to see degradation of batteries and put in place things like optimal replacement programs,” said Wetherall.

Another feature the team is working on is overlaying information and using the annotation tool to allow people to analyze the data, flag an annotation, pull it out, and put in a request to a workshop to service that vehicle.

They’re also looking at calendar control to overlay information about how many jobs a vehicle has responded to in a day, to help with scheduling.

Looking Ahead

“When we first started on the Grafana journey, we had a lot of pushback from our development team,” Wetherall admitted. “They kept coming up with reasons why we couldn’t do it that way, and we needed to keep building the Angular dashboards. But we’ve been able to deal with all those requirements from our development team and our users, and I think we’re sort of at the stage where we’ve got a couple use cases that would really enhance it.”

The world map is a key part of those enhancements, and so are horizontal charts (which are coming soon). “And we’re really interested in pulling in things like live data sources,” said Wetherall. “At the moment we are pulling databases, and I think there’s some real improvements in that space.”

“We’ll look at a whole bunch of things to really let us ingest a whole lot more data and analyze it on the fly, aggregate it up, throw it in a data store,” said Wetherall. Apache Spark, Prometheus, and InfluxDB are technologies of interest as they focus on developing the ability to store data and retrieve it later for analysis, and building up Trapeze’s message queues, anomaly detection, and big data stores.

“We sort of feel like Grafana gives us that visual ability,” he said, “so you can start conceptualizing it now, and you can start really building these use cases.”

For more from GrafanaCon 2019, check out all the talks on YouTube.