How runners and cyclists can sync Garmin devices to Grafana Cloud to analyze fitness metrics

Hi, I’m Emil, a Solutions Engineer here at Grafana Labs. If we’ve met already (this wonderful world of IT is small), you know that I have been doing enterprise software and services for quite a while. What’s funny is that somehow the insights and value of log file analytics escaped me until recently. 

 . . . Well, that’s not completely true: Around 2004, I seriously thought about developing a system that would allow me to “tail” log files on different systems in the browser. I guess if I’d done it, I would be a rich man now. But would I be happier? Not necessarily, because what really makes me happy is running marathons (not exclusively, though).

Thanks to that passion, I recently had another big idea: Why not combine log file analytics and running? My end goal wasn’t to improve my running pace or anything. I really just wanted to try this out for the fun of it — and because it’s good to have some real IoT experience in this ever-more-connected world.

The capabilities Grafana Loki provides are perfect for this project. In case you don’t know Loki, it’s the open source, horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus that powers Grafana Labs’ log analytics products Grafana Enterprise Logs and Grafana Cloud Logs. 

The idea for the Loki/running mashup was born in a discussion with my dear colleague and running mate Stefan. Stefan, please introduce yourself:

Hello, my name is Stefan, and like Emil, I’m a Solutions Engineer at Grafana Labs. I definitely didn’t think of tailing logs from multiple systems in the browser back in 2004 — I’m too young for that! — but I will admit that I was “tailing” and “grepping” logs in a SSH command line session for way too long before I learned about Grafana Loki. And it’s true, I also love running, though mostly on the nice trails in and around the Austrian Alps.

In this post, we’re going to walk you through our project and the process of collecting metrics (our heart rates) from an IoT sensor (our sport watches), ingesting those metrics in JSON format, and visualizing them on a Grafana dashboard.

The basics

The data generators we used for this were Garmin devices that collect all kinds of human body telemetry as we run. (Bike computers would work, too.) However, the wider use case is not limited to these devices and this kind of telemetry. Rather, this architecture serves as an IoT example with a much broader area of application: simple devices with no direct internet connection talking to edge devices with more compute power and internet connection.

In our example, the simple devices are watches and bike computers, and the edge devices are smartphones. In the real world, they might be sensors in chemical plants, factory floors, or kitchen appliances that monitor, control, and alert on critical processes — or just make the lives of the operators easier.

Setting up data collection

Since we wanted to be able to do analytics on the data the devices are collecting, we decided to consolidate the data in the cloud. We also needed a scalable — i.e. stateless — infrastructure listening for potentially massive amounts of data from the edge devices. Then, we needed a scalable layer that would persist the data and allow us to query it. 

The first thing we did was build a device application using Garmin’s CONNECT IQ software development kit that was running on our smartphones. It reads information from our device’s sensors and sends it to our Grafana Agent, which will persist it in Grafana Cloud.

Normally, the Grafana Agent scrapes metrics, listens for traces, and reads log files, but what makes it extra handy is that we were able to configure it to also listen for incoming JSON payloads. With that done, we had a stateless listener. 

Put the agent into a container and throw it at Google Cloud Run, and we had a stateless and scalable layer. Finally, we configured the Grafana Agent to send the data to Grafana Cloud Logs. 

(If you don’t have Grafana Cloud yet, sign up for free now!)

Putting it all together

Here’s what you need to get started:

• A Google Cloud Account
• Google Cloud SDK installed and authenticated
• Google Cloud Run APIs enabled
• A Google Cloud Run service created. Name: $YOUR_GCP_CLOUDRUN_SERVICENAME
• Four environment variables added to the Cloud Run service called API_KEY, URL, PASSWORD and USERNAME
• Visual Studio Code
• Garmin Connect IQ SDK installed and setup (incl. developer key) 

To begin, build a docker container containing your Grafana Agent and configuration:

git clone <https://github.com/digitalemil/LokiRuns.git>

Change to the LokiRuns folder and build the container.

Dockerfile:

Copy

FROM grafana/agent:v0.25.1
 
RUN apt-get update -y
RUN apt-get upgrade -y
 
COPY lokiruns-agent-config.yaml /opt
 
ENTRYPOINT /bin/agent -config.expand-env 
-enable-features integrations-next 
--config.file=/opt/lokiruns-agent-config.yaml

Modify the env.sh script to reflect your settings and execute it. It will ask you for an API_KEY, which will be used to authenticate a device by the agent. The API_KEY will be written to strings.xml file in your Garmin resources.

Copy

#!/bin/sh

export VERSION=0.0.1
export PLATFORM=linux/amd64
export GCP_REPO=eu.gcr.io
export GCP_PROJECTID=$YOUR_GCP_PROJECTID

export CLOUDRUN_ENDPOINT=https://thegym.theblackapp.de/collect
read -p "Please insert the API_KEY shared by Grafana Agent 
and device: " API_KEY 
export API_KEY

sed "s@{CLOUDRUN_ENDPOINT}@$CLOUDRUN_ENDPOINT@g; 
s@{API_KEY}@$API_KEY@g;" 
garmin-connectiq/resources/strings/strings.xml.template 
>garmin-connectiq/resources/strings/strings.xml

Next, set the environment variables of your Cloud Run service. For this, you need the details from your Grafana Cloud Log service. Also create an API_KEY, which we will refer to as PASSWORD.

Execute make.sh to build and push your container:

emil@Emils-MacBook-Air LokiRuns % ./make.sh

Update your Cloud Run service through the Google Cloud console or the gcloud command. It should look similar to this:

Copy

gcloud run deploy $YOUR_GCP_CLOUDRUN_SERVICENAME \
--image=eu.gcr.io/$YOUR_GCP_PROJECTID/lokiruns-v0.0.1@$H
ASHASH_OF_YOURCONTAINER \
--region=europe-west4 \
--project=$GCP_PROJECTID \
 && gcloud run services update-traffic 
$YOUR_GCP_CLOUDRUN_SERVICENAME --to-latest

Your devices need to talk to the edge device, and the edge device needs to talk to your listener service. This is what we use Garmin Connect IQ for. Applications for Garmin devices are written in Monkey C, and in this case you only need to read sensor data then remotely execute an HTTPS POST on the edge device (smartphone) with the JSON data provided via Bluetooth LE. 

Find an excerpt from the code below. The code gets executed on the device (e.g. watch) and reads data from the sensors. It then creates a WebRequest object including the data, which is sent to the connected phone via Bluetooth and gets executed there. The result of the HTTPS call will be returned via Bluetooth to the device.  

Copy

function onSensor(sensorInfo) {
       hr= sensorInfo.heartRate;
       speed= sensorInfo.speed;
       altitude= sensorInfo.altitude;
       heading= sensorInfo.heading;
       …
       var line = {
       "message" => logline,
       "level" => "info",
       "context" => labels ,
       "timestamp"=> dateString
       };
       
       var logs = {
         "logs" => [line]
       };
       var key=Ui.loadResource(Rez.Strings.API_KEY);
 
       var options = { 
          :method => Communications.HTTP_REQUEST_METHOD_POST,     
          :headers => { "Content-Type" =>   Communications.REQUEST_CONTENT_TYPE_JSON, "x-api-key" => key },                                                        
          :responseType => Communications.HTTP_RESPONSE_CONTENT_TYPE_URL_ENCODED
      };
      var responseCallback = method(:onReceive);     
      Communications.makeWebRequest(url, logs, options, method(:onReceive));
…
function onSensor(sensorInfo) {
       hr= sensorInfo.heartRate;
       speed= sensorInfo.speed;
       altitude= sensorInfo.altitude;
       heading= sensorInfo.heading;
       …
       var line = {
       "message" => logline,
       "level" => "info",
       "context" => labels ,
       "timestamp"=> dateString
       };
       
       var logs = {
         "logs" => [line]
       };
       var key=Ui.loadResource(Rez.Strings.API_KEY);
 
       var options = { 
          :method => Communications.HTTP_REQUEST_METHOD_POST,     
          :headers => { "Content-Type" =>   Communications.REQUEST_CONTENT_TYPE_JSON, "x-api-key" => key },                                                        
          :responseType => Communications.HTTP_RESPONSE_CONTENT_TYPE_URL_ENCODED
      };
      var responseCallback = method(:onReceive);     
      Communications.makeWebRequest(url, logs, options, method(:onReceive));
…

Note: If you want to test the above…

Build the executable for your device from the source — otherwise, we found no good way for data input (URL, etc.) on the device. Without text input on the device, the values are hard coded.

Open up your Visual Studio Code and install the Monkey C extension.

Download a Connect IQ SDK (we use 4.1.4) via the SdkManager.

Open your clone of this repository in Visual Studio Code and add the garmin-connectiq folder to your Workspace (Menu: File => Add Folder to Workspace). Then, from the Visual Studio Code command palette choose “Monkey C: Build for Device” and build the binary. You will be asked for the device you want (add yours to the manifest.xml under iq:products) and the output folder.

Connect your device via USB cable and it should show up as a volume on your computer. Copy the binary to Garmin/Apps on this volume. Disconnect it, and you should be good to go.

The Application should show up as LokiRuns as an Activity. On a Fenix and Forerunner, you’d press the button on the top right then press Down three times until LokiRuns is highlighted. Another press of the top right button should start the applications, and you should see data in your Grafana Cloud stack soon after.

Creating a Grafana dashboard

The only thing left is setting up a nice dashboard where you can monitor your running metrics! Assuming you’re Grafana experts, we’re keeping things simple here and just putting a couple of panels together with metrics obtained from the logs.

Here is one we use for plotting the heart rate by user over time: 

avg_over_time ({app="lokiruns"} | logfmt | line_format{{.message}}| json  | unwrap heartrate [2s]) by (user)

Basically, what you need to do is:

1. Select the right stream: {app="lokiruns"}
2. Get rid of the non-json parts (e.g. timestamp="2022-07-30 10:36:39 +0000 UTC" kind=log message=")by: | logfmt | line_format{{.message}}
3. Parse the json message with the sensor data: | json 
4. Unwrap the value (so you can use it in an aggregate function: | unwrap heartrate 
5. Average the values over time by user: avg_over_time (... [2s]) by (user)

You’ll get something like the dashboard below. Here, we’re monitoring the last location, heart rate over time, pace over time, speed over time, and the last heart rate.

Seeing all of these metrics is helpful when analyzing your run and current fitness status. With Grafana 9.1 you can even make your dashboard public if you want to share your latest run with your trainer or friends and family.

Thanks for reading this — and run or bike fast, hard, and long!

Grafana Cloud is the easiest way to get started with metrics, logs, traces, and dashboards. We have a generous free forever tier and plans for every use case. Sign up for free now!