Build a data source backend plugin

Introduction

Grafana supports a wide range of data sources, including Prometheus, MySQL, and Datadog. In some cases, though, you already have an in-house metrics solution that you’d like to add to your Grafana dashboards. This tutorial teaches you to build a new data source plugin to query data.

A backend component provides a number of additional capabilities to your plugin, such as custom authentication methods. To learn more, refer to the documentation on Backend plugins.

In this tutorial, you'll:

• Build a backend for your data source
• Implement a health check for your data source
• Enable Grafana Alerting for your data source

Prerequisites

• Grafana v10.0 or later
• Docker
• Go (Version)
• Mage
• LTS version of Node.js

Create a new plugin

The Grafana create-plugin tool is a CLI application that simplifies Grafana plugin development, so that you can focus on code. The tool scaffolds a starter plugin, all the required configuration, and a development environment using Docker Compose for you.

1. In a new directory, create a plugin from a template using the create-plugin tool. When prompted for the kind of plugin, select and answer yes to "Do you want a backend part of your plugin?":

   npx @grafana/create-plugin@latest

2. Go to the directory of your newly created plugin:

   cd <your-plugin>

3. Install the dependencies:

   npm install

4. Build the plugin frontend:

   npm run dev

5. In a new terminal window, build the plugin backend:

   mage -v build:linux

6. Start Grafana:

   docker compose up
7. Open Grafana, by default http://localhost:3000/, and then go to Administration > Plugins. Make sure that your plugin is there.

You can also verify that Grafana has discovered the plugin by checking the logs:

INFO[01-01|12:00:00] Plugin registered       logger=plugin.loader pluginID=<your-plugin>

Now, let's verify that the plugin you've built so far can be used in Grafana when creating a new data source:

1. On the side menu, go to Connections > Data Sources.
2. Click Add data source.
3. Search for the name of your newly created plugin and select it.
4. Enter a name and then click Save & Test. If a "randomized error" occurs, you may ignore it - this is a result of the health check explained further below.

You now have a new data source instance of your plugin that is ready to use in a dashboard.

To add the data source to the dashboard:

1. Create a new dashboard and add a new panel.
2. On the query tab, select the data source you just created. A line graph is rendered with one series consisting of two data points.
3. Save the dashboard.

Troubleshooting

Grafana doesn't load my plugin

Ensure that Grafana has been started in development mode. If you are running Grafana from source, add the following line to your conf/custom.ini file:

app_mode = development

note

If you don't have a conf/custom.ini file already, create it before proceeding.

You can then start Grafana in development mode by running make run & make run-frontend in the Grafana repository root.

If you are running Grafana from a binary or inside a Docker container, you can start it in development mode by setting the environment variable GF_DEFAULT_APP_MODE to development.

note

By default, Grafana requires backend plugins to be signed. To load unsigned backend plugins, you need to configure Grafana to allow unsigned plugins. For more information, refer to Plugin signature verification.

Anatomy of a backend plugin

The folders and files used to build the backend for the data source are:

file/folder	description
Magefile.go	It’s not a requirement to use mage build files, but we strongly recommend using them so that you can use the build targets provided by the plugin SDK.
/go.mod	Go modules dependencies.
/src/plugin.json	A JSON file describing the plugin.
/pkg/main.go	Starting point of the plugin binary.

The plugin.json file

The plugin.json file is required for all plugins. When building a backend plugin, pay attention especially to these properties:

property	description
backend	Set to true for backend plugins. This tells Grafana that it should start a binary when loading the plugin.
executable	This is the name of the executable that Grafana expects to start. Refer to plugin.json reference for details.
alerting	If your backend data source supports alerting, set to true. Requires backend to be set to true.

In the next step we will look at the query endpoint!

Implement data queries

We begin by opening the file /pkg/plugin/datasource.go. In this file you will see the Datasource struct which implements the backend.QueryDataHandler interface. The QueryData method on this struct is where the data fetching happens for a data source plugin.

Each request contains multiple queries to reduce traffic between Grafana and plugins. So you need to loop over the slice of queries, process each query, and then return the results of all queries.

In the tutorial we have extracted a method named query to take care of each query model. Since each plugin has their own unique query model, Grafana sends it to the backend plugin as JSON. Therefore the plugin needs to Unmarshal the query model into something easier to work with.

As you can see the sample only returns static numbers. Try to extend the plugin to return other types of data.

For example to generate three floats equally spaced in time, you can replace the two static numbers generated, using the following code:

duration := query.TimeRange.To.Sub(query.TimeRange.From)
mid := query.TimeRange.From.Add(duration / 2)

s := rand.NewSource(time.Now().UnixNano())
r := rand.New(s)

lowVal := 10.0
highVal := 20.0
midVal := lowVal + (r.Float64() * (highVal - lowVal))

// add fields.
frame.Fields = append(frame.Fields,
  data.NewField("time", nil, []time.Time{query.TimeRange.From, mid, query.TimeRange.To}),
  data.NewField("values", nil, []float64{lowVal, midVal, highVal}),
)

You can read more about how to build data frames in our docs.

Add support for health checks

Implementing the health check handler allows Grafana to verify that a data source has been configured correctly.

When editing a data source in Grafana's UI, you can Save & Test to verify that it works as expected.

In this sample data source, there is a 50% chance that the health check will be successful. Make sure to return appropriate error messages to the users, informing them about what is misconfigured in the data source.

Open /pkg/plugin/datasource.go. In this file, you'll see that the Datasource struct also implements the backend.CheckHealthHandler interface. Go to the CheckHealth method to see how the health check for this sample plugin is implemented.

To learn more, refer to other Health Check implementations in our examples repository.

Add authentication

Implementing authentication allows your plugin to access protected resources like databases or APIs. To learn more about how to authenticate using a backend plugin, refer to our documentation.

Enable Grafana Alerting

1. Add the top level alerting property with a value of true to specify that your plugin supports Grafana Alerting, e.g.

   src/plugin.json

   {
     ...
     "backend": true,
     "executable": "gpx_simple_datasource_backend",
     "alerting": true,
     "info": {
     ...
   }

2. Restart your Grafana instance.

3. Open Grafana in your web browser.

4. Verify that alerting is now supported by navigating to your created data source. You should see an "Alerting supported" message in the Settings view.

Create an alert

note

The following instructions are based on Grafana v10.1.1, consult the documentation for alerting for version appropriate guidance.

1. Open the dashboard you created earlier in the Create a new plugin step.
2. Edit the existing panel.
3. Click on the Alert tab underneath the panel.
4. Click on Create alert rule from this panel button.
5. In Expressions section, in the Threshold expression C, set the IS ABOVE to 15.
6. Click on Set as alert condition on Threshold expression C. Your alert should now look as follows.

   

   Expression section showing B "reduce" with Input: A, Function: Last, Mode: Strict, C Threshold with Input: B, Is Above: 15 and Alert Condition enabled indicator

7. In Set alert evaluation behavior section, click on New folder button and create a new folder to store an evaluation rule.
8. Then, click on New evaluation group button and create a new evaluation group; choose a name and set the Evaluation interval to 10s.
9. Click Save rule and exit button.
10. Save the dashboard. After some time, the alert rule evaluates and transitions into the Alerting state.

Run multiple queries concurrently

note

This feature is only available for the Grafana backend plugin SDK version 0.232.0 and later.

By default, multiple queries within a single request (that is, within one panel) are executed sequentially. To run multiple queries concurrently, you can use the concurrent.QueryData function that the SDK exposes.

To use concurrent.QueryData, specify how to execute a single query and a limit on the number of concurrent queries to run. Note that the maximum is 10 concurrent queries.

import (
	...
	"github.com/grafana/grafana-plugin-sdk-go/experimental/concurrent"
	...
)

func (d *Datasource) handleSingleQueryData(ctx context.Context, q concurrent.Query) (res backend.DataResponse) {
  // Implement the query logic here
}

func (d *Datasource) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
	return concurrent.QueryData(ctx, req, d.handleSingleQueryData, 10)
}

Summary

In this tutorial you created a backend for your data source plugin.