How we use the k6 load-testing tool for developing Grafana

On the last day of GrafanaCONline in June, our CEO Raj Dutt announced that Grafana Labs had acquired k6, the company behind the open source load-testing tool. In fact, our relationship with k6 had started more than two years earlier.

At the beginning of 2019, we were working on replacing Grafana’s “remember me” cookie solution with a short-lived token solution for the Grafana 6.0 release. During the development, we wanted to test how the refresh of short-lived tokens would behave when the system is put under heavy load and/or when scaling out and running multiple Grafana instances. Hence the search began for an OSS load-testing tool. 

I only had some limited experience using Locust, but it was implemented in Python and also required load tests to be implemented in Python, a language I’ve never used. One nice thing I remembered I enjoyed back then was that it had a UI to control and visualize the load test results. 

The Grafana backend and most of Grafana Labs’ other software is implemented using Go. So we looked for a tool that could be run and managed with the same simplicity and ease of use as other Go tools and services, and that still would allow load tests to be implemented in a language many of us already use. 

Enter k6

After searching a bit, I happened to find k6, which checked off the OSS and Go requirements. Load tests are written using JavaScript ES2015/ES6 — a good fit since the Grafana frontend/UI already uses it and some of the backend developers, including me, are familiar with it. Before trying it out, I remember I was really impressed by the documentation and examples showing the simplicity of the tool and how easy it looked to get started writing and running a load test. 

k6 is a high-performing load testing tool, scriptable in JavaScript. To achieve this, one of the trade-offs is that it doesn’t run in a browser, hence it doesn’t render web pages the same way a web browser does. The advantage of this approach is that it is significantly less resource-intensive, and thus more cost-effective, than other browser-based testing tools. Due to this, k6 load tests are preferably written targeting a HTTP API and/or HTTP endpoints returning non-visual responses that are easy to interpret for machines, like JSON. When running a k6 load test, there’s a large number of options, but as a simplified version, you can use a fixed duration or a fixed number of iterations together with a number of virtual users (VUs) to simulate load. In addition, k6 supports two different ways to verify the outcome of load tests: checks (assert) and thresholds (pass/fail). For more information, see the k6 documentation.

Here’s what a basic k6 script looks like:

Copy

import http from 'k6/http';

import { sleep } from 'k6';



export default function () {

  http.get('https://test.k6.io');

  sleep(1);

}

I started to write a load test that would simulate a user loading a Grafana dashboard that refreshes every 5 seconds. The scenario consisted of an initial iteration, called a setup function, to log in the user, followed by next iterations making one annotation request and 20 data source requests each. Each iteration was verified using checks, e.g., the user was authenticated with a properly set login cookie set. In addition, k6 verifies whether each HTTP request has returned an expected 200 OK status code by default (though this behavior can be changed).

To make the load test code easier to read, the code for providing a HTTP client for interacting with Grafana’s log-in endpoint and HTTP API and utility functions have been extracted to a reusable module. This is currently shared by all load tests. We also created a shell-script that acts as a convenient wrapper around the k6 Docker image. It allows us to provide an API for running load tests with sane defaults, while letting us override certain parameters when running local vs. scale-out scenarios. Check out all the details and readme in the load test directory of the Grafana GitHub repository.

Compared with my earlier experience using Locust, I really enjoyed the fact that with k6, the development workflow is optimized for developers. The ability to run a load test in the terminal and get instant feedback in a nicely organized output that’s easy to share in a textual representation is really nice and makes my life as a developer easier.

One of the other great things that I didn’t realize at first is the ability to monitor Grafana or your application while the k6 load tests are running. We have a Grafana high availability (HA) test setup using Docker Compose. With this setup, we can scale up the number of Grafana instances we want, use MySQL or Postgres as the backend database, and use Prometheus and Loki to ingest Grafana and database metrics and logs. We can then run our k6 load test against this setup, and alongside the k6 load test results we can visualize Grafana, Docker, and database metrics stored in Prometheus, and tail logs stored in Loki, allowing us to identify potential bottlenecks and problems and/or help us validate that certain load tests perform as expected.

Grafana ❤️ k6

Using k6 to verify the implementation of the new short-lived token solution was a great success. We managed to identify bottlenecks and errors in the code before actually shipping the code. 

A couple months later, we got reports that upgrading to Grafana v6.2 in combination with using the Grafana auth proxy resulted in a lot of 500 internal server errors when trying to load data for dashboard panels. An auth proxy load test was implemented to reproduce the problem, develop a solution, and verify the solution by running the load test comparing the results. k6 once again helped us track down a nasty bug.

Here are some other examples of how we’ve successfully used k6:

• A suggested improvement of the short-lived token solution could be verified by running the existing load tests, reducing risks for introducing new bugs, and getting it released to end users quickly. 
• Improved database query performance by implementing a k6 load test, running it to test and tune query performance of a slow database query using different indices and a rewrite of the original query. 
• As mentioned in this blog post, we used k6 to investigate how to protect against DDoS attacks and optimize the throughput for rendering images.

Conclusion

We’ve already gotten a lot of value out of k6 load testing, and now that k6 is part of the Grafana Labs family, I’m looking forward to working with the tool even more. 

For anyone who’s going down the same road we were on back in 2019, you should know that k6 already integrates well with CI/CD tools, and k6 Cloud can be used to schedule regular load tests so that you can keep an eye on how performance changes over time. This should allow you to be more proactive and catch bugs and problems earlier. 

I’m happy to report that there’s also an official k6 Cloud plugin for Grafana, so you can view your test results stored in k6 Cloud in Grafana. Alternatively, you can also use k6’s Prometheus remote_write feature to send metrics to Grafana Cloud. To learn more about using k6 with Grafana, be sure to sign up for the free webinar on Sept. 16, “Intro to load testing with Grafana and k6.”

Update: Grafana k6 load and performance testing is now available natively in Grafana Cloud. Existing k6 Cloud customers will not be impacted; they will be migrated to the new experience at a later date. Get started with Grafana Cloud k6 →