Documentation Grafana Cloud Testing and synthetics Performance testing Analyze test results Collect profiles with Grafana Pyroscope

Collect profiles with Grafana Pyroscope

Grafana Cloud k6 enables you to execute performance tests that simulate real-world scenarios, providing critical insights into how your systems perform under load. By integrating testing and profiling data, you can dive deeper into data to understand the root causes and context of performance issues. The integration automatically links test metadata to profiling labels, ensuring a seamless workflow.

With this integration, you can:

• Access k6 performance test results and Pyroscope profiling data in a single interface, making it easier to correlate test metrics with application performance.
• Automatically connect k6 test metadata with profiling data, with minimal configuration required.
• Use flame graphs in the Profiles tab within Grafana Cloud k6 to identify bottlenecks and optimize application performance.
• Evaluate performance changes over time by comparing profiling data from multiple test runs.
• Visualize k6 test data in Explore profiles for deeper analysis.

To use this integration, you need to:

1. Set up instrumentation: Import the Pyroscope instrumentation library into your k6 test scripts and add middleware to your application.
2. Run performance tests: Execute your tests as usual. Profiling data is automatically captured and linked.
3. Investigate data using Explore Profiles: View profiling insights in the Profiles tab within Grafana Cloud k6 or dive deeper into the Pyroscope Explore Profiles feature.

Profiles and k6 test data

Connecting performance testing and profiling provides powerful contextual information about the tests you’re running. This integration links the two concepts together and enables you to go directly from your performance test to the profile performance implications of the load test.

Integrated testing and profiling data can be used for:

• Bottleneck Analysis: Diagnose why performance tests fail to scale and pinpoint inefficient code paths.
• Regression Testing: Compare profiling data across test runs to validate improvements or catch regressions.
• Scalability Validation: Test and optimize systems to handle increased workloads effectively.

To illustrate an example of how this feature can be leveraged, consider a web server that serves a REST API over HTTP. The service must support a throughput of 10k requests per second. To test this, a k6 load test is written to find the upper bound of the RPS throughput. After the test is run, it is observed the throughput of the service is only 4k requests per second. The question now is: Why couldn’t the service achieve the target 10k RPS?

This integration between Grafana Cloud k6 and Grafana Cloud Profiles can help quickly pinpoint the issue with throughput. Every request sent by a Virtual User (VU) of k6 attaches metadata about the load test (test run ID, scenario name, and several others) in the header of the request using OpenTelementry baggage. The Pyroscope SDKs offer extensions to their core library, which provide HTTP and gRPC middleware to map the k6 test metadata from the Baggage header into dynamic profile labels. This creates a correlation between a k6 test run and the corresponding profile information of that service during the test, which a developer could use to find the root cause of any bottlenecks in their application.

Before you begin

You can use the integration with the open source or Grafana Cloud editions of k6 and Pyroscope. Any data generated from open source k6 or Pyroscope needs to be stored in Grafana Cloud to be accessed by the integration.

To enable this integration, you’ll need:

• A Grafana Cloud account
• Grafana Cloud Profiles enabled
• Grafana Cloud k6 enabled

If you want to use a local installation to generate data, you can use:

• k6 data sent to Grafana Cloud from k6 v0.56 or higher for running tests locally
• Profiling data from any version of Pyroscope

For the Pyroscope integration to work, you need:

• An application that serves gRPC or HTTP requests set up to send profiling data with Pyroscope
• A Go or Java application
• (optional) OpenTelemetry baggage propagation configured to propagate k6 test metadata to upstream services

Set up instrumentation

There are two steps to instrument this integration:

• For k6: Import the Pyroscope instrumentation library and enable the integration to send metadata.
• For Pyroscope: Add the middleware to the application to tell the system under load to translate the k6 metadata header to dynamic profiler labels.

Configure the k6 test script

1. Import the Pyroscope instrumentation library in your k6 test script:

   JavaScript Copy

   import pyroscope from 'https://jslib.k6.io/http-instrumentation-pyroscope/1.0.1/index.js';

2. Add the line pyroscope.instrumentHTTP(); to your script, before the export default function:

   JavaScript Copy

   pyroscope.instrumentHTTP();
   
   export default function() {
     // ...
   }

The instrumentHTTP function ensures that all requests made by the HTTP module from this point forward have a baggage context attached. This context can be used by Pyroscope to create profiling data.

Configure Pyroscope middleware

The Pyroscope middleware for this integration is available for Go and Java.

Go

If you are using the Pyroscope Go SDK for profiling, you can use the library with useful helper functions to make it easy to enable dynamic labeling of profiles with k6 metadata. To install the helper library, run:

bash Copy

go get github.com/grafana/pyroscope-go/x/k6

This library offers two helper middleware functions for HTTP and gRPC.

To use the HTTP middleware, add the LabelsFromBaggageHandler into your HTTP middleware stack.

Go Copy

package main

import (
    "fmt"
    "net/http"
    "github.com/grafana/pyroscope-go/x/k6"
)

// Index handler
func index(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "hello world")
}

func main() {
    mux := http.NewServeMux()

    // Wrap the index handler with the helper middleware. Any requests
    // to this handler will have k6 metadata extracted and used to
    // annotate profiles.
    mux.Handle("/", k6.LabelsFromBaggageHandler(http.HandlerFunc(index)))

    // TODO: also initialize the Pyroscope profiler.

    http.ListenAndServe(":8080", mux)
}

Test the integration

To verify the integration is set up successfully, save the following test script into a load.js file.

JavaScript Copy

import http from 'k6/http';
import { check } from 'k6';
import pyroscope from 'https://jslib.k6.io/http-instrumentation-pyroscope/1.0.1/index.js';

pyroscope.instrumentHTTP();

const APP_URL = __ENV.APP_URL;

export const options = {
  vus: 1,
  duration: '1m',
};

export default function () {
  let res = http.get(`${APP_URL}`);

  check(res, {
    'is status 200': (r) => r.status === 200,
  });
}

Run the test script, replacing <APP URL> with your application URL:

bash Copy

APP_URL=<APP URL> k6 run load.js

Finally, open your Grafana instance and navigate to Explore Profiles > All services and select Labels on your service. After a few moments, you should begin to see k6 metadata being added as labels to your service.

This indicates everything is set up correctly and you should be able to run tests from Grafana Cloud k6.

Run performance tests

To use this integration, you need to run a k6 test script in Grafana Cloud k6. You can do that in three different ways:

• Option 1: Run a local k6 script in Grafana Cloud:

  bash Copy

  k6 cloud run script.js

• Option 2: Run a local script in your machine and stream results to Grafana Cloud:

  bash Copy

  k6 cloud run --local-execution script.js

• Option 3: Run a test created via the script editor.

After you run the test:

1. Locate the test data in your Grafana Cloud account. Open Testing & synthetics > Performance > Projects.
2. In the Projects page, open the project your test was executed on, then open your test, and click on the latest test run.
3. In the test run results page, at the bottom of the page, locate the Profile tab.

The Profile lets you view a flame graph of the profiling data generated by your application as a result of the load test you’re inspecting. You can use this data to find bottlenecks in your application and know exactly what line of code needs to be fixed.

The Profile tab has several options:

• Service Name: A drop-down list of all of the services that have profiles. One service may collect multiple profile types.
• Profile Type: Any service that the profile collects. The supported profile types vary based on the SDK and the programming language.
• Filters: The Add filter drop-down lets you choose which additional filters you want to use.
  • Scenario: The name of the scenario in the k6 test script.
  • URL: The URL values sent by requests made in the k6 test script.
• Open in Pyroscope: Open Explore Profiles with the same service, profile, and filters applied to it.

You can modify the flame graph similarly to the Explore Profiles application.

Analyze with Explore Profiles

When you’re in the Profile tab of a k6 test run result, Select Open in Pyroscope to open Explore Profiles.

Explore Profiles provides an intuitive interface for exploring your profile data. Using Explore Profiles, you can get a high-level view of all of your services and how they’re functioning, find areas to optimize your services, or identify root causes of issues.

All of the parameters used in the Profile tab in k6 are automatically set in Explore Profiles, such as the test run id and time range of the k6 test. This allows you to navigate throughout the Explore Profiles app and further explore the profiling data.

When you select other profiles, some filters may be removed. This occurs because some profile types don’t support dynamic labels, so they won’t have the k6 test run metadata as labels. To add these filters back again, you can use the Filters section of Explore Profiles query builder.

Profiling information provided in the k6 UI is opinionated and optimized for at-a-glance analysis. If this is insufficient, Explore Profiles provides more tools to support deeper explorations.

Compare test runs

A core part of performance testing is checking for regressions in your application’s performance. You can do that in Grafana Cloud k6 by comparing test runs, to visualize the performance overview and metrics of your test run results. This also applies to the Profile tab, where you can compare two test runs that are instrumented with the profiling library.

To compare two test runs:

1. Go to your Performance testing app in Grafana Cloud.
2. Open a test run result.
3. In the top right, select Compare result.
4. Select the test run you want to compare the test to.

The layout of the page changes when you’re comparing test results. The left side shows the baseline test, and the right side shows the comparison test.

When you open the Profile tab in comparison mode, you can see a diff flame graph. It provides an overview to investigate what happened between two test runs. You can click on Open in Pyroscope to go to Explore Profiles, and see the same diff flame graph view, and continue to investigate any bottlenecks or performance issues in your application.