Documentation Grafana Cloud Testing and synthetics Performance testing Author and run tests Set up private load zones

Set up private load zones

In Grafana Cloud k6, there are 20+ load zones that you can use to run your tests from different locations. But what happens if you:

• Want to run your tests from a location that’s not included in the default load zones?
• Want to test an internal service that isn’t exposed to the internet?

Private Load Zones (PLZ) are load zones that you can host inside your network. They can run on top of any Kubernetes cluster you have. These PLZs are built on top of the existing k6 Operator project as an additional custom resource definition (CRD). You can start a cloud test in a PLZ by referencing it by name from your script, and the test executes on the nodes of your Kubernetes cluster.

Before you begin

To set up and use a Private Load Zone, you’ll need:

1. A Grafana Cloud account.
2. A Kubernetes cluster (version 1.23 or higher):
   • That can access the APIs and services you wish to test.
   • That can access “https://ingest.k6.io” and “https://api.k6.io”

Set up a Private Load Zone

First, you need to set up a Private Load Zone in your Kubernetes cluster.

Install k6 Operator

There are two ways to install k6 Operator in your cluster.

With bundle

Install k6 Operator by using the k6 Operator YAML bundle generated on every release:

bash Copy

kubectl apply -f https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml

By default, k6 Operator is installed into the k6-operator-system namespace. You can check that it’s running there with:

bash Copy

kubectl get pods -n k6-operator-system

And the output should look similar to:

bash Copy

NAME                                              READY   STATUS    RESTARTS   AGE
k6-operator-controller-manager-6f88ddf8df-7vvgx   2/2     Running   0          12s

With Helm

Install k6 Operator by using the Helm chart:

bash Copy

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install k6-operator grafana/k6-operator

With the command above, the operator will be installed into the k6-operator-system namespace. You can check that it’s running there with:

bash Copy

kubectl get pods -n k6-operator-system

And the output should look similar to:

bash Copy

NAME                                              READY   STATUS    RESTARTS   AGE
k6-operator-controller-manager-6f88ddf8df-7vvgx   2/2     Running   0          12s

Create the PLZ CRD

1. Next, you must pick the namespace where you want to create your Private Load Zone and where k6 PLZ tests are to be executed by creating a custom resource definition (CRD). Namespace can be different from where the k6 Operator executes.

   Create a new namespace (replace plz-ns with the name you would like to use):

   bash Copy

   kubectl create namespace plz-ns

2. Generate a new Grafana Stack API Token:

   • Log in to your Grafana Cloud account.
   • Go to the Performance testing app.
   • Select Settings.
   • Select the Grafana Stack API token tab.
   • Select Create token.
   • Give your token a name and select Create new token.
   • Copy and save the token.

   Note

   This must be a Grafana Stack API Token and not a personal token.

3. Then, create a Kubernetes secret with the token by using the following command:

   bash Copy

   kubectl create secret generic grafana-k6-token -n plz-ns \
     --from-literal=token=GRAFANA_STACK_API_TOKEN

   Note

   You can also use external-secrets or other tools to create a secret.

4. Now, create a definition of your Private Load Zone as a plz.yaml file. Replace PLZ_NAME with the name you’d like to use in your tests:

   bash Copy

   apiVersion: k6.io/v1alpha1
   kind: PrivateLoadZone
   metadata:
     name: PLZ_NAME
     namespace: plz-ns
   spec:
     token: grafana-k6-token
     resources:
       limits:
         cpu: 256m
         memory: 1024Mi

   The token and resources.limits parameters are required. Resources must indicate how much CPU and memory a pod executing a k6 test can take up on a node.

   You can only configure resources while creating a Private Load Zone. To change your resource values, you have to delete and re-create the Private Load Zone.

5. Now that the Private Load Zone is defined, create it in your cluster:

   bash Copy

   kubectl apply -f plz.yaml

   Warning

   The k6 Operator only supports one Private Load Zone per cluster. If you add multiple Private Load Zones to a cluster, your tests might not work correctly. Refer to PLZ lifecycle for additional information.

6. Check that your Private Load Zone was created:

   bash Copy

   kubectl -n plz-ns get privateloadzones.k6.io

Run a test in the PLZ

After you create a Private Load Zone, any user in your organization with the ability to run a test in Grafana Cloud k6 can run a test using your PLZ.

Running a test in a PLZ has certain limitations:

• You can only specify one load zone, and the percent property value should be 100.
• You can’t specify multiple PLZs in a single script.
• You can’t use a PLZ with public load zones.
• You can’t run k6 with browser tests.
• You can’t pass CLI arguments to k6.
• You can’t pass more complex node configurations, such as affinity rules.

From the CLI

Make sure that you have authenticated with the k6 CLI.

Then, go to a project or create a new project, and copy the ProjectID by selecting the Copy to clipboard icon.

Create a new file and name it plz-test.js. You can copy the example script below, and make sure to change:

• PLZ_NAME to the Private Load Zone name you defined in the metadata property of your CRD file.
• PROJECT_ID to the projectID you copied in the previous step

js Copy

import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { target: 200, duration: '3m30s' },
    { target: 0, duration: '30s' },
  ],
  cloud: {
    projectID: PROJECT_ID,
    distribution: {
      private: { loadZone: 'PLZ_NAME', percent: 100 },
    },
  },
};

export default function () {
  const result = http.get('https://test-api.k6.io/public/crocodiles/');
  check(result, {
    'http response status code is 200': result.status === 200,
  });
}

Now run your test with:

bash Copy

k6 cloud run plz-test.js

From the UI

Go to a project, or create a new one.

In the top right corner of the project page, click on Create new test.

You can use the Test Builder or Script Editor to create a new test.

Script Editor

By default, the Script Editor will create an example test script. Replace the load zone section from:

js Copy

'amazon:us:ashburn': { loadZone: 'amazon:us:ashburn', percent: 100 },

To:

js Copy

'private': { loadZone: PLZ_NAME, percent: 100 },

Make sure to replace PLZ_NAME with the Private Load Zone name you defined in the metadata property of your CRD file.

Make other changes to the script as necessary, and select Create and Run to run the test.

Test Builder

By default, your test will be empty. Under Options, select Load zones, and then select the load zone drop-down and pick your PLZ. Add all the requests that your test needs, and select Create and Run to run your test.

Additional information

PLZ configuration options

There are a few additional options you can use in your Private Load Zone definition:

Example of an extended definition:

yaml Copy

apiVersion: k6.io/v1alpha1
kind: PrivateLoadZone
metadata:
  name: PLZ_NAME
  namespace: plz-ns
spec:
  token: grafana-k6-token
  resources:
    limits:
      cpu: 256m
      memory: 1024Mi
    requests:
      cpu: 256m
      memory: 512Mi
  serviceAccountName: plz-sa
  nodeSelector:
    foo: bar
  image: <REGISTRY>/<IMAGE>

When a test is created, serviceAccountName and nodeSelector are passed to the spec of all pods as spec.serviceAccountName and spec.nodeSelector respectively. Resources are passed only to runner pods.

Environment variables

You can set cloud environment variables and reference them in your PLZ tests to make your tests more reusable and configurable. For example, they can be used to switch between test environments or to store credentials.

Private Load Zone lifecycle

The k6 Operator only supports one Private Load Zone per cluster. The k6 Operator reads the PLZ definition, registers it against Grafana Cloud, and starts the polling loop for PLZ test runs.

If you add a second Private Load Zone to the same cluster, it’s not picked up by the k6 Operator on its own: all other PLZ objects except the first one are ignored. Additionally, if there are two or more PLZ objects in the cluster and the Operator Pod is restarted, only one PLZ object is picked up and processed by the Operator, and it’s impossible to predict which one. Avoid having more than one PLZ object per k6 Operator deployment at the same time.

If you need another PLZ definition, delete the previous one and create a new one with kubect apply -f plz.yaml.

Multiple deployments

As a workaround for the lack of multiple PLZ support, you can deploy multiple k6 Operator deployments and configure them to watch different namespaces within the same cluster. You can set up a different namespace and k6 Operator deployment for each PLZ you need.

Connectivity loss

If your k6 Operator deployment with the registered PLZ stops polling Grafana Cloud, the PLZ is marked as “Offline” after 10 minutes. The PLZ can then be deleted from the UI. An “Offline” PLZ is considered an “invalid load zone”, and a test cannot be started with it.

There are two common cases where the PLZ polling can suddenly stop:

• Your cluster has a connectivity issue that doesn’t impact the PLZ object directly. In this case, after the cluster comes back online, the PLZ should come back “Online” in the UI, allowing you to start tests with it again.
• Your cluster has an issue that damaged the PLZ object. In this case, restoring the cluster is insufficient: the “Offline” PLZ must be removed from the UI first, then a new PLZ can be registered.

Test lifecycle

When you run a test in a PLZ, during its creation Grafana Cloud k6 estimates how many pods are required to run such a test. Then, k6 Operator starts the test by creating a TestRun CRD with all the parameters GCk6 provides.

If your Kubernetes setup doesn’t have enough resources to run the test, GCk6 will wait for 10 minutes (to wait for auto-scaling or some other event that frees / creates resources). If enough resources are still not available after that time, the test will fail to be scheduled.

After the test is finished, k6 Operator deletes the TestRun CRD.

You can find more details about the PLZ logic and implementation in Private Load Zone under the hood.