This is documentation for the next version of K6. For the latest stable release, go to the latest version.
Troubleshooting
This topic includes instructions to help you troubleshoot common issues with the k6 Operator.
Common tricks
Test your script locally
Always run your script locally before trying to run it with the k6 Operator:
k6 run script.js
If you’re using environment variables or CLI options, pass them in as well:
MY_ENV_VAR=foo k6 run script.js --tag my_tag=bar
That ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, running locally can help you check if the configured options are doing what you expect. If there are any errors or unexpected results in the output of k6 run
, make sure to fix those prior to deploying the script elsewhere.
TestRun
deployment
The pods
In case of one TestRun
Custom Resource (CR) creation with parallelism: n
, there are certain repeating patterns:
There will be
n + 2
Jobs (with corresponding Pods) created: initializer, starter, andn
runners.If any of these Jobs didn’t result in a Pod being deployed, there must be an issue with that Job. Some commands that can help here:
kubectl get jobs -A kubectl describe job mytest-initializer
If one of the Pods was deployed but finished with
Error
, you can check its logs with the following command:kubectl logs mytest-initializer-xxxxx
If the Pods seem to be working but not producing an expected result and there’s not enough information in the logs, you can use the k6 verbose option in the TestRun
spec:
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
name: k6-sample
spec:
parallelism: 2
script:
configMap:
name: 'test'
file: 'test.js'
arguments: --verbose
k6 Operator
Another source of info is the k6 Operator itself. It’s deployed as a Kubernetes Deployment
, with replicas: 1
by default, and its logs together with observations about the Pods from the previous section usually contain enough information to help you diagnose any issues. With the standard deployment, the logs of the k6 Operator can be checked with:
kubectl -n k6-operator-system -c manager logs k6-operator-controller-manager-xxxxxxxx-xxxxx
Inspect TestRun
resource
After you deploy a TestRun
CR, you can inspect it the same way as any other resource:
kubectl describe testrun my-testrun
Firstly, check if the spec is as expected. Then, see the current status:
Status:
Conditions:
Last Transition Time: 2024-01-17T10:30:01Z
Message:
Reason: CloudTestRunFalse
Status: False
Type: CloudTestRun
Last Transition Time: 2024-01-17T10:29:58Z
Message:
Reason: TestRunPreparation
Status: Unknown
Type: TestRunRunning
Last Transition Time: 2024-01-17T10:29:58Z
Message:
Reason: CloudTestRunAbortedFalse
Status: False
Type: CloudTestRunAborted
Last Transition Time: 2024-01-17T10:29:58Z
Message:
Reason: CloudPLZTestRunFalse
Status: False
Type: CloudPLZTestRun
Stage: error
If Stage
is equal to error
, you can check the logs of k6 Operator.
Conditions can be used as a source of info as well, but it’s a more advanced troubleshooting option that should be used if the previous steps weren’t enough to diagnose the issue. Note that conditions that start with the Cloud
prefix only matter in the setting of k6 Cloud test runs, for example, for cloud output and PLZ test runs.
PrivateLoadZone
deployment
If the PrivateLoadZone
CR was successfully created in Kubernetes, it should become visible in your account in Grafana Cloud k6 (GCk6) interface soon afterwards. If it doesn’t appear in the UI, then there is likely a problem to troubleshoot.
First, go over the guide to double-check if all the steps have been done correctly and successfully.
Unlike TestRun
deployment, when a PrivateLoadZone
is first created, there are no additional resources deployed. So, the only source for troubleshooting are the logs of k6 Operator. See the previous subsection on how to access its logs. Any errors there might be a hint to diagnose the issue. Refer to PrivateLoadZone: subscription error for more details.
Running tests in PrivateLoadZone
Each time a user runs a test in a PLZ, for example with k6 cloud run script.js
, there is a corresponding TestRun
being deployed by the k6 Operator. This TestRun
will be deployed in the same namespace as its PrivateLoadZone
. If the test is misbehaving, for example, it errors out, or doesn’t produce the expected result, then you can check:
- If there are any messages in the GCk6 UI.
- If there are any messages in the output of the
k6 cloud run
command. - The resources and their logs, the same way as with a standalone
TestRun
deployment
Common scenarios
Issues with environment variables
Refer to Environment variables for details on how to pass environment variables to the k6 Operator.
Tags not working
Tags are a rather common source of errors when using the k6 Operator. For example, the following tags would lead to parsing errors:
arguments: --tag product_id="Test A"
# or
arguments: --tag foo=\"bar\"
You can see those errors in the logs of either the initializer or the runner Pod, for example:
time="2024-01-11T11:11:27Z" level=error msg="invalid argument \"product_id=\\\"Test\" for \"--tag\" flag: parse error on line 1, column 12: bare \" in non-quoted-field"
This is a common problem with escaping the characters. You can find an issue in the k6 Operator repository that can be upvoted.
Initializer logs an error but it’s not about tags
This can happen because of lack of attention to the preparation step. One command that you can use to help diagnose issues with your script is the following:
k6 inspect --execution-requirements script.js
That command is a shortened version of what the initializer Pod is executing. If the command produces an error, there’s a problem with the script itself and it should be solved outside of the k6 Operator. The error itself may contain a hint to what’s wrong, such as a syntax error.
If the standalone k6 inspect --execution-requirements
executes successfully, then it’s likely a problem with TestRun
deployment specific to your Kubernetes setup. A couple of recommendations here are:
- Review the output of the initializer Pod: is it logged by the k6 process or by something else?
- :information_source: k6 Operator expects the initializer logs to contain only the output of
k6 inspect
. If there are any other log lines present, then the k6 Operator will fail to parse it and the test won’t start. Refer to this issue for more details.
- :information_source: k6 Operator expects the initializer logs to contain only the output of
- Check events in the initializer Job and Pod as they may contain another hint about what’s wrong.
Non-existent ServiceAccount
A ServiceAccount can be defined as serviceAccountName
in a PrivateLoadZone, and as runner.serviceAccountName
in a TestRun CRD. If the specified ServiceAccount doesn’t exist, k6 Operator will successfully create Jobs but corresponding Pods will fail to be deployed, and the k6 Operator will wait indefinitely for Pods to be Ready
. This error can be best seen in the events of the Job:
kubectl describe job plz-test-xxxxxx-initializer
...
Events:
Warning FailedCreate 57s (x4 over 2m7s) job-controller Error creating: pods "plz-test-xxxxxx-initializer-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found
k6 Operator doesn’t try to analyze such scenarios on its own, but you can refer to the following issue for improvements.
How to fix
To fix this issue, the incorrect serviceAccountName
must be corrected, and the TestRun or PrivateLoadZone resource must be re-deployed.
Non-existent nodeSelector
nodeSelector
can be defined as nodeSelector
in a PrivateLoadZone, and as runner.nodeSelector
in the TestRun CRD.
This case is very similar to the ServiceAccount: the Pod creation will fail, but the error is slightly different:
kubectl describe pod plz-test-xxxxxx-initializer-xxxxx
...
Events:
Warning FailedScheduling 48s (x5 over 4m6s) default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.
How to fix
To fix this issue, the incorrect nodeSelector
must be corrected and the TestRun or PrivateLoadZone resource must be re-deployed.
Insufficient resources
A related problem can happen when the cluster does not have sufficient resources to deploy the runners. There’s a higher probability of hitting this issue when setting small CPU and memory limits for runners or using options like nodeSelector
, runner.affinity
or runner.topologySpreadConstraints
, and not having a set of nodes matching the spec. Alternatively, it can happen if there is a high number of runners required for the test (via parallelism
in TestRun or during PLZ test run) and autoscaling of the cluster has limits on the maximum number of nodes, and can’t provide the required resources on time or at all.
This case is somewhat similar to the previous two: the k6 Operator will wait indefinitely and can be monitored with events in Jobs and Pods. If it’s possible to fix the issue with insufficient resources on-the-fly, for example, by adding more nodes, k6 Operator will attempt to continue executing a test run.
OOM of a runner Pod
If there’s at least one runner Pod that OOM-ed, the whole test will be stuck and will have to be deleted manually:
kubectl -f my-test.yaml delete
# or
kubectl delete testrun my-test
In case of OOM, it makes sense to review the k6 script to understand what kind of resource usage this script requires. It may be that the k6 script can be improved to be more performant. Then, set the spec.runner.resources
in the TestRun CRD, or spec.resources
in the PrivateLoadZone CRD accordingly.
PrivateLoadZone: subscription error
If there’s an issue with your Grafana Cloud k6 subscription, there will be a 400 error in the logs with the message detailing the problem. For example:
"Received error `(400) You have reached the maximum Number of private load zones your organization is allowed to have. Please contact support if you want to create more.`. Message from server ``"
To fix this issue, check your organization settings in Grafana Cloud k6 or contact Support.
PrivateLoadZone: Wrong token
There can be two major problems with the authentication token:
If the token wasn’t created, or was created in a wrong location, the logs will show the following error:
Failed to load k6 Cloud token {"namespace": "plz-ns", "name": "my-plz", "reconcileID": "67c8bc73-f45b-4c7f-a9ad-4fd0ffb4d5f6", "name": "token-with-wrong-name", "secretNamespace": "plz-ns", "error": "Secret \"token-with-wrong-name\" not found"}
If the token contains a corrupted value, or it’s not an organizational token, the logs will show the following error:
"Received error `(403) Authentication token incorrect or expired`. Message from server ``"
PrivateLoadZone: Networking setup
If you see any dial or connection errors in the logs of the k6 Operator, it makes sense to double-check the networking setup. For a PrivateLoadZone to operate, outbound traffic to Grafana Cloud k6 must be allowed. To check the reachability of Grafana Cloud k6 endpoints:
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -it dnsutils -- nslookup ingest.k6.io
kubectl exec -it dnsutils -- nslookup api.k6.io
For more resources on troubleshooting networking, refer to the Kubernetes docs.
PrivateLoadZone: Insufficient resources
The PrivateLoadZone insufficient resources problem is similar to insufficient resources issue. But, when running a PrivateLoadZone test, the k6 Operator will wait only for a timeout period. When the timeout period is up, the test will be aborted by Grafana Cloud k6 and marked as such, both in the PrivateLoadZone and in Grafana Cloud k6. In other words, there is a time limit to fix this issue without restarting the test run.