Comparing k6 and JMeter for load testing

When I joined the k6 team in late November, the foremost question on my mind was “How does this compare to JMeter?” You see, I come from a performance testing consulting background, and in the last few years, JMeter has been far and away my tool of choice.

So what am I doing on the k6 site, talking about how much I love JMeter? Well, firstly, it turns out that the k6 team is pretty open and transparent, especially about potential improvements in k6. Secondly, I want to share my reasons for wanting to explore k6 as a load testing tool — the same reasons that ultimately convinced me to join the team that builds it.

What is JMeter?

JMeter is an open source load testing tool built entirely in Java by the Apache Foundation. It was first released in 1998, and it made waves because of its audacity in taking on more popular but proprietary load testing tools. JMeter took what other companies were charging a lot of money for and published it online for everyone to use — for free. Although scripts can also be extended using code, the majority of the scripting in JMeter is done using the UI. At the time of this writing, the latest version of JMeter is 5.4.

What is k6?

k6 is an open source load testing tool released by Load Impact in 2017. LoadImpact had previously consisted of the SaaS platform (now Grafana Cloud k6) and professional services. k6 is written in Go, but scripts are written in JavaScript. Taking the opposite tack to JMeter, k6’s main selling point is its code-based scripting and heavy prioritization of the developer experience. The latest version of k6 is 0.30.

Which is better?

This article isn’t about which tool is better, because as much as we might prefer a clear-cut answer, there’s no “best tool.” Everything depends on the situation your team is in: what, how, why, and when you’re testing.

The question that I want to answer here is: What situations does each tool excel in?

When does JMeter excel?

When you want to get started quickly

JMeter has a GUI. Many testing teams make decisions on load testing tools based on this fact alone. Sometimes, if you’re in a situation where people haven’t done load testing before and need to learn a new tool, a GUI-driven tool like JMeter is simply the most frictionless option. A tool with a GUI will be less daunting for many testers that are used to the mostly no-code UIs of the likes of Postman or SOAPUI.

I’ve always thought of JMeter as user-friendly, but a developer friend recently pointed out that the Test Plan screen that greets you when you first start JMeter doesn’t provide any indication of how to create an HTTP sampler. None of the icons add elements to the test plan. It takes a bit of exploring to realize that right-clicking on the test plan, or going to Edit > Add, will show the “Add” menu that you need. He has a point there: UI is subjective to a certain degree. However, I’d argue that it’s definitely easier for non-developers to explore a UI than a bit of code.

k6 doesn’t have a GUI packaged within the tool, but the k6 Test Builder is available for free. It’s a way to create tests with a GUI interface, and despite its inclusion in k6 cloud, it doesn’t require any subscription to use. However, it isn’t as fully featured as JMeter.

JMeter also supports many protocols and features out of the box. Here’s the official list of the protocols that JMeter supports:

• Web - HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
• SOAP / REST web services
• FTP
• Database via JDBC
• LDAP
• Message-oriented middleware (MOM) via JMS
• Mail - SMTP(S), POP3(S) and IMAP(S)
• Native commands or shell scripts
• TCP
• Java Objects

In contrast, k6 supports the following protocols:

• Web - HTTP/1.1, HTTP/2 (Java, NodeJS, PHP, ASP.NET, …)
• WebSockets
• gRPC
• SOAP / REST web services

The protocols above are those that are natively supported. However, both tools allow the development of third-party plugins, adding support for more protocols. The number of k6 community extensions, through xk6, is growing, but JMeter plugins still outnumber them. In many cases, the protocols your application uses may make the decision of which tool to you for you.

JMeter also has most of the features you’ll need for a basic load test, and you don’t have to code any of it up yourself. Its parent-child element structure means that you can modify a specific element, like an HTTP request, or modify all HTTP requests, with equal ease. Adding think time to all requests is as easy as adding a Uniform Random Timer to your thread group, and all requests within that group will inherit it. With code-based load testing tools, it can be a bit more difficult to find out what’s possible without a UI. Some of the gap, however, can be bridged by an IDE with good autocompletion.

When you want a mature community with tons of documentation

JMeter’s old, and sometimes that’s a good thing. It’s been around since 1998, so at this point, it’s had 22 years to improve and to build a following. JMeter has a presence on nearly every community networking site, and no matter how specific your use case for JMeter is, the chances are that it’s been done before. Searching “How to load test X with JMeter” is bound to yield thousands of hits and some videos showing you exactly how to do it.

JMeter’s Component Reference is proof of JMeter’s extensive and thorough documentation. Every element, function, and property is discussed in more detail than most people could ever want, and that’s just the “official” documentation that’s on the Apache site. There are thousands of JMeter books, tutorials, and courses by its passionate users.

The JMeter GitHub repository has 17,290 commits as of December 2020 — evidence of a community of developers that have committed some time and experience to JMeter over the last two decades.

In comparison, the k6 community is growing, but still small. There is an official and central community forum that allows users to share their experiences with k6, which is something missing from the JMeter community. The k6 repository does have more stars, but significantly fewer commits at 3,795. Much of the disparity, of course, is because k6 is very new — the open-source tool k6 was only released in 2017.

When you need a cost-effective way to do distributed load testing

One of JMeter’s absolute best features is that it gives you a framework for running distributed load tests with it. This is pretty special for a free and open source tool.

Distributed testing means ramping up the amount of load you’re generating with your load tests, usually by increasing the number of virtual users and running multiple instances of your script on other load generators. JMeter accomplishes this by designating a controller node and letting you set up worker nodes. Each worker node needs to have a copy of jmeter-server, a utility included with every JMeter installation (in jmeter/bin).

These worker nodes then run copies of your test plan and exert load against your application server. You can use on-premises or cloud machines as your worker nodes. A distributed execution like this takes a bit to set up, but it does work surprisingly well.

k6 supports distributed load testing mode via a Kubernetes operator: k6 Operator allows you to create a k6 custom resource object within a Kubernetes cluster. Aside from its usefulness for applications already operating within Kubernetes, the operator also allows for easy distributed testing.

It’s important to note that JMeter can be quite resource-hungry, and thus is more likely to require multiple load generators to generate load than k6. Depending on the resources of the load generator, JMeter can run about a thousand virtual users on average, and scaling up your test beyond that point will require a distributed execution setup. A single instance of k6, on the other hand, can run tens of thousands of virtual users given the same resources. More on that later.

When you want canned reports built into your load testing tool

JMeter’s many listeners allow you to decide exactly what format you want to view results in. Debugging? Use the View Results Tree. Wanting metrics for your load test? Use the Summary or Aggregate Report.

In addition, JMeter can generate HTML reports based on your load tests. They’re currently quite limited, but functional, and they have several default graphs to help you analyze your load test. They are not interactive, but they are pretty easy to generate and a good starting point for putting together a report template for your tests.

k6 does not have built-in reporting. Its modular nature lends itself better to integrating with many other data analytics tools, including Grafana, Datadog, New Relic, Amazon CloudWatch, and k6 Cloud — all of which are better analysis tools and can provide more insights than what JMeter’s reports can provide, but they do require you to set up another tool to analyze your tests. If you’re running low on time and can make do with its reports, JMeter does really well at providing that extra value out of the box.

When does k6 excel?

When you want to get started quickly

Yup, I put this one in both the JMeter and k6 sections. JMeter and k6 can each be easy to get started with, but for very different reasons.

Installation

k6 installs extremely quickly; on my first installation of it on macOS, it took seven seconds via brew install k6. It doesn’t require nodeJS or any other dependencies, either.

JMeter itself is fairly easy to install, but before you install JMeter, you need to install Java. Especially if you’re using Windows, it’s a bit of a rite of passage for a new JMeter user to run into issues adding the environment variables needed for Java to function.

It can also be confusing to determine which version of Java is the right one. There’s JDK, JRE, and recently Oracle has added to the confusion by announcing that some versions of Java (but not all) are going to require a license.

Note: The Oracle Technology Network License Agreement for Oracle Java SE is substantially different from prior Oracle Java licenses. Oracle suggests you review the terms before downloading and using it.

While JMeter is still free via OpenJDK, I’ve spoken to a few testers who mistakenly thought they’d have to pay for JMeter. It’s a little disconcerting to see Oracle flexing its muscles here, especially if you’ve built a whole performance testing suite on Java through JMeter.

Plugins

k6 supports plugins, but they’re not necessary. For most use cases, k6’s built-in functionality should be more than sufficient. This is in sharp contrast to JMeter, where everything is a plugin … even the Plugin Manager!

Most JMeter users would argue that there is a standard set of plugins that you should download before you even begin to use it. A new JMeter user might not necessarily know where to find these and may have a less-than-satisfying experience out of the box.

Here are a few things you won’t have if you stick to a vanilla, no-plugin JMeter installation:

• Thread groups: No custom load profile, stages, rendezvous
• Controllers: No parallel controller to execute requests in parallel
• Samplers: No HTTP/2 support
• Test Data: No randomization of lines within a CSV file (sequential only)
• Timers: No throughput shaping timer for setting throughput in stages

These functions are all available within k6 from the get-go.

Scripting

k6 scripts are written in JavaScript, making them simple to write and run. All you need is one installer (on Microsoft Windows) or a single command on your terminal and any text editor. For JMeter, you’ll need to install a few things locally before you can get started: Java, JMeter itself, and the standard set of plugins you’ll need. This can be confusing for new users. Who hasn’t struggled with environment variables at some point when learning to use JMeter, especially on Windows?

It also takes time to learn any new UI, whereas JavaScript is near-ubiquitous and it’s a good language to learn when you’re load testing web sites anyway.

When you want to maximize performance and efficiency

Test your test tools!

Why does all this matter?

When we’re talking about simulating many virtual users, slow performance = high cost. There is a cost to the number of load generators you use, whether it’s the cost to acquire and maintain them (on-prem) or the cost of a cloud service.

It makes sense to consider the performance limitations of your load testing tool before you use it to improve your own application’s performance.

k6 builds on Go’s stellar performance

k6 is written in Go, and Go is built with performance in mind. Go is a compiled language and does not need to be interpreted, unlike Java or Python. There is no added layer of complexity. It also has no external dependencies, which is a good thing. Fewer moving parts for a performance tester means fewer potential sources of a bottleneck.

The simplest and most common way for a load testing tool to run a virtual user is to assign one virtual user to one kernel or OS thread. However, the 1 Thread: 1 Virtual User paradigm has a serious flaw. When a virtual user is waiting for a response or executing a programmed sleep(), the thread is also blocked, and cannot process other work.

In k6, each virtual user is run on a goroutine, not a thread. What difference does that make? Well, goroutines can be controlled by something called the Go Scheduler, which acts like a traffic cop. It reuses idle threads and intelligently assigns work, by allowing “work stealing” and work hand-offs between threads. Does this sound familiar? This is the same principle that load balancers are built on: an external monitor that oversees the flow of work improves general performance. Go itself is intrinsically load-balanced in a way that many programming languages aren’t, which makes it the perfect foundation for a load testing tool.

Being able to leverage Go’s innate performance optimizations also translates into significantly less memory utilization. One thread running k6 doesn’t go above 100 KB, whereas a JVM thread like JMeter uses, for instance, uses the default of 1 MB. That’s 1000% more than k6! Of course, Java allows a user to tweak an app’s memory utilization, so the difference is usually not quite as stark, but it’s still interesting to note that Go has a much lower starting point.

“Creating [a] new thread in Java is not memory efficient. As every thread consumes approx 1 MB of the memory heap size and eventually if you start spinning thousands of threads, they will put tremendous pressure on the heap and will cause shut down due to out of memory. Also, if you want to communicate between two or more threads, it’s very difficult.” - Keval Patel

Performance benefits in practice

Fewer load generators needed

k6’s comparatively better performance means it needs fewer load generators to execute a given amount of load. Rafaela Azevedo made a comparison of the memory used by k6 and JMeter, and here are her results:

JMeter took up 760 MB of memory.

Meanwhile, k6 took up 256 MB of memory.

Rafaela’s findings are further affirmed by our own benchmarks.

k6’s lower memory footprint is the reason it can run more virtual users and generate more load than average. Whether you’re using load generators that are on-premises or in a cloud, you’ll pay less for provisioning costs when using k6. This cost saving makes k6 a great tool for budget-conscious teams.

“The common misconception of many load testers is that distributed execution (ability to launch a load test on multiple machines) is required to generate large load. This is not the case with k6.” - Rafaela Azevedo

There are many different types of load testing tools, but what makes k6 is different is the way it handles hardware resources. A single k6 process will efficiently use all CPU cores on a load generator machine. A single instance of k6 is often enough to generate load of 30.000-40.000 simultaneous users (VUs). This amount of VUs can generate upwards of 300,000 requests per second (RPS).

No out of memory errors

k6 is also good for when you’re short on time. Using JMeter means getting familiar with how to performance tune Java and how to fix the most common Java performance problems, because there are several. The one that I’ve personally run into the most while using JMeter is:

Copy

ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: 
Java heap space. 

You’re not really a JMeter load tester until you’ve seen this error. :)

The cause for this error is usually that there was an insufficient amount of Heap memory allocated to JMeter. Yup, in addition to monitoring your load generator’s memory usage, you’ll also need to monitor the JVM’s memory usage. You can set the amount of heap memory allocated to JMeter by modifying the JMeter binary itself and changing this line:

Copy

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
: "${HEAP:="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m"}"

Even JMeter official documentation suggests that you increase the Java Heap size beyond 1 GB before you even run your first load test.

Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run." - Apache JMeter

No GUI means no resource overhead during load tests

GUIs typically add significantly more overhead to an application, which is one of the reasons why k6 doesn’t have one.

When you’re running a load test, the best practice is to run it from the command line anyway — because otherwise, it interferes with your results. Code-based load testing tools are performant from the beginning.

JMeter is peppered with warnings about the GUI’s effect on performance. Here’s one from the JMeter documentation: “Don’t run load test using GUI mode!”

And here’s a message that occurs upon startup of JMeter:

Not having a GUI means there’s one less gotcha. Any load testing scripts you create in k6 are ready for prime-time execution whenever you’re ready.

When you want to do goal-oriented testing

The most important question a load tester can ask when starting a new load testing project is “Why?” Test plans should be made to directly address a team’s reasons for wanting to do load testing, and that should be reflected in the non-functional requirements. Non-functional requirements, in turn, should then give some guidance as to the bounds of acceptable performance. A common example for these bounds, or thresholds, is an average response time of less than 3 seconds for all transactions during the test. k6 does this natively with global thresholds worked into the script, and you can also create your own metric to use in a threshold:

Copy

import http from 'k6/http';
import { Rate } from 'k6/metrics';

const myFailRate = new Rate('failed requests');

export const options = {
  thresholds: {
    'failed requests': ['rate<0.1'], // threshold on a custom metric
    'http_req_duration': ['p(95)<500'], // threshold on a standard metric
  },
};

export default function () {
  const res = http.get('https://test-api.k6.io/public/crocodiles/1/');
  myFailRate.add(res.status !== 200);
}

JMeter doesn’t natively support thresholds at the test-level. There are a few ways that I’ve been able to work around this:

• Duration Assertion: You can add a Duration Assertion to any sampler in JMeter, which allows you set an acceptable response time for that request in milliseconds, beyond which the request (and the transaction that encompasses it) would be marked as failed. However, this is still just at the individual request level, and even putting the duration assertion element at the Test Plan level only applies the same threshold to each child request. It doesn’t constitute pass/fail criteria for the test as a whole.
• Timers: For criteria related to throughput, you can use the Constant Throughput Timer or the Throughput Shaping Timer (plugin). This approach controls how many requests per second are spent. In my experience, it’s worth testing how these timers behave when using nested transaction controllers - it may take some rearranging to get right.
• Performance Plugin: When integrating JMeter with Jenkins, you can use the Performance Plugin to set error thresholds and mark test runs (“builds”) as unstable or failed within Jenkins.
• Custom Code: You can use the JSR223 Sampler to write a bit of Groovy or BeanShell code that collects and parses response metrics.

The problem is that even though these options make setting thresholds possible in JMeter, they’re still workarounds that don’t come with JMeter out of the box or don’t adequately address the creation of different types of thresholds (error, response time, throughput, CPU, memory). What I end up doing, more often than not, is exporting the raw data from JMeter and doing the analysis myself using another tool.

When you’re part of a team of scripters

k6 shines in situations where more than one person could be doing the scripting.

It strikes a middle ground that developers and testers can compromise on

Software quality should not be the sole responsibility of testers; quality should be baked into every activity when building software. One of the challenges in implementing this concept, however, is that there has traditionally been a segmentation in tools and languages used by team members with different functions. Developers use one tool; testers use another. It’s difficult to encourage developers to test and testers to code in this environment, and many testing tools only further encourage this divide. A developer can’t use a tool like JMeter for anything but testing, so the incentive to invest the effort in learning it is lower.

k6 bridges the gap by bringing testing concepts and features to an environment and language that developers are already spending their time in. JavaScript is useful for more than just writing load testing scripts, and k6 works well with any IDE or text editor. A frictionless developer experience is central to the k6 philosophy.

It’s easier to collaborate on scripts

Once you’ve convinced both devs and testers to contribute to load testing scripts, how do you keep track of changes? How do you manage scripts that multiple people are working on at the same time? With tests as code, the easiest way is to just use versioning tools like Git, the way you probably are already using for your application code. Again, no new framework or processes to learn here. k6’s conciseness and portability really proves to be an advantage in collaboration situations. To illustrate this, let’s consider a simple test script that contains a few things:

• A GET request to our test site, https://test.k6.io
• A transaction that calls this request 01_Home
• A check to fail non-HTTP 200 responses or responses that don’t have the text “Collection of simple web-pages suitable for load testing”
• Variable think time
• We’ll just run that as a single user, for a single iteration. Here’s what that script looks like in k6:

Copy

import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
  const res = http.get('https://test.k6.io', { tags: { name: '01_Home' } });
  check(res, {
    'is status 200': (r) => r.status === 200,
    'text verification': (r) =>
      r.body.includes('Collection of simple web-pages suitable for load testing'),
  });
  sleep(Math.random() * 5);
}

You can track JMeter scripts on Git, too. Here’s what the same script looks like in JMeter:

Copy

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.3">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <TransactionController guiclass="TransactionControllerGui" testclass="TransactionController" testname="01_Home" enabled="true">
          <boolProp name="TransactionController.includeTimers">false</boolProp>
          <boolProp name="TransactionController.parent">true</boolProp>
        </TransactionController>
        <hashTree>
          <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">test.k6.io</stringProp>
            <stringProp name="HTTPSampler.port"></stringProp>
            <stringProp name="HTTPSampler.protocol">https</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path"></stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
            <stringProp name="HTTPSampler.connect_timeout"></stringProp>
            <stringProp name="HTTPSampler.response_timeout"></stringProp>
          </HTTPSamplerProxy>
          <hashTree>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="-745788246">Collection of simple web-pages suitable for load testing</stringProp>
              </collectionProp>
              <stringProp name="Assertion.custom_message"></stringProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">16</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
        </hashTree>
        <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Uniform Random Timer" enabled="true">
          <stringProp name="ConstantTimer.delay">1000</stringProp>
          <stringProp name="RandomTimer.range">4000</stringProp>
        </UniformRandomTimer>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Which one is easier to understand? If you had to modify the script to add a request, which format would you choose to do it in?

Scripting in XML makes merge conflicts more likely and breaking changes in automation pipelines way more difficult to pinpoint.

I’ve collaborated with others on JMeter scripts in the past, and we always opened up scripts in the GUI. It’s way easier to spot changes there — but that also created some new issues.

When you want something easy to maintain

Versioning for load testing

When ramping up load tests to more than one load generator, keeping track of the versions of necessary components ensures that one load generator’s traffic is the same as another’s. The value of reducing variables in scientific experiments holds true for load testing.

With k6, you just need to worry about the version of k6 and the script. No other dependencies are required. If you’re using plugins, you’d need to track those too, but as mentioned previously, plugins in k6 aren’t as necessary as they are in some other tools – and k6 is available as a Docker image to reduce those issues.

With something like JMeter, there are more levels to consider. There’s the version of JMeter itself, Java, versions of plugins, and then the script. Every new layer introduces more changes for incompatibilities. In previous projets I’ve worked on, someone would inevitably try out a new plugin and forget to discuss it with the team, and anyone who tried to open that script in their copy of JMeter would see this error:

The error means there’s a plugin that the script uses that JMeter doesn’t recognize, but depending on the plugin, it’s not always this easy to pinpoint which one (it was the Ultimate Thread Group in this case). You don’t want that happening in the middle of your load test because you forgot to add a plugin to some of your load generators!

It’s easier to link test code with application code

If your application code is in JavaScript, k6 is a shoo-in as a load testing tool. You could build tests that are tightly integrated with application code by importing objects and having your tests interacting with them directly. This approach would reduce a lot of time spent in refactoring test scripts when objects are changed.

If you’re already using VS Code, k6 even has a VS Code plugin.

Test as code gives you control

Having your scripts be pure code reduces ambiguities in that you’re not limited by poor UI design or bugs in third-party plugins. k6 scripts are Typescript-typed, so most IDEs support autocompletion while you’re scripting — the code equivalent of helpful tooltips, except way more universal. k6 scripts give you the freedom to import and use JS libraries, so you can build on code others have already written. While k6 is not NodeJS, there are even some NodeJS packages that you can bundle for use with your scripts.

No external dependencies

k6 doesn’t require NodeJS or anything else to run, which also reduces the likelihood of bugs that might affect your scripts. For example, since JMeter runs on Java, it is also dependent on Java. When Java receives a new version, that often requires a JMeter update as well. JMeter versions understandably lag a bit behind Java versions as developers refactor code, but this gap can lead to potential security issues in a previous version of Java that may remain unpatched in the meantime.

With k6, development can proceed independently of any third-party component, so major issues can be addressed more easily.

Comparison table: JMeter vs k6

CHART

Conclusion

The best advice, when attempting to choose a load testing tool, is to do a proof of concept with the most promising candidates. Some features or bugs may prove more or less important when used in an actual testing cycle. However, don’t put too much weight on comparisons of the load test results from two different tools — each tool records metrics differently, and it’s more meaningful to compare results against previous runs with the same tool. When switching tools, reestablish a baseline in the new tool each time.

Here’s a summary that might help you decide between JMeter and k6.

Both tools do well at:

• Generating protocol-level load on an application server by scripting complex user flows
• Realistic scripting with the use of dynamic think time, test data generation and reuse, and customizable workload models
• Documentation of features and consistency of releases

Neither tool supports:

• Detailed results analysis (JMeter has pre-generated HTML reports and listeners, but they are far from ideal). Users should expect to integrate results with databases and data visualization software.

JMeter is best for:

• Traditional testing teams
• Those looking for a GUI-driven testing tool with tons of third-party tutorials and extensive protocol support
• Previous users of commercial tools like LoadRunner and NeoLoad

k6 is best for:

• Collaborative, cross-functional engineering teams where testing spans acrosss multiple roles
• Those looking for a simple and lightweight, yet fully-featured load testing tool
• Teams looking to integrate testing into existing development workflows and CI/CD pipelines

Load testing scripting tools are by no means the most important consideration for the success of load testing. Knowing why you’re testing, what the requirements are, and understanding and communicating results are all arguably more important. The right tool will enable you to address those concerns while providing as little friction as possible. There’s no unequivocal “best” tool; there’s only the right tool for your project and context.

See also

• This article summarized in a video: JMeter vs k6: Comparing two popular open source load testing tool
• Load tests: Jmeter vs K6, Rafaela Azevedo.
• K6 vs. JMeter - Head2Head, Twan Koot.
• Olo delivers faster testing and improved collaboration
• k6 testimonials

Grafana Cloud is the easiest way to get started with Grafana k6 and performance testing. We have a generous forever-free tier and plans for every use case. Sign up for free now!