A look inside how the Prometheus Conformance Program works and why it's important

NB: This blog post is written with my Prometheus hat on.

Prometheus is the industry standard in cloud native metric monitoring with hundreds of thousands of installations, millions of users, and billions in market value.

Speaking as a member of the Prometheus team, we have seen the project become a victim of its own success. While most people may be using Prometheus, not everybody is following the same operating standards. Over the years, we’ve seen again and again that there is a ton of informal knowledge among projects and vendors about who is actually compatible with Prometheus and who might not be, but there is strong reluctance to engage in a public dispute. The result is that we have confusion in the user base about who is actually doing things the Prometheus way and who is just claiming to be.

To help end users better determine which tools are compatible with Prometheus, the Prometheus project with the help of CNCF has developed the Prometheus Conformance Program — a suite of tests to determine an offering’s interoperability with Prometheus that developers can run on their products or Prometheus maintainers can help execute. Going forward, the Prometheus project will publish and maintain a list of compatible products and services. CNCF will also assign a distinct mark that projects and vendors can use to show their compatibility.

Our intention as Prometheus maintainers is to help users choose Prometheus-friendly solutions with confidence while also guiding the community to adopt Prometheus protocols as they innovate and update their offerings. The following is a summary of my recent talk at KubeCon + CloudNativeCon North America 2021 about the program.

How the Prometheus Conformance Program works

Anyone can run the Prometheus conformance tests. 

The end score is calculated by multiplying all of the compliance scores that are relevant to you. Software is only deemed “Prometheus Compatible” if it reaches 100% with all relevant compliance tests.  

Conformance % = Compliance % multiplied with each other

Currently software and services are being put in the following categories:  

• Metrics Exposers
• Agents/Collectors
• Prometheus Storage Backends
• Full Prometheus Compatibility

Your compatibility score is based only on the Prometheus interfaces in your offering. If you don’t offer, for example, storage or querying, it would be unfair to require alert generation as part of your conformance testing. 

CNCF uses trademark law to grant the usage of a Prometheus Conformance Program logo that participants can showcase to promote that they have gone through testing and are approved. Stakeholders will sign a contract to ensure that they conducted good faith testing (i.e., no cheating) and that they agree to the terms of usage. 

The agreement also specifically outlines the definitions for three commonly used terms in the Prometheus Conformance Program so that there is a universal understanding of the terminology. (This will also prevent any “clever” interpretations of the terms in the future.)

• Conformance: name of the program 
• Compliance: measure of how well the software or service works with specific Prometheus specifications and interfaces
• Compatibility: score assigned at the conclusion of testing 

As an end user, there’s one phrase that has an indisputable meaning: “Prometheus Compatible,” which is a distinction that users can now look for when selecting products and services. 

Why the Prometheus Conformance Program is important  

The initial cadence for renewing a Prometheus Compatible status is aggressive: Compatibility must be tested every 12 weeks or after two minor releases, whichever is longer. As a result, as testing components or requirements change, some projects may even lose their mark of compatibility, which means they have to reimplement, implement, or fix something to retain or regain their distinction.  

The short cadence, however, is intentional. It is meant to incentivize a lot of people to keep each other honest and to move the complete Prometheus ecosystem forward.

Not to mention there’s a direct monetary incentive for people, for enterprises, and for managers to improve their own Prometheus compatibility and to keep the vendors they work with honest about their status, too. Because Prometheus is recognized as the industry standard, developers want to avoid any surprises as they invest in new technologies, and as businesses scale, they want to have the confidence that their monitoring solutions can take on the increasing load. 

By enforcing a fair and transparent universal testing model, the Prometheus team hopes that a more comprehensive testing suite will evolve to cover more potential failures and more incompatibilities, all while reinforcing the operating standards of the Prometheus community.

How we can help!

If you need help to convince your manager to buy into the program, talk to the Prometheus maintainers! We can certainly make a good case as to why you should be doing this. We can also help you with the test suite. We can help you with the design document. We can help you with everything! We structured the Prometheus Conformance Program as openly as humanly possible. 

Check out our design documents where you can contribute anything and everyone can give feedback no matter if they’re vendors, on the Prometheus team, or from somewhere else entirely. We want the community to shape this effort to create a fair, transparent, and consistent testing approach that ensures interoperability between different projects and vendors in the Prometheus monitoring space.