Grafana Agent 0.20.1 and 0.21.2 released with security fixes

Note: We released fixes for CVE-2021-41090 and CVE-2021-43798 within 24 hours and mixed them up in one of the three blog posts. To make it clear: CVE-2021-41090 is for the Grafana Agent and CVE-2021-43798 is for Grafana the software. Only CVE-2021-43798 was a 0day exploit.

Grafana Labs internally identified a security vulnerability in the Grafana Agent. This vulnerability has been assigned CVE-2021-41090 and has a CVSS score of 7.2 (high) and can be found on Github. 

We haven’t found any evidence that this vulnerability has been exploited, but out of an abundance of caution, we want to notify all users so they can take action. 

Inline Secrets Exposure (CVE-2021-41090)

Summary

On May 24, 2021, Grafana Agent v0.14.0 added a new/-/config endpoint to its API that returns its runtime config. This release revealed an issue where some inline secrets used for configuring the agent would be exposed in plaintext over both the /-/config endpoint and, for users of the scraping service, /agent/api/v1/configs/{name}.

An “inline” secret is an API key or password which is directly stored into the config YAML. This means that file-based secrets, such as password_file, are not impacted by this vulnerability. The following secrets are known to be exposed:

• Inline secrets for metrics instance configs in the base YAML file are exposed at /-/config
• Inline secrets for integrations are exposed at /-/config
• Inline secrets for Consul ACL tokens and ETCD basic auth when configured for the scraping service at /-/config
• Inline secrets for the Kafka receiver for OpenTelemetry-Collector tracing at /-/config
• Inline secrets for metrics instance configs loaded from the scraping service are exposed at /agent/api/v1/configs/{name}

Impacted secrets are not limited to Grafana Cloud or Enterprise API keys, but any secret used. For example, if running an integration that uses credentials to connect to a database, these credentials are also improperly exposed.

We released Grafana Agent v0.20.1 and v0.21.2 on 2021-12-08 to resolve this vulnerability, which will correctly obscure the secrets listed above when returning YAML configs. As a precautionary measure, these security patches also disable the /-/config and /agent/api/v1/configs/{name} endpoints by default. Start the agent with the --config.enable-read-api to re-enable them after patching.

Solutions and mitigations

We strongly recommend that all users upgrade, even if not currently impacted by this vulnerability. 

Note that the Grafana Agent listens on all network interfaces by default. Depending on firewall rules and the networks Grafana Agent is exposed to, we also highly recommend changing any impacted secrets.

Please refer to the following flowchart to help determine what actions you should take: 

As a general security recommendation, we recommend only using secrets that contain the minimum set of permissions needed for the Grafana Agent to operate. 

If you wish to avoid downtime when rolling keys, you should create new API keys and update your Grafana Agent configs before deleting the old impacted key. 

Grafana Cloud

1. Log in to your Grafana.com account and access the Grafana Cloud Portal (grafana.com/orgs/“yourorgname”). This is different than your .grafana.net URL.
2. In the Portal, on the left-hand nav menu look for the “Security” section and click on “API Keys.”
3. On the API Keys page, delete any key which was improperly exposed through this vulnerability. If you have used the onboarding walkthrough to send metrics to Grafana Cloud, this includes the automatically generated <“org name”>-easystart-prom-publisher key.
4. Recreate your API Keys.

   • If you created your own keys and agent configs

     • On the API Keys page, click on “+ Add API Key” to create a new Grafana Cloud API key 

       1. Give the key a name (e.g., “grafana-agent-key”)
       2. Assign a Role of type “MetricsPublisher”

   • If you used one of our integrations via the onboarding walkthrough to send metrics to Grafana Cloud

     • Install a new integration or navigate to an existing integration to fetch a new agent config with a new automatically generated key.
5. Update your Grafana Agent configs to use the new keys that were created.
6. Repeat the API key creation and deletion process for each affected org. You can select a different org you belong to from the Cloud Portal, via the dropdown on the top left-hand corner of the page.

Grafana Enterprise Metrics

1. Using the GEM Grafana Plugin or manually using Admin API, for each of the affected API keys, create a new parallel API key for the same Access Policy which should already have metrics:write permissions.
2. Update your Grafana Agent configs to use the new API keys.
3. Delete old impacted API keys.

Integrations

Please see the documentation for the respective integrations you are using for the list of minimum required permissions:

• MySQL
• Github
• Kafka
• MongoDB
• PostgresQL
• Redis 

Please consult the documentation for each software to remove old credentials that were impacted.

Next steps 

Stay tuned for a post-mortem for this incident alongside our plans to prevent similar issues going forward.

Reporting security issues

If you think you have found a security vulnerability, please send a report to security@grafana.com. This address can be used for all of Grafana Labs’ open source and commercial products (including, but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer that you encrypt your message to us by using our PGP key. The key fingerprint is

F988 7BEA 027A 049F AE8E 5CAA D125 8932 BE24 C5CA

The key is available from keyserver.ubuntu.com.

Security announcements

We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes.

You can also subscribe to our RSS feed.