Grafana Labs update regarding CircleCI security updates

Update as of January 13, 2023: Instructions updated to avoid errors on some OSes/platforms. GPG key file updated to contain only the current key (the old file contained both a revocation certificate for the old key, and the current key). A more detailed description of these changes can be found at the end of this post.

On January 4, CircleCI published a security update on their blog asking all customers on their platform to rotate secrets. While CircleCI is no longer a part of our CI/CD pipeline, we have made use of their services in the past.  

As a best practice, we have rotated or invalidated any secrets that may have been present on CircleCI. Following a thorough review, we have not identified any signs of suspicious activity and we believe, with a high level of confidence, that none of our systems or code are or have been at risk of compromise.

In our review, we identified two GPG keys used to sign a subset of binaries and Helm charts. Best practice recommendations for these follow below.

GPG key rotation and trust relationship

Who is affected:

• Users who have installed Grafana via our package repositories (packages.grafana.com, apt.grafana.com, rpm.grafana.com)
• Users who use our binary .deb or .rpm releases and validate the GPG signatures with a local copy of the impacted GPG key

Who is not affected:

• Users who have installed Grafana via their OS package repository
• Users who have installed Grafana via a Docker image
• Grafana Cloud / Hosted users

If you are affected, you should follow the instructions below to remove the impacted GPG key from your keyrings and install our new key.

How to confirm if your GPG trust relationship is configured correctly

Debian/Ubuntu

The following command should return ‘0’:

apt-key list | grep -i grafana | wc -l

If the above command returned anything except ‘0’, remove the old key:

Copy

# Remove the old key
apt-key del 4E40DDF6D76E284A4A6780E48C8C34C524098CB6

# Verify that the GPG key was removed. This should return 0.
apt-key list | grep -i grafana | wc -l

Regardless of whether the above command returned ‘0’ or not, you will need to configure the new key as well:

Copy

# Download the new key and place it in the right location
sudo wget -q -O /etc/apt/keyrings/grafana.key https://apt.grafana.com/gpg.key

# Replace the apt source
# This command will replace the contents of the grafana.list file.
echo "deb [signed-by=/etc/apt/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

# Update your repositories list
sudo apt update

rpm (RHEL, CentOS, Fedora, etc.)

Our rpm-based installation instructions do not add the identified GPG key to the package manager, but rather suggest downloading the .rpm and gpg key and performing a manual verification. If you have saved a local copy of the Grafana gpg key (fingerprint: 4E40 DDF6 D76E 284A 4A67 80E4 8C8C 34C5 2409 8CB6), you will need to replace it with the rotated key, which can be downloaded at https://rpm.grafana.com/gpg.key.

yum and dnf (RHEL, CentOS, Fedora, etc)

yum and dnf cache repository gpg keys. In order to force yum or dnf to fetch and use the rotated gpg key, you will need to flush this cache. Instructions are as follows:

1. Locate your yum cache directory. This will default to /var/cache/dnf for systems running dnf, but may be at /var/lib/yum/repos/ or /etc/yum/repos.d on systems running original yum. If you don’t see either of these, check your yum.conf file for a persistdir parameter. This parameter overrides the default and specifies a cache directory manually.
2. Locate the cache for the Grafana repository inside your yum cache directory. On dnf systems this should be something like /var/cache/dnf/grafana-369737ffb0934a7b/. On yum systems this will likely be an architecture-specific subdirectory under /var/lib/yum/repos/.
3. Remove the identified GPG key from the cached keyring. Inside the Grafana repository cache directory, there will either be a pubring folder or a gpgdir and gpgdir-ro folders. These are where the keyrings are actually cached. To remove the identified GPG key from the cached keyring, use the following gpg command:

Copy

gpg --batch --homedir YOUR_PATH_GOES_HERE --delete-key 4E40DDF6D76E284A4A6780E48C8C34C524098CB6

If you have both a gpgdir and a gpgdir-ro directory, run this command on both of them.

4. Run dnf update or yum update (depending on which you have installed). You should be prompted more than once to accept a new GPG key for the Grafana repository with fingerprint

Copy

0E22 EB88 E39E 1227 7A77 60AE 9E43 9B10 2CF3 C0C6

GPG revocation - Helm chart signing

A GPG signing key used for signing Helm charts was also identified in our CircleCI account. After some discussion, rather than replacing this with a new key we have instead decided to no longer sign Helm charts. Our data indicates Helm chart provenance is not commonly used and doesn’t provide sufficient value to offset the risks and complications of managing a GPG key used for such purpose.

If you were previously using Helm chart provenance, the below GPG key can no longer be trusted and must be removed from any GPG keyrings:

Copy

6737 B033 4836 C334 9DFF  2DD4 7054 A955 9D3C FB0B

Changes from original post

Some users were reporting issues with the instructions as originally posted not working on their systems– most of these issues centered around one of two things: the format of the GPG key fingerprint provided to apt-key and gpg commands (the original instructions had a fingerprint with spaces in it, which caused problems on some systems), and the format of the gpg.key file served by our package repositores (it contained both a revocation certificate for the old key and the new key, which some package managers did not process correctly). We have updated the gpg.key file served to only contain the new public key.

If you would prefer to load the full set of keys (the old key, followed by a revocation certificate for it, followed by the new key) a file containing all three can be found at https://apt.grafana.com/gpg-full.key. As mentioned above, many package managers have idiosyncratic behavior when loading key files containing revocations, so please consult your package manager documentation for specific guidance.

Reporting security issues

If you think you have found a security vulnerability, please send a report to security@grafana.com. This address can be used for all of Grafana Labs’ open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer that you encrypt your message to us by using our PGP key. The key fingerprint is

Copy

225E 6A9B BB15 A37E 95EB 6312 C66A 51CC B44C 27E0

The key is available from keyserver.ubuntu.com.

Security announcements

We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes.

You can also subscribe to our RSS feed.