Grafana security update: Post-incident review and timeline for GPG signing key rotation
On Aug. 24, 2023, we reported that the GPG private key and passphrase that we use for signing packages was unintentionally shared. As a best practice, we immediately revoked the exposed certificate on our full GPG public keychain and issued a new public key. We then advised users of our apt and yum repositories to update their trusted certificate to safely continue using our repositories.
Erring on the side of speed and transparency, we announced this as soon as we could, during the key change.
We have now finished our post-incident review and wanted to share this update.
Summary
On Aug. 24, 2023, the GPG private key and passphrase that we use for signing packages was exposed in our CI logs. As part of a reproducible-build overhaul, we made a bad assumption about how our CI service handles obfuscating secrets and subshells.
Unfortunately the key was logged in clear text, so we began the process of rotating and re-signing our packages. The only other occurrence we found in our extensive logs of this key exposure was on Aug. 1 before it was identified.
How the incident happened
Signing keys are not stored in CI/CD; however, for some pipelines, we do currently require the key to be available at build time. In a small number of legacy CI pipelines, we currently present signing keys for signing. We have already been working towards updating our build mechanisms to remove this dependency, but the change had not been rolled out to our Grafana builds at the time of the leak.
It is important to clarify that the keys were not exposed due to any kind of privilege escalation. At no time did third parties have access. We did not identify any discrepancy in artifacts distributed through our secure channels — our download page, apt repository, or rpm repository. We believe that our builds are correct and integrity has been preserved at all times. The key leakage was discovered by internal processes, and the leaked key was not trivially discoverable, having been written to only a small number of log files.
Impact
The risk of this key being used for a practical attack is limited, as an attacker would still need a way to get a malicious package to users. Due to the fact that apt.grafana.com and rpm.grafana.com both serve packages over TLS (and our installation instructions specify that the repos be added as TLS URLs), a successful attack would require either a man-in-the-middle with an additional compromised TLS certificate, or for a user to retrieve the packages from an alternative source, such as a local mirror. Due to internal validation and reconciliation mechanisms, we are confident that the integrity of artifacts deployed to Grafana Cloud and released to our partners were not impacted.
Action items
We are taking further action to decouple the signing process from package creation. After this work is completed, CI pipelines will no longer have access to private key material at all — only the central signing service will. We expect this work to be completed by the end of September. The majority of the code that is used for creating Grafana packages is available at https://github.com/grafana/grafana-build.
Timeline and post-incident review
Based on our completed post-incident review, here is the detailed timeline starting from when we originally learned of the issue. All times are in UTC.
- 2023-08-01 15:21 UTC - Key initially leaked to logs
- 2023-08-24 15:33 UTC - Key discovered by Grafana engineer in logs
- 2023-08-24 15:37 UTC - Incident declared
- 2023-08-24 15:47 UTC - Relevant Drone repos set
internalto prevent further leakage. PR raised to correct underlying behavior - 2023-08-24 26:50 UTC - Key rotated and published
- 2023-08-24 18:50 UTC - Initial blog published
- 2023-08-31 16:00 UTC - Updated initial blog post to state we will publish post-incident review and incident timeline
- 2023-09-06 14:39 UTC - Post-incident review complete
- 2023-09-06 16:00 UTC - Post-incident review and timeline published on the blog
Reporting security issues
If you think you have found a security vulnerability, please send a report to security@grafana.com. This address can be used for all of Grafana Labs’ open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer that you encrypt your message to us by using our PGP key. The key fingerprint is
225E 6A9B BB15 A37E 95EB 6312 C66A 51CC B44C 27E0
The key is available from keyserver.ubuntu.com.
Security announcements
We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes.
You can also subscribe to our RSS feed.
