Grafana Labs Hackathon projects: Where are they now?

Here at Grafana Labs, we know the power of innovation. Grafana itself, afterall, started out as one developer’s dream before it grew into the ubiquitous data visualization tool that it is today. 

We also know, however, how important it is to foster that innovation across the organization, at any level — even if it means hitting pause on our day-to-day responsibilities. And this is the exact idea behind our internal Hackathons. 

Held three times a year, these week-long, company-wide events are the perfect opportunity for Grafana Labs teams to come together, set aside their daily work commitments, and focus exclusively on exploring new technologies. With these Hackathons, we aim to give all Grafanistas the freedom (and time) to experiment and think outside the box.

“These Hackathons are so important,” says Grafana Labs Co-founder Anthony Woods. “We want to make sure that there is always an avenue for talented people across the company, not just in R&D, to innovate.”

Time and time again, these Hackathons culminate in something truly special — not only for Hackathon participants, but for the entire Grafana community. Yes, we name finalists and crown internal winners after each Hackathon event. But these projects also play a significant role in shaping and driving our product roadmap. 

“The way we innovate at Grafana Labs is definitely unique and part of our culture,” Woods says. “Many of our products today started with one person, or a small group, building on an idea.”

In fact, of all the projects that we’ve seen across our last 8 Hackathons, almost 50% have either shipped, are on our roadmap, or are in progress. 

In this blog post, we want to take a minute to celebrate that. Here’s a look at three cutting-edge observability features within Grafana Cloud or the Grafana LGTM (Loki for logs, Grafana for visualization, Tempo for traces, Mimir for metrics) Stack that made their initial debut as a Hackathon project. 

1. Adaptive Metrics

Hackathon project name: Aggregator

Project lead: Mauro Stettler, Senior Software Engineer 

We’re constantly looking for ways to make observability as cost-effective as possible. Aggregator, a project born out of our December 2021 Hackathon, was very much aligned with that goal. 

The idea for project Aggregator was sparked by Mauro’s realization that aggregation, when applied to ingested time series metrics, could lead to significant cost savings. This is because, as metric cardinality grows in cloud native environments, so does the cost to store and retrieve the data. So, using open source technologies, Mauro prototyped an aggregation concept to help users lower their metric cardinality and, therefore, their costs. 

Ultimately, project Aggregator evolved into Adaptive Metrics, a feature in Grafana Cloud that enables teams to aggregate unused and partially used metrics into lower cardinality versions of themselves. Adaptive Metrics analyzes usage patterns and, based on those patterns, recommends which metrics to aggregate, helping users cut costs without compromising on their observability strategy.

Adaptive Metrics became generally available in all tiers of Grafana Cloud in November 2023 — and users quickly saw the benefits. Identity security provider SailPoint, as one example, was able to cut its metrics volume by 33% through a combination of internal efforts and their adoption of Adaptive Metrics.

“Adaptive Metrics really helps us to grow efficiently from here on out, without just blowing up our metrics and our costs,” says Omar Lopez, head of the observability team at SailPoint. “It’s given us more control over our metrics.”

Given how effective Adaptive Metrics has been at helping our users manage and reduce their observability costs, we’re planning to apply this powerful concept in other areas. For example, as noted at ObservabilityCON 2023, we’re currently working on Adaptive Logs — so stay tuned there.

To learn more about Adaptive Metrics, check out our documentation and our webinar, “Grafana Cloud Adaptive Metrics: Reduce Prometheus high cardinality metrics.” 

2. Distributed Tracing in Grafana Cloud k6

Hackathon project name: CrocoSpans  

Project leads: Joe Elliott, Principal Engineer; Daniel González, Senior Technical Program Manager; and Mario Rodríguez, Senior Software Engineer

In June 2021, a team of Grafana Labs engineers set out to solve a problem: a lack of system visibility during load testing. They essentially wanted to break down the wall that separated what a performance testing tool sees during a test and what happens internally to the systems being tested. In traditional performance testing scenarios, this disconnect often made it challenging for engineering teams to fully understand test results and, especially, to act on them if something went wrong. 

The solution? CrocoSpans, a Hackathon project that incorporated tracing data into load testing to increase visibility and enable faster troubleshooting of failed performance tests.

Today, this feature is known as Distributed Tracing in Grafana k6, and it became generally available in September 2023. With Distributed Tracing in Grafana k6, engineering teams can correlate performance testing and distributed tracing to reduce MTTR when a performance issue surfaces — and, as a result, proactively improve reliability. To date, there have been more than 5,000 tests run with this integration enabled.

For full implementation details and best practices, see our Integration with Grafana Cloud Traces documentation.

3. Structured metadata

Hackathon project name: LILI 

Project leads: Sandeep Sukhani, Senior Software Engineer; Ned Andreev, Principal Software Engineer; Salva Corts Sánchez, Software Engineer II; and Vladyslav Diachenko, Senior Software Engineer 

In March 2023, the Hackathon team behind project LILI was looking for a way to optimize needle-in-the-haystack queries in Grafana Loki, our open source, horizontally scalable, highly available, multi-tenant log aggregation system. In that pursuit, they discovered something big: a better way to manage high cardinality data in Loki that would significantly improve query performance. 

Some metadata — especially infrastructure-related metadata — is difficult to embed in log lines and is too high cardinality to store as indexed labels without compromising performance. The team’s work on project LILI inspired a new Loki feature to address this challenge: structured metadata. 

Structured metadata is a way to attach metadata to logs without indexing them or including them in the log line content itself. This lets users keep long-lived streams with low cardinality labels, while also having a place to store high cardinality data, which is necessary for filtering and grouping at query time. Structured metadata is now generally available in both Grafana Loki and Grafana Cloud Logs.

Notably, project LILI also helped pave the way for optimizing needle-in-the-haystack queries using Bloom filters, a feature that is currently in the works and should be out soon with the Grafana Loki 3.0 release.

To learn more about structured metadata, you can check out our technical documentation and our “Concise guide to Loki” blog series. 

And if you’re interested in learning more about Grafana Labs’ culture of innovation, more broadly, check out our new four-part documentary series, “The Story of Grafana.”