Inside Grafana Labs hackathons: how they work and what projects ended up on the product roadmap

Three times a year, Grafanistas around the world step away from their daily responsibilities for one week and put their creative energy into what has quickly become a cultural touchstone at Grafana Labs: Our company-wide hackathon. 

In just 8 months, Grafana Labs has hosted three hackathons that average about 130 participants and have produced more than 150 different projects, from exciting new features that have been added to our roadmap to answering life’s most pressing questions, such as “Can Grafana run Doom?” (Answer: Yes!)

In the latest episode of “Grafana’s Big Tent” — our new podcast about people, community, tech, and tools around observability — hosts Tom Wilkie and Mat Ryer (who was part of the winning team from Grafana Labs Hackathon 2.0!) are joined by Ash Mazhari, Grafana Labs VP of Corporate Development who leads the Grafana Labs hackathons. 

You’ll also hear from Grafana Labs Senior Principal Solutions Engineer Ward Bekker, whose team won the top prize at Grafana Labs Hackathon 1.0, and Grafana Labs VP of Applications Ryan McKinley, representing the winners from Grafana Labs Hackathon 3.0. Together they’ll bring you inside the competitive yet collaborative atmosphere of the Grafana Labs hackathon. 

Also if you want to check out more of our hackathon projects, you can watch the recent “Hackathon showcase” from GrafanaCONline 2022 on demand. 

Note: This transcript has been edited for length and clarity.

How hackathons work at Grafana Labs

Tom Wilkie: What is a hackathon at Grafana Labs? Because it can mean different things to different companies. 

Ash Mazhari: For us, it’s all about building cool stuff off vector. There’s always the top-down mandates of what you need to do for customer feedback, priorities and roadmap, and then where you think you need to go in the industry to stay ahead of competitors. But I think what’s cool about our hackathon is it’s completely open. You can build whatever you want and really leverage the expertise and knowledge that you have about our products or just tech in general and build something cool. … And it’s open to everybody, which I think is also somewhat unique.

Tom: Set the scene — How long do they run for? How does one take part?

Ash: Our hackathons run for about a week. We give people time away from their day job to go and build something on the clock. Because what we’ve found is a lot of times if you don’t do that, people just won’t take time out of their personal lives. So we thought it was important to carve out that time and give people that space to go and build and then also give them the flexibility to do that in an unbounded way. A lot of times you see hackathons that have a theme or a specific objective; we wanted to not have that so that it’s completely open ended. Folks go and build for a week, come back, present videos to the judges, and then we work our way to the finalists. And we present those at the [monthly] company all-hands, which is also really cool because it brings a certain visibility to the hackathon that’s company-wide.

Tom: Don’t people get to build cool stuff during their day job?

Ash: They do, especially at a company like Grafana Labs where we have a pretty cool open source mantra and a lot of interesting projects that people are excited about and passionate about. But in general there’s always that mandate of some broader mission that relates to the company objectives. The great thing about the hackathon is it takes all that away. So forget about whatever priorities there might be from customers and what not. Go do what you’ve always thought is exciting or interesting.

Ryan McKinley: To me, this is giving me a new kind of hope and optimism as we grow. I joined Grafana Labs as a very small company where everything was really engineering led. We were building stuff that we use. We know exactly what we want. As we’re getting bigger, we’re putting in new infrastructure, new systems, new teams that help manage that and make it less chaotic, which is great and required to be mature and stable. But we don’t want to lose what made us interesting. I work here because I want to build stuff. I think that’s true for a lot of us. So this has been great because I think it’s devoting real time for everyone to express themselves in the way we have from the beginning.

Mat Ryer: When I joined Grafana Labs, I got the sense that there is this history for engineers, and you really want to give people a chance to flex those muscles and take part in that. And I think the hackathon does that quite well.

Ryan: And to say engineers only, I think isn’t quite fair. Because the other value I see is that we’re given a chance to really work across the company. When you look at the projects that win, they’re always ones that involve someone across all of our disciplines, and I think giving people the chance to ignore all meetings for a week to ….well, we say don’t do your work. It’s never really true. There is always something else going on…. but you did have an excuse at times to cancel some meetings and that’s great.

Ward Bekker: Indeed. I’m a solutions engineer. So that means that I work with customers that are testing our software, and I can’t really say to the customer, “Oh, sorry, you can’t test our software right now. I’m busy.” We definitely take that into account with the planning. But, overall, I did spend some extra time [on my project], but it’s really nice to be able to work on something in a week with folks who believe that it is something worth sharing. And being able to present it to the whole company, that really energized me. That also made it very easy for me just to spend a little bit more time after my normal day job on this.

“We’re given a chance to really work across the company. When you look at the projects that win, they’re always ones that involve someone across all of our disciplines.”
— Ryan McKinley

Tom: One of the reasons we’re also super keen on hackathons is coming from that startup background, I think I do my best work when I’m given the freedom to pursue my own interest. The best open source projects come from scratching your own itch. I think giving the engineers freedom to explore other ideas that motivate them personally, to solve problems that they might experience on a day-to-day basis, does tend to lead to new and interesting ways of solving them. And you find things you don’t expect.

Ash: What’s really cool is that, especially here at Grafana Labs, I’ve noticed people are so talented and creative. You give people that space to go and do something that they’re passionate about and the kinds of things people have done so far have just blown me away. Just the fact that they can do a really cool project like that in a week’s time speaks to the people here as well.

To judge or not to judge?

Tom: Ash, you mentioned earlier that we judge the hackathons. We pick a winner. We’ve tried to gamify it. Why did we do that? 

Ash: You want to have a competitive spirit to these things. I think engineers, by nature, may be somewhat competitive.

Tom: Not just engineers, but humans … You know, engineers are humans too.

Ash: That’s true in most cases [laughs]. But nonetheless, I think having that judging and presenting were really important, because we wanted people to have a sense of the pitch and how they present that. And the judges needed to be in tune with how the projects are evaluated, how they could be important. And ultimately it came together in a really great way. … There was excitement and you could feel it in the pitches. That’s fueled a lot of the excitement about this because now people, when they work on this, they’re thinking about the pitch, they’re thinking about how they’re going win over the judges. 

Tom: [to the hackathon winners] What was your opinion of the judging aspect of this? Would you have done the hackathon even if it weren’t gamified?

Ward: Oh, that is a great question. I think yes, because for me it was just an opportunity to get something I wanted in Grafana for a long, long time already. For the folks who might not know, the project that we worked on was called Stonks ’n’ Crypto, and one of the key parts of it was a new visualization in Grafana to actually visualize stocks, because I was working on a pet project creating a foundation of algorithmic trading. I wanted to send the data to Grafana Loki and then visualize it in a candlestick diagram, which is quite standard in the stocks and crypto world. And there wasn’t any, so it was I wanted to make sure that this is there. [Note: The candlestick visualization was released in Grafana 8.3; below is an excerpt look at the team’s hackathon presentation.]

Mat: It’s funny, for me, the competitive side of it wasn’t the appeal. It was much more about the collaboration. In the last hackathon, somebody built something into the Grafana Incident tool, which is what my squad builds, and we took our time out of our day to help them do that. We wanted that to be a successful project as well, even if it competes with something that we are doing. So, although it’s got this competitive edge to it, I think the spirit in which everyone engages in [the hackathon] is very positive. There’s no sabotage there. Now that I’ve said it, there will be … 

Ryan: Going back to your question of whether the competition is what drives it. I’d say it’s nice. It’s good that it sort of puts real stakes on the table, but I don’t think it’s the main motivator for most people. I really think the main motivator is that everyone we’re working with has ideas, and the competition helps with visibility.

Tom: So here’s another controversial opinion. Part of what we do at Grafana Labs is we also give out prize money — and it’s quite a lot.

Ash: It’s four figures.

Tom: There’s going to be a lot of people hearing that going, “That is crazy, giving out money to win something you’re paid to do in the first place.” Do you think the prize money is a meaningful part of the hackathon?

Ryan: Definitely. We live in capitalism; this is the world we live in. So absolutely.

Tom: Mat, what did you do with your prize money?

Mat: I gave mine to Charity, which is the name of my wife. [laughs] Not really… 

Tom: I’ve met your wife, and she’s not called Charity. So does she know about Charity?

Mat: This is not public, is it? [laughs] Something tells me my wife is not going to be listening to Grafana Labs’ podcast….

Presentation vs. product

Mat: The final pitches are 5-minute videos that get played for the company. [Note: The entire company then votes and determines the winner.] So that’s how you pitch. But it isn’t just about making a video though, right?

Tom: No, but I think it’s important. One of the judging criteria is how well was this presented as a project or how good was the pitch? I distinctly recall how the quality of the pitches between the first and second and the third hackathons has drastically improved because people realize they’re being judged on this. It’s important to me and important to Grafana Labs that our team know how to communicate their ideas and know how to communicate them effectively. Honestly, Mat, one of the reasons you won the hackathon is because your pitch video was hilarious. It just had everyone in fits of giggles. It was just really well put together.

Ward: Presentation is really important, and we see an incredibly high level of quality there that it actually makes it harder for people that have great ideas, but don’t have the presentation skills to win here.

Ash: I think we want to try to help everybody who wants video production assistance to do that. So we’re thinking of getting a production crew contracted for that week and having open office hours. Then when people are creating their videos, they can get help on how to make [the video presentation] better. I don’t want it to be a distraction from the actual work and make it seem like all smoke and mirrors. But at the same time, I do think upleveling the pitches makes it more fun for everyone else as well.

Ryan: But interestingly, production quality is one thing, but it doesn’t really matter how flashy it is without a good script and content.

“Although it’s got this competitive edge to it, I think the spirit in which everyone engages in [the hackathon] is very positive. There’s no sabotage there. Now that I’ve said it, there will be …”
— Mat Ryer

Mat: To be fair, Ryan, your last pitch could have just been an empty screen that just said what it was going to do, and I would’ve still voted for that.

Ryan: Maybe it’s worth saying what the project was …? [laughs] So anyone who’s worked with Grafana for awhile has always figured out, “Hey, what’s your story with GitOps?” How do I actually manage what goes in and do version control. We’ve tried many awkward approaches in the past, and so as a hackathon team, we started over and just took a new approach to how to do GitOps and make it first class. So we spent a week prototyping it, showing how we could do it. Some of it was throwing out projects that are in the works, like things that people are really dedicated to saying, “Do we really need to do that approach?” And it has been really interesting to watch how things change as you get the visibility company-wide to say, this is important and it’s important to do it soon.

Mat: Another nice thing about this is that it’s a weeklong hackathon. So you have permission to cut corners and really focus on what matters, whether that’s the key bits in the storytelling, even by choosing what you build. So we did this project, which was basically dynamic runbooks. So given an alert, you change variables inside that and put graphs in and things so it’s like runbooks that are alive. One of the things we actually built was an editor for this, and that’s just a piece of the story. It was a tough choice to make. We decided to build the actual editor, where you are typing and writing your markdown runbook and then you see the preview on the other side with all the context added. So we skipped over other things that you might need. So that’s nice, too, being able to just focus on those bits.

Tom: What’s going to happen to that project?

Mat: That was another thing when we did it — we were thinking about whether we could design something that’s actually deliverable quickly. Something that doesn’t have too big a scope. What happens now? At some point we have to get that scheduled. We have to turn that into real work and put that into somewhere. 

Tom: But it is part of the roadmap now for your team and for their product … That’s the power of the demo, right? That’s the power of putting it together — now people have something concrete, they can aim for.

Ryan: One hundred percent. And enthusiasm and excitement to do something that’s real and meaningful instead of like another approach that may not actually go anywhere. There’s now momentum. This will happen. Get on board or …

Tom: There is no “or” — just get on board.

Don’t want to miss the latest episodes? You can now subscribe to “Grafana’s Big Tent” podcast on Apple Podcasts and Spotify.