Grafana Hackathon 2021: Meet The Alchemists

At the end of June, Grafana Labs hosted our first company-wide hackathon: a week where Grafanistas from all over the org joined forces to invent, build, and solve challenging technical problems related to our products, community, and business. 

The final projects ranged from a live stocks and crypto monitoring platform (which was named the overall winner and dubbed the Best Pitch) to a feature that allows Grafana users within an organization to share recordings of your favorite dashboards, panels, and interactions. 

The trio known as The Alchemists took home the prize for Best Integration/Plugin. Here’s a look at how their project came together.

The elevator pitch

We were looking to leverage the existing Arrow-based data frames to create a unified querying interface for all Grafana data sources using the Arrow DataFusion project.

The team members

Stephanie Closson, Software Engineer 
Todd Treece, Software Engineer 
Travis Patterson, Senior Software Engineer 

Their hackathon experience

Where did the project idea come from?

Todd: Justin Hunthrop, a Senior Engineering Manager at Grafana Labs, was discussing some ideas for creating a unified query interface for data sources. Query interfaces are currently created to match the capabilities of each data source, so there is a lot of maintenance involved. One of the ideas was to ingest data so that we could build a query interface around a common query language like SQL. Our project was based on this same approach, but instead of ingesting and storing data, we used Arrow DataFusion to query results in real time as they were returned by the existing data sources.

Travis: I was in a similar situation as Todd. I talked with Justin about how a unified interface for querying data sources could be a huge help, and he put me in touch with Todd.

Stephanie: Todd showed me what he was doing, and I thought it was really innovative. I’ve been interested in improving query editors and thought doing the front end for this project would be very interesting.

Have you done a hackathon before?

Stephanie: Not quite like this one in terms of its scale! [Note: There were 125 participants or 25% of the company at that time.]

Todd: I’ve participated in one before at a previous job, but due to other responsibilities, I think I was only able to contribute a few hours to that project. 

Travis: No!

How did this team come together?

Todd: Justin Hunthrop mentioned that Travis was also thinking about this same issue. Travis and I were still fairly new to Grafana, so I reached out to Stephanie to see if she would lend her data source expertise to the team.

Had you worked with each other before or met before?

Todd: We all work on the Grafana Enterprise plugins team.

Tell us about the division of labor. Did anyone have any surprising skills or expertise that they brought to the table?

Todd: Stephanie worked on a unified query interface. I worked on creating a DataFusion data source query proxy that was compatible with Grafana’s plugin interface. Travis worked on parsing the serialized expression tree format we used for DataFusion’s push down filters and also created several data source backend connectors.

Any particularly funny or memorable moments from the week?

Todd: Although this project hit a roadblock with the complexity around querying column-oriented datastores in a relational way, this was one of the most interesting projects I had worked on in years. I learned a lot more about how plugins work from talking with Stephanie, and Travis really helped push the project forward by breaking a proof of concept down into testable components that we could build upon.

Travis: The most memorable part of this project for me was seeing just how complex the space is. It’s one thing to say, “we need a unified querying interface,” and it’s something else altogether to take a shot at implementing it.

What’s the biggest thing you learned during the hackathon?

Todd: Some data sources only return aggregated data, and that doesn’t really align with our project’s approach. My main goal for the week was to see if this approach would hit any roadblocks, and we reached that goal. But I think the biggest thing I learned was that for a project this complex, spending the extra time at the beginning making sure everyone is starting with the same context is critical. It seemed like we needed to dig in and start coding right away to get things done by the end of the week. With hindsight, it would have been better to spend additional time discussing the problems we were trying to solve.

Travis: Unifying querying across heterogeneous sources and shapes of data is hard.

What was the best part of the hackathon?

Todd: Watching the first round of presentations was amazing. Everyone put in a ton of work, and there wasn’t a single team that failed to impress me.

Travis: I agree — seeing everyone’s work come together was amazing.

Stephanie: I really liked getting to know both Todd and Travis. I had not worked with either of them for very long before and so it was nice to see their enthusiasm and passion come out for the project.

Grafana Labs is hiring! We’re a fully remote team spread across the globe, in 40+ countries on 6 continents. Check out our job openings here.