From fish tanks to data banks: finding Grafana on a farm

I started a fish farm because I realized that the three marine biologists in the world weren’t giving up their jobs. 

It was 2014 and I had just finished a degree in marine ecology, so it seemed like a good idea in my head. Of course, actually building and operating a fish farm — technically, an aquaponics farm — turned out to me much harder than I ever expected. 

This all took place in another (work) life, long before I joined Grafana Labs as a senior solutions engineer, but not before I became familiar with Grafana. In fact, Grafana’s monitoring capabilities played an important role in how I managed and improved my farming techniques. 

But I’m getting ahead of myself. First, I want to tell you what I tell you from experiencing the highs and lows of building a farm and how that ultimately led me to use Grafana. I want to explain how many of the lessons I learned as a farmer have helped me in my time as an engineer, and how it’s come full circle here at Grafana Labs. 

Aquaponics 101

Aquaponics is a portmanteau of aquaculture (the farming of fish) and hydroponics (growing vegetables in a liquid medium). The idea is that the fish happily swim and poop in a tank, and then an enterprising farmer will take that water and perform a little filtering, fermenting, and refining. The water then goes through to containers where the vegetables can absorb the nutrients from the liquid. This does two things: It fertilizes the plants, and it cleans the water. You can then pipe the clean water back to the fish, resulting in an elegant symbiotic system. 

At least, that’s the idea on paper. The reality is a hodgepodge of pipes and plumbing, welding, ankle-deep fish excrement, broken pumps and aerators, dead fish, dead plants, high-voltage electric shocks, so many snakes, and a realization that farming is far more difficult than just watching grass grow.

I began the project in the Western Cape, South Africa, and named the company Afriponics. This region is a hot, dry, and windy part of the world that stretches from Cape Town to the Namib desert. Water is scarce and the farming here is mostly limited to sheep. The aquaponics project would need a large amount of water to start, but once it was up and running the water loss would be minimal across the entire system — a bonus in a water-stressed area. 

Burning Man, geodesic domes, and tilapia 

The first challenge I had to overcome was how to house the tanks and grow areas. Most greenhouses or agricultural tunnels are square or rectangular, but I had round fish tanks, approximately 32 feet (~10 meters) in diameter. I didn’t want to waste space so I looked for round structures. There were none to be found. 

Square tanks were a no-go because of the flow characteristics — the water would stagnate in the corners and debris would build up. So I turned to Burning Man — obviously. Though I’d never been, I’d seen pictures of geodesic domes at the annual festival, and it became the inspiration for my aquaponics space.

I built a production line where I cut and drilled several hundred meters of steel tubing to build a round home for my round tanks. These tanks would house the Nile tilapia (Oreochromis niloticus) we were growing.

The greenhouse tunnels came next. The troughs for the plants to grow in were pretty simple — plastic-lined with a board of styrofoam floating on top. This allowed the roots to dangle in the nutrient-rich solution flowing through the system. 

I quickly realized that to attempt a project like this, I needed to be somewhat proficient in a bunch of different skills: plumbing, welding, electrical engineering, tree-felling, and construction — not to mention the actual farming stuff. But I was bullish and headstrong, so I thought I could do it all. And honestly, it was a great way to learn; it was a time in my life where I developed skills that still serve me to this day. And I figured the bits I couldn’t do would be augmented by the occasional YouTube tutorial. 

What could go wrong? Quite a lot it turns out. 

‘All the fish are dead’

One morning, about three months into full production, I got a call from my foreman, Wellos: “All the fish are dead.” I raced back to the farm and walked into a fish apocalypse. An air blower had failed in the night and the oxygen saturation of the water slowly dropped until it couldn’t sustain the number of fish in the tanks. That was a very long day. I felt guilty that I had failed in my role as custodian of these animals, frustrated and angry that I hadn’t foreseen this happening, and ashamed that I had failed as a novice farmer. 

I turned introspective: Why did this happen? I had a very basic system of electronic controllers based on timers and float switches. I thought simple equaled safe. I thought that even if something did go wrong, then there would always be enough time to fix it. Surely, it wasn’t possible that everything could go wrong at the same time. I was obviously wrong. 

In South Africa, a lot of problems are solved by just adding more people to the equation. Advice I received was always along the lines of, “Just get someone to watch it all the time.” That didn’t feel right though. There had to be another way. 

I slept in an uninsulated shipping container — yes, an actual shipping container, not to be confused with the containers we’re more accustomed to in tech — right next to the tanks until I got it sorted, doing patrols of the farm every two hours. Eventually, I didn’t even need an alarm to go check if something was wrong; I could hear the change in pitch of an air blower or a pump stopping when it shouldn’t, and that would wake me in a panic. I still dream of those sounds.

I devoted a large part of my day to fixing my mistakes. There were off-the-shelf solutions for automating pumps and other elements, but I was pretty much broke at that point. I had to build my own. 

I had zero knowledge of IoT at this point, and very little coding knowledge, but I was fairly computer literate. I sat and I learned. But wow, that was a steep learning curve. Still, after all that, I finally had a working device, a small Arduino board to turn a pump on and off, and it changed my world.

I went mad with power. Every single piece of machinery on the farm got a microcontroller. Every tank and trough got multiple probes, checking parameters like salinity, pH, temperature, dissolved solids, O2, and nitrogen content. Pumps had multiple in-line flow meters installed. Water levels were checked with float switches and double-checked with ultrasonic sensors. (In hindsight, getting sub-millimeter water levels was perhaps a little bit too much resolution, but to paraphrase Jeff Goldblum in “Jurassic Park,” I was only preoccupied with whether I could, and didn’t stop to think if I should.)

Learning to code on a fish farm, and visualizing data with Grafana

I spent hours each day learning and coding these little devices, networking them together, and pulling the data out. Then I spent hours each night with my laptop balanced on the rim of a fish tank trying to debug what I had done wrong during the day. 

The boards I used were Arduino and Raspberry Pis, as well as a variety of off-the-shelf sensors for the environmental parameters. Over time a control and monitoring system evolved on the farm. I wrote an iOS app for my phone that alerted me if there was an error anywhere. (No more night patrols!) 

I used PubNub to query all the boards and then dump that data to a PostgreSQL database running on a central server (an old laptop in, again, a shipping container). At first I would just print this data to a terminal but I stumbled across a visualization tool called Grafana, and it was like walking out of the dark cave and into the light. I could see my data!

And this wasn’t just about making things look pretty. Because I could visualize the data from multiple sources from around the farm in one dashboard, it helped me see patterns that I hadn’t even noticed before. It challenged assumptions I’d made about the relationship between certain variables and parameters because now I could see more than one data series, not just a wall of numbers printed out to the terminal. 

How does pump velocity affect the dissolved oxygen in a tank? Does the pH change with temperature? These were questions I could answer quickly and easily with Grafana. I finally had visibility into the entire system, and I could explore what was happening within each data stream. There was a direct correlation between visualizing my data in a cohesive manner and improving how I farmed. 

Moving on and coming back

After about five years, I left farming to scratch the coding itch. I sold the farm; it was producing fish and vegetables, employing about 10 people, and just about making money. I moved to London and started working as a software engineer. 

And now, in one of those weird things that life does, I’m working at Grafana Labs, the same company that was so instrumental in the farm being successful. The two experiences are symbiotic, much like the aquaponics system. I use the skills I learned on the farm in my everyday life at Grafana Labs, and not just around software development. 

I’ve learned how future planning can only work if you know your current state, and how you can only know your current state if you build the systems to provide you with that data. And how all systems tend towards entropy, so we need to monitor them to prevent or fix that. Also, there’s things like the power of collaboration (you’ll notice I didn’t mention a lot of other people above, that’s a bad thing), asking for help when you need it, and welding. 

But as much as I’m enjoying my time here, I will be going back to farming one day. We need to change the system by which we grow our food, especially in places where food is a scarce and uncertain resource. I will be building more domes as they are mesmerizing structures and very, very cool. And I’ll be bringing my Grafana skills with me.

Learn more about the Grafana Labs team and our remote-first culture, and find out how you can become an official Grafanista by visiting our careers page.