Monitorama Preview: Observability Talks

The Monitorama Conference kicked off this week in Portland, Oregon, and Grafana Labs’ Tom Wilkie will be presenting his talk, “Grafana Loki: Like Prometheus, But for Logs,” tomorrow.

If you’re not at Monitorama, check out our recap blog post of Tom’s presentation on Loki at KubeCon, and watch the video of it here:

Yesterday at Monitorama, Dave Cadwallader, Site Reliability Architect at DNAnexus, reprised the popular talk on observability that he gave earlier this year at GrafanaCon, and added a new element: a beginner’s guide to Prometheus.

“This is about teaching Grafana to kids and what I learned in the process,” he said at GrafanaCon. “But it’s also about how we can be more empathetic in how we teach fellow grownups and how we can have more fun and hack our own brains to learn as fast as children can.”

In case you missed it, here’s a recap of that talk.

Cadwallader recounted that sometimes his young children would pass by when he was looking at a Grafana dashboard on his computer, and they’d be riveted by what was on his screen. He would usually be triaging an incident, and wouldn’t have time to answer their questions immediately. But he started wondering how he could explain monitoring to his 4- and 6-year-old kids.

He started by buying some inexpensive wireless temperature and humidity sensors, which communicate using a common radio frequency, 433 megahertz, that’s used by a lot of things in a typical home. “There’s this cool little $20 receiver that you can plug into your Raspberry Pi that will actually receive radio frequencies and turn them into digital signals that then you can pipe to a Linux process and do just about whatever you want with them,” he said. In this case, he used it to get the signals from the temperature sensors.

An Introduction to Grafana

Each sensor would broadcast its own unique sensor ID, which the family would record on the sensor or battery. “And every 30 seconds, these send out a little ping with their sensor ID, the temperature, and the humidity levels,” he said. “From there, all we had to do was send that data to InfluxDB and graph it with Grafana – and the kids had their first dashboard to play with.”

The first experiment he did with the kids was monitoring which rooms in the house were warmer or more humid. But “we live in Colorado, so pretty much everything is zero humidity, dry as a bone,” Cadwallader said. “So that wasn’t too exciting. But we started to learn about concepts like zooming in and zooming out and understanding the X and Y axis and plotting lines from those points.”

One room that they assumed would have a pretty constant temperature ended up having a sawtooth pattern. “So we used our eyes and our ears and we noticed that whenever the furnace would kick on, we’d have this little increase, and the furnace would turn off, and we’d have this little decrease," he said. “So that started to make sense. And this was another example for the kids about making a connection between the environment around them and the data that they observed on the graph.”

Looking at Historical Data

The next exercise was monitoring the family’s two pet cats, Hobbes and Rooibos. “We were worried that, you know, what if they were cold at night, and did these cat beds really work to keep them warm?” Cadwallader said. “So we thought, ‘Hey, we could actually use Grafana to answer this question.’”

They put sensors in the cat beds and graphed the two cats.

The next morning, Cadwallader went over the graph with his kids and asked them what they thought was happening that caused the temperature to go up or down. The verdict: “Rooibos, who’s the much fluffier cat, had a much steeper slope towards her peak bed temperature, and Hobbes, who’s the short hair, was a little bit slower to get there, but they eventually converged,” he said. “And we can see that Hobbes actually got up to have a late night snack and then got back in bed.”

And the lesson for the kids was how to look at historical data to piece together a story of what may have happened – “just like analyzing a production incident by looking at historical data,” he said. “The kids were pretty amazed to learn that this kind of thing is a big part of what I do at work all day. But now it was approachable. So after we had a week’s worth of data, we could zoom in and start to learn about trends, and we talked about how the two cat beds were correlated and followed the same shape a day after day.”

Sampling and Thresholds

The last experiment involved measuring the movement of a small trampoline that the kids bounced on. Cadwallader bought an ultrasonic range finder, which “measures the round trip of how long it takes a signal to bounce back off of something, and then you can send that to your Raspberry Pi,” he said. “The Raspberry Pi has these nice little pins called GPIO where you can plug in just about any sort of electronics. And so the kids helped me wire this up, and it was really fun.”

The first graph they created looked like a sawtooth (which his son Boden had predicted based on their first experiment). Playing with sampling and thresholds, “we figured out that if we took a sample every 50 milliseconds that it would filter out all of those noisy, tiny little peaks and valleys,” he said. “And it wound up with something that looked like this where we could count just peaks and valleys that were in between jumps.”

“I, as the teacher, got to have an experience of learning from my student,” he said, “and being like, ‘Oh yeah, that idea you had before, that’s totally something we could use now.’”

Lessons Learned

Cadwallader said that teaching his kids about monitoring gave him greater insight into how to work with his colleagues more effectively.

From watching his son play with Grafana, he said, he realized that “there’s not always a right or wrong way to do it. It’s more of an art than a science.” And sometimes it’s best to just let someone work through his or her own art of graphing.

Especially when they’re enjoying the process. “Sometimes in education or even at work, there’s extrinsic versus intrinsic motivation,” he explained. “Extrinsic is where you do something to get a reward. Intrinsic is where you do something just for the joy of doing it. Boden was so enamored of Grafana that he just wanted to use it for experiments of his own invention. So I thought, ‘Wouldn’t it be cool if we could bring this excitement back to work as well?’”

Cadwallader organized a game day at his office, during which “you get everybody in a room and you try to break stuff, and we elected a member of our team to be the chaos monkey and try to shut things down and kill processes,” he said.

For some people, it was their first real exposure to Grafana. “The joy on their faces when they made a prediction about what was going to happen when we messed something up, and then seeing it in Grafana was actually really, really cool,” he said. “And I got a lot of feedback from people like, ‘Hey, I learned so much more from doing this than I did just from reading the docs.’”

The takeaway? “When we have fun while we’re learning, we learn better.”

His overall advice: “Explain like they’re five, and learn like you’re five,” he said. “So you’re all the subject matter experts on Grafana and observability, and many of you are the go-to person in your org. But when you’re helping somebody do something with Grafana or observability, try to resist the temptation to do all the work for them just because it’s easier. Let them tinker, let them struggle. And if you can be there next to them, when they have that ‘a ha’ moment, when it clicks, you can share in that emotional learning moment.”