Turn mistakes into wins: How our no-blame culture works at Grafana Labs

Malcolm joined Grafana Labs in 2018 and now works on release processes for Grafana. He is also a founding member of Grafana’s Wellbeing ERG (employee resource group), which organizes activities such as mindfulness courses, book clubs, and lightning talks for all Grafanistas.

I’ve recently joined a new team at Grafana Labs, one that is focused on releasing new versions of Grafana. With so many security issues being found recently, this has made this work pretty full on.

In the course of the last few weeks, I’ve made some mistakes that have had quite visible consequences. First, I didn’t sufficiently prioritize the migration of a legacy service onto more modern hardware, and as a consequence, the service went down and we had to migrate under “emergency” circumstances, with the service being down for about 8-9 hours. Fortunately, it isn’t a central service, but still, decisions I made had an impact on some Grafana users, as well as on those colleagues that were sucked into the “incident” to complete the migration and recover the service.

Last month, a colleague and I built packages for the 7.5.14 private Grafana release. This was sent out to our Grafana Enterprise customers. But just as we were about to announce this to the world, I spotted an error. The package was supposed to include three fixes, but only included one! For a short few moments, I mulled on whether I could get away with hiding this. Surely no one would notice that I had messed up when making these packages?

But I only had to imagine a customer succumbing to a security incident because the package they installed didn’t include a fix that they thought it did, and I realized I had to come clean. I declared the issue to my colleagues, and collectively we set about making 7.5.15, which did include the fixes, and announcing this release to customers. This impacted many people who were waiting on the next patch release of Grafana, which now needed to wait an additional week  — all this because of my original mistake.

Now, the curious thing about these events: At no point have I heard any colleagues attempting to attribute fault, nor even discussing who was at fault. During a postmortem meeting, we discussed what we can do to prevent a similar incident from occurring again, but I did not hear anyone saying, “Malcolm messed up!”

That it happened like this was no accident. It is an example of the conscious “no-blame” culture in action at Grafana Labs. It is accepted that humans are fallible and will make mistakes. Therefore, it’s better to build systems that can handle humans making mistakes, rather than berating people for not being perfect. I, for one, have been extremely grateful for this culture over the last few weeks.

But there is more to it. This blame-free culture can also have positive implications for us personally. I may be lucky enough to operate within such a company culture, but what about within myself? My colleagues might not be blaming me, but am I doing that to myself?

Psychologically, there are valid reasons for the adoption of a no-blame culture. When we hear, “you did that wrong,” our natural response is likely to protect ourselves. It is as if we are under attack. Our sympathetic nervous system is triggered; we start to explore fight vs. flight vs. freeze. Whether this is extreme or subtle, it can be quite hard to avoid. When the sympathetic nervous system is activated, our brain diverts energy to those places that will need it for escape, e.g., away from digestion and towards our legs, arms, etc. Certain higher brain functions are put on hold, our sole aim is to get out of the “dangerous” situation we find ourselves in.

“I may be lucky enough to operate within such a company culture, but what about within myself? My colleagues might not be blaming me, but am I doing that to myself?”

While this is a great response to finding ourselves met with a ferocious lion, it doesn’t work so well for the sorts of daily challenges and threats we typically meet in the modern world. Running away isn’t a very helpful response to an accusation of “you messed up.”

A no-blame culture takes this psychology into account. It recognizes that attributing blame generally places people into a frame of mind where they will find it harder to reason about the issue. If we avoid blame, recognize that any event that occurred was a part of a wider web of conditions, of which we ourselves were just one part, then our brain stays within the safe zone of the parasympathetic nervous system, and we retain our higher brain functions. 

With this culture, organizations are likely to see stronger adaptability to failures and more effective adaptations to prevent future occurrences.

But what of ourselves? This psychology is human; it applies in our personal and our inner lives. If I promise to meet a friend at a certain time and arrive late, I can choose which approach to use to respond: blame or acceptance. Blame will likely put me in a mindset where all I need to do is “get away,” in which case the exact same event is likely to happen again. If, however, I respond with acceptance — e.g., “Thank you for your patience and waiting. I sometimes struggle with timeliness” — we keep our brain faculties and can explore ways in which this might be different. (“I will let you know next time if I’m running late” or “Next time I’ll leave 15 minutes earlier.”)

In this case, I didn’t try to run away or rush to resolve the uncomfortable feeling that accompanies a mea culpa. Instead I showed myself some grace and acknowledged that mistakes can happen. That allowed me to stay in a space where I could focus my energy on thinking about how to avoid repeating a similar situation in my work.

Training ourselves to respond to everything that happens in this way can have many benefits. Many of us have heard of “mindfulness.” Some of us have even tried to practice it, too. This no-blame culture is right there in Jon Kabat-Zinn’s definition of mindfulness, where he says:

Mindfulness is awareness that arises through paying attention, on purpose, in the present moment, non-judgmentally.

Here, he recommends approaching our experience “non-judgmentally.” This is essentially the no-blame culture directed towards our own lived experience. We can see from this that this approach is nothing new; it has been known to humans for millennia. Yet, it is something we can actively practice, here and now.