Pro Tip: Instantly Turn Slack Messages into Grafana Annotations with the Memo Tool

I have been a Grafana power user since almost the day it was conceived.

During this time, I’ve gotten acquainted with a few quirks but also many features, some of which are rather obscure.

One of these features that few know about but I absolutely love is annotations. Pulling in point-in-time textual events and time frames and overlaying them on top of metric or log data is a fantastic way to correlate disparate pieces of information: marketing events on top of pageviews, configuration deployments over service latency charts, and so on.

The possibilities to gain novel insights and attribute pattern changes are fairly obvious.

Grafana has a very useful API to create and manage annotations. As such, I recommend anyone who builds deployment systems, configuration management systems, marketing automation, and anything that may cause certain charts to go funky to make use of this API to create annotations to embellish your charts. It works especially well across different domains because you can tag the annotations to classify them and only show the relevant ones based on the use case.

My colleague Joe already shared how we use Loki and kubernetes-diff-logger to show events (in particular Kubernetes deployments) as annotations. But the world is not perfect. We don’t have 100% automation. Sometimes we make a change or trigger some event for which we have no automation and want to manually add an annotation.

For those cases, we need an as-frictionless-as-possible way to add categorized annotations into Grafana.

Introducing… “memo.”

Memo is designed to do one thing and do it well: meet you where you are, and get your message into Grafana.

Turn a Slack message like this …

… into an annotation like this:

Luckily this memo was shared on Slack; otherwise somebody might freak out if they see this chart!

Currently, the memo tool supports the shell and Slack. In the shell, it will by default add tags for your username and hostname; in Slack it will include tags for the author and channel name.

While you can also customize the tags, this already allows for a great deal of categorization. For instance, we have different Slack channels tied to distinct hosted services, which makes it easy to pull in the annotations for those services in question, and possibly the ones you depend on as well.

If something happened, and you want to tell your colleagues about it, just put “memo” in front of your sentence. The Slackbot will pick it up and turn it into a Grafana annotation.

And… that’s it. Curious to hear what you think!