Loki’s Path to GA: Docker Logging Driver Plugin & Support for Systemd

Launched at KubeCon North America last December, Loki is a Prometheus-inspired service that optimizes storage, search, and aggregation while making logs easy to explore natively in Grafana. Loki is designed to work easily both as microservices and as monoliths, and correlates logs and metrics to save users money.

Less than a year later, Loki has almost 6,500 stars on GitHub and is now quickly approaching GA. At Grafana Labs, we’ve been working hard on developing key features to make that possible. In the coming weeks, we’ll be highlighting some of these features. This post will focus on:

• Docker logging driver plugin
• Support for systemd

Both serve to extend the sources from which users can pull logs into Loki.

Docker Logging Driver Plugin

For Non-Kubernetes Workloads

In order to ship logs to Loki, we created promtail, which also allows you to add labels and information to the logs. Promtail log tailing works out of the box when you’re running workloads on Kubernetes – but not everyone uses Kubernetes.

For a variety of reasons, some people prefer to use just plain Docker, and the problem with that is that you won’t get the same benefits from promtail when you’re not running it on Kubernetes. Until now you didn’t get the service discovery – or all the labels that allow you to filter by application (which is a basic requirement to search logs) – so we built the Docker driver for Loki.

Ease of Use

The Docker driver for Loki is a plugin that you can install on the Docker daemon on your server. You can tell it that when you run an application, it should use that log driver. As soon as your container starts, instead of saving the logs to a file, it will just straight away send it to Loki.

Because we are within Docker, we are able to do service discovery around the name of the container and the stack that is within. That makes using Loki way easier, because you don’t need to set up anything else. You just do it once on the daemon, and then it’s ready to go. Unlike promtail, the Docker driver doesn’t require you to configure any files.

The reason why we added this feature is to make sure that we have an easy onboarding for everyone, whatever they use. The Docker driver, for instance, is the easiest way to ship logs on ECS. People who are currently using Splunk on ECS would have a seamless experience switching to Loki and our logging driver.

For more about Docker logging drivers, read the docs.

Configuring the Driver

You can try our new Docker driver by signing up for a free Loki trial.

You will need to install the Docker driver on your host:

Copy

docker plugin install  grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Then simply start your Docker container with the Docker log-driver flag as shown below:

Copy

docker run --log-driver=loki \
    --log-opt loki-url="https://<user_id>:<password>@logs-us-west1.grafana.net/api/prom/push" \
    --log-opt loki-retries=5 \
    --log-opt loki-batch-size=400 \
    grafana/grafana

This command will configure the Grafana container to start with the Loki driver, which will send logs to logs-us-west1.grafana.net (which is our Grafana Cloud Hosted Logs demo endpoint), using a batch size of 400 entries. It will retry a maximum of 5 times if it fails.

You can find your user_id and password in your Grafana Cloud account in the Hosted Logs Demo section.

For more configuration options, refer to our Docker driver documentation.

Support for Systemd

While the Docker driver allows you to push logs from a container into Loki, the systemd feature works in the opposite direction: It pulls all systemd journal logs from the system into Loki using promtail. But both enable Loki to get logs from more sources. In particular, the new functionality – which was built by another Grafana Labs member, Robert Fratto – helps users pull logs from legacy workloads, which almost all companies have.

It’s really useful if you want logs from software that’s not running inside of Docker and Kubernetes, but rather is either running as a systemd service or logs to the systemd journal explicitly. Promtail was only able to read plain-text files before, and not all software logs that way.

Here’s an example of how to enable systemd support in the promtail config file:

yaml Copy

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/api/prom/push

scrape_configs:
  - job_name: journal
    journal:
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

You can read more about the feature in the docs.

In future blog posts, we’ll be focusing on other Loki features, including loki-canary, the pipeline stage, and query optimization. Be sure to check back for more content about Loki.