How to customize your Loki deployment with Ansible

Michal Vaško is a DevOps engineer at cloudWerkstatt, with a passion for open source technology and a deep love for observability.

While operations or platform teams have long relied on visibility into metrics to react swiftly, the idea of doing the same thing with logs was once just a dream. Thankfully, Grafana Loki has revolutionized the logging stack, giving you the same level of visibility with logs that you get with metrics.

If you haven’t embarked on the Loki journey yet, now is the time with the release of Loki 3.0!

There are several deployment methods available for various deployment modes: Helm, Tanka, Docker or Docker Compose, local, and from source. For more details on all these methods, you can refer to the Grafana Loki installation guide.

All these installation methods are great, serving their purpose and providing flexibility to users so you can deploy Loki in a manner that fits your specific scenario. But let me introduce you to another approach — the Ansible Loki Role. This role supports Debian, Ubuntu, and Red Hat. Originally migrated from Voidquark’s Loki repository to the Grafana Ansible collection, I’m excited to be part of the ongoing maintenance and improvements to the role.

Deployment of Loki is primarily focused on Kubernetes, for scalability reasons. But what if you’re not at that phase yet? Or you simply want to experiment? Or you have a small organization where deploying Loki is desired and you are already familiar with Ansible? Well, you’re in the right place. Let’s dive into setting up this Ansible role.

What you need to get started

• Ensure you have an Ansible controller deployed.
• Install the Ansible Grafana collection on your Ansible controller.
• Familiarize yourself with the README of the Ansible Loki Role.
• Create a simple default playbook or customize it to fit your requirements.

I’m going to assume you already have Ansible set up and have read the Loki role README to become more familiar with it.

Now, let’s proceed with installing the Ansible Grafana collection:

bash Copy

ansible-galaxy collection install grafana.grafana

How to use the default deployment

Let’s demonstrate an out-of-the-box deployment with the default configuration.

First, create a playbook named function_loki-default.play. This playbook is straightforward and deploys Loki with default settings.

yaml Copy

- name: Deploy Loki using the default configuration
  hosts: loki
  become: true
  roles:
    - role: grafana.grafana.loki

Next, execute this playbook to deploy Loki:

bash Copy

ansible-playbook function_loki-default.play

The default deployment for Loki is now in place, but most people will want to tweak it for their own needs. Let’s move on to customizing it.

How to customize your deployment

What if you want to tinker a bit more? Well, you’re not limited! You can configure each component exactly as you wish. Let’s delve into an example where we’ll use:

• Filesystem storage
• A ruler with Alertmanager
• A simple SSH alerting rule
• A compactor with a four-week retention period
• Other custom parameters

Let’s create a playbook named function_loki-customized.play for this customized deployment:

yaml Copy

- name: Deploy Loki using the local filesystem
  hosts: loki
  become: true
  roles:
    - role: grafana.grafana.loki
  vars:
    loki_querier:
      max_concurrent: 16
      engine:
        max_look_back_period: 672h
    loki_storage_config:
      tsdb_shipper:
        active_index_directory: "{{ loki_working_path }}/tsdb-index"
        cache_location: "{{ loki_working_path }}/tsdb-cache"
      filesystem:
        directory: "{{ loki_working_path }}/chunks"
    loki_ingester:
      wal:
        enabled: true
        dir: "{{ loki_working_path }}/wal"
      lifecycler:
        address: 127.0.0.1
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
        final_sleep: 0s
      chunk_idle_period: 1h
      max_chunk_age: 2h
      chunk_target_size: 1048576
      query_store_max_look_back_period: 672h
    loki_limits_config:
      split_queries_by_interval: 0
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_query_length: 0
      max_query_series: 50000
      retention_period: 672h
      allow_structured_metadata: false
      max_query_lookback: 672h
    loki_compactor:
      working_directory: "{{ loki_working_path }}/compactor"
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: filesystem
    loki_common:
      path_prefix: "{{ loki_working_path }}"
      storage:
        filesystem:
          rules_directory: "{{ loki_working_path }}/rules"
      replication_factor: 1
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: inmemory
    loki_ruler:
      rule_path: "{{ loki_working_path }}/rules_tmp"
      ring:
        kvstore:
          store: inmemory
      enable_api: true
      enable_alertmanager_v2: true
      alertmanager_url: http://localhost:9093
    loki_ruler_alerts:
      - name: Logs.sshd
        rules:
        - alert: SshLoginFailed
          expr: |
            count_over_time({job=~"secure"} |="sshd[" |~": Failed|: Invalid|: Connection closed by authenticating user" | __error__="" [15m]) > 6
          for: 0m
          labels:
            severity: critical
          annotations:
            summary: "{% raw %}SSH authentication failure (instance {{ $labels.instance }}).{% endraw %}"
            description: "{% raw %}Increase of SSH authentication failures in last 15 minutes\\n VALUE = {{ $value }}{% endraw %}"

Now that your customized Loki instance is deployed and ready to use, the next step is to send logs to Loki. There are various ways to send logs to Loki, such as using Grafana Alloy or Promtail. All methods are described in this section of the documentation.

Once your logs are in Loki, you can query them using the LogCLI utility or Grafana. However, before you can explore logs from Grafana, you must connect the “Loki” data source in your Grafana instance. If you’re deploying Grafana via Ansible, note that the Grafana Ansible role is included in the same collection. You can configure this data source in your inventory and re-run the Grafana Ansible role.

If you’re not using the Ansible role, you can simply add the data source as described in the documentation on managing data sources.

Once the data source is set up, simply navigate to your Grafana URL, followed by “/explore,” and select the “Loki” data source to start exploring your logs. For more details on how to explore logs in Grafana, check out this section of the documentation.

One more thing: The role is idempotent, designed to deploy Loki with configurations aligned to your inventory. If you want to clean up and remove the entire deployment, simply use the loki_uninstall tag. If no tag is specified, the role operates in deployment mode.

bash Copy

ansible-playbook function_loki-customized.play -t loki_uninstall

What’s next?

Your Loki instance is now up and running — congrats! — and I assume your logs are being delivered to Loki.

To continue learning about Loki, try creating amazing dashboards tailored to your needs. If you need inspiration, check out the NGINX logs dashboard or my dashboards like SSH Logs, Promtail Monitoring, SUDO Logs or Nextcloud Logs, which utilize Loki as a data source. You can also explore the Loki configuration and Loki role documentation.

I wish you a smooth deployment and lots of enjoyment on your Loki journey! And if managing the Loki stack seems overwhelming, consider Grafana Cloud. With Grafana Cloud, you can enjoy hassle-free log management without the need to manage Loki.