Loki 1.3.0 Released!

1.3.0 Released (and 1.2.0 and 1.1.0)!

Welcome to 2020! (We’re a little slow with that on the Loki team.) To kick off the year we are releasing Loki 1.3! Anyone running Loki in microservices mode will be excited by this release as it introduces the Loki Query Frontend. (If you aren’t using microservices, be patient – good things will be coming your way soon.)

The query frontend sits in front of the queriers and allows sharding queries based on time. This lets your queriers execute large queries in parallel by breaking them up into smaller pieces.

Late last year we also released 1.1.0 and 1.2.0 without much fanfare, so I’ll talk through highlights from those releases as well.

1.1.0 and 1.2.0

Mostly these releases addressed some incompatibilities in the Loki HTTP APIs with respect to the Prometheus APIs. But we also found that the hashing algorithm used on the labels to calculate the stream fingerprint was more prone to collisions than we had realized, so we modified this behavior inside Promtail and Loki to mitigate it. This mitigation should help reduce instances where you get out of order errors from Loki, which may have been caused by fingerprint hash collisions.

There were quite a few other PRs merged in these releases. I recommend taking a look at the 1.1.0 and 1.2.0 change logs.

1.3.0 Performance!

As mentioned, the highlight of this release is the query frontend, and we plan on having a dedicated post explaining this change in greater detail coming soon! However, if you are eager to see how we are using this now, take a look at the jsonnet changes.

Aside from the query frontend, we made several changes to improve Loki’s performance:

Configurable Compression Algorithm

PR 1411 introduced the ability to select which compression algorithm you would like to use. We found that gzip was very good for compression but was very slow, and this was causing slow query responses.

LZ4 was a good compromise of speed and compression performance. However, we had some issues with non-deterministic output of compressed chunks (2 ingesters compressing the same data would produce a chunk with a different checksum, even though they would still decompress back to the same input data). This was interfering with the change below for syncing chunks to reduce duplicates.

Snappy is currently our compression algorithm of choice. It performs much better than gzip for speed, but it is not as efficient in storage. This was an acceptable tradeoff for us.

Target Chunk Size

PR 1406 added the chunk_target_size configuration, which instructs Loki to try to fill chunks to the target size before flushing. This allows you to create bigger chunks than before, which can help reduce overhead on storage systems as well as overhead on querying. We are currently targeting 1572864bytes (1.5MB) chunks, which is helping our GCS burden. But it has also caused a number of problems for Memcached, which does not like storing objects this big. We will be looking soon to see if Redis performs better with objects of this size.

Better Sync of Flushed Chunks to Reduce Duplicates

If you have a replication factor greater than 1, PR 1438 will be interesting for you. Extending the work done by Bryan Boreham for Cortex, we added to Loki a mechanism for periodically syncing chunks across ingesters. This allows leveraging the code in Cortex for de-duplicating chunks when flushed to the backend store. In our environments, we have seen a reduction of written log data from about 2.9x source data size (replication factor of 3) to 1.5x source data size.

We also plan a separate post to better explain how this works and how to use it!

Promtail Syslog Support

Thanks to Sebastian Widmer for contributing a change to Promtail, which allows it to listen for syslog messages and send them to Loki!

There has been some debate about what the scope of the Promtail agent should be, and if it should do more than just tail files, but we like our agent and we want people to use it as much as possible. Many people expressed interest in expanding the capabilities to include syslog support, and so with a big contribution from Sebastian, Promtail now supports this.

We haven’t really put this feature through its paces, so you still might want a full-fledged syslog server for collecting your syslog data in front of Promtail, but give it a try and let us know how it works!

What’s Next?

We have a few more Loki-related blog posts coming up to talk more about what’s coming for Loki in 2020, so stay tuned!

In the meantime, check out the full changelog for this release.