Loki v1.5.0 released, with no more dependency on a separate index store
Today we released version 1.5.0 of Loki! This release comes with some really exciting news and a little bit of caution if you operate Loki installations.
The big news
The feature I really want this release to be remembered for is the introduction of a new index option named boltdb-shipper. Thanks to Sandeep Sukhani and team for putting in a lot of hard work on this feature. This new index allows you to run Loki with only an object store (S3, GCS, Filesystem, etc.); you no longer need a separate dedicated index store (DynamoDB, Bigtable, Cassandra, etc.)!
The boltdb-shipper index uses the in-memory boltdb index but send snapshots to the object store at regular intervals. This allows for sharing the index information via the object store. There are still many challenges around this approach, bugs to fix, and changes to be made, but this is the direction we are moving Loki. The benefits are huge, drastically simplifying the deployment of Loki and further reducing the cost to operate.
This feature is very new and very much experimental – that being said, it’s available to check out today in 1.5.0. So go try it out! Let us know how it works for you and help us flush out any bugs as we make this the future index store for Loki!
Important upgrading notes
A word of caution to Loki operators. The 1.5.0 release of Loki has updated the Cortex library dependency version to Cortex 1.0, which contained a large amount of breaking config changes. The Cortex maintainers took the opportunity to standardize their config options in 1.0 – which might result in errors running your current Loki config with version 1.5.0.
We apologize for this and are always striving to reach the right compromise of code simplicity and user/operating experience. In this case we opted to keep a simplified configuration in line with Cortex rather than have a more complicated and error-prone internal config mapping, or support for multiple config names for the same feature. This does result in breaking config changes for some configurations; however, these will fail fast and should hopefully be easy to fix and move on.
We recommend downloading one of the Loki 1.5.0 release binaries from the GitHub release page and running it with -config.file=yourconfig.yaml to see if any config-related errors are thrown. Any invalid config options will fail fast, and you can reference this annotated list of changes to see what the new value should be.
Another important change to be aware of: The user the Loki process runs as in the Docker image changed from root 0:0 to loki 10001:10001. This might impact you if you run Loki using non-default ports that are lower than 1000 (for either HTTP or GRPC).
The Docker user change might also affect you if you use the filesystem store. Existing files need to be readable by the new user, and the location needs to be writeable by the new user as well.
Last if you are running Loki via docker-compose or directly via docker and are attaching a volume to persist storage, the directories Loki is looking by default for files has changed from /tmp/loki/index -> /loki/index and /tmp/loki/chunks to /loki/chunks. You may need to adjust your mounts accordingly.
The upgrading guide contains more detailed information on all these changes.
More on Loki
If you can’t get enough of new Loki features and improvements, check out the full changelog for more highlights. This release actually included almost 140 PRs!
If you want to hear more about the boltdb-shipper, other new features, and even more exciting features to come, check out the GrafanaCONline talk Cyril Tovena and I are giving tomorrow (Thursday, May 21 at 9:30am PT / 12:30pm ET / 16:30 UTC) about the Future of Loki!
