Grafana Loki 3.3 release: faster query results via Blooms for structured metadata
The Grafana Loki 3.3 release is here, and it brings a fresh wave of enhancements aimed at making your log management experience faster, more efficient, and more scalable.
While this update includes the usual round of bug fixes and operational improvements, the standout feature is a shift in how Loki leverages Bloom filters—going from free-text search to harnessing the power of structured metadata. This will make it easier to find a specific needle in your haystack of logs, while improving the query experience for our OpenTelemetry community as well.
Query acceleration via Bloom filters is considered experimental for our Loki and Grafana Enterprise Logs (GEL) 3.3 releases, and is available in public preview for large Grafana Cloud customers.
Let’s dive into what this means for your workflows and how it sets the stage for Loki’s exciting future.
Query acceleration via Bloom filters
With Loki 3.0, we announced Bloom filters, a very promising, experimental feature intended to speed up needle in a haystack queries. Since then, we’ve received great insights from early adopters and learned a lot about running Blooms at a large scale—and the tradeoffs our users were willing to make.
With this in mind, and with the increasing adoption of OpenTelemetry, we adjusted our approach and decided to create Blooms leveraging structured metadata.
What is structured metadata in Loki?
Structured metadata allows you to attach metadata labels to each of your log lines. It was first introduced in Loki 2.9 for high cardinality labels that couldn’t be indexed, and for when it was too difficult to extract commonly-accessed log content into labels at query time. Then, In Loki 3.0, we added native OpenTelemetry support, which leverages structured metadata to store all the attributes OTel sends with each log line.
And now, this same structured metadata can also be used to accelerate queries.
Note: If you are not using OTel, you can still attach structured metadata to your log lines with Grafana Alloy’s Loki pipelines.
A refresher on Bloom filters, and why you would use them
Loki leverages Bloom filters in order to tell it what data not to search. In managing Loki, we’ve observed that many log searches touch lots of log lines but only need the results from a select few that meet a certain criteria.
As an example, maybe you’re looking for all log lines that have a certain user ID because you need to debug why an ecommerce user is being logged out of their account when they add an item to their shopping cart. Or maybe you want to find all the log lines that match a certain device ID to help troubleshoot a payment device or game controller.
How we are using Blooms to accelerate queries
Just like the previous version of Blooms, we still build Blooms per series. This time, instead of n-grams computed out of the whole log content, Blooms now contain the key of each structured metadata field seen within the series, along with all the existing key-value pairs.
Blooms built out of structured metadata are orders of magnitude smaller, thus faster to build, download, and query.
How to use query acceleration
To take advantage of Blooms, your queries need to satisfy the following criteria:
- Bloom blocks should be created and available in the Bloom gateways for the time interval of the query.
- Queries should invoke acceleration by using proper syntax to query structured metadata
For example, if you want to look for a specific trace ID, and the trace ID is available as structured metadata, the following expression should be accelerated if they are available:
{env="prod"} | traceID="3f54c1d2ae0c" | json | line_format "{{.api_route}} time={{.req_time}}"In this query, the {env=”prod”} stream selector will narrow down the search space to the matching series. Then, with traceID="3f54c1d2ae0c" we use Blooms to further narrow down the search space by discarding all the chunks from the matching series that we’re sure don’t contain this structured metadata key-value pair. Finally, we only apply the json parser and the line formatting to the logs from the matching chunks. This significantly reduces the amount of data Loki needs to download, unpack, and process.
How to upgrade
The new Bloom blocks have a new format that’s incompatible with the previous version of Blooms. As a result, any previously created blocks are now obsolete. We recommend you clean your Bloom blocks before upgrading to the new Loki version by deleting the bloom path inside the object store (see upgrade notes).
In addition to building Blooms for structured metadata, we also made some others changes and improvements for Blooms:
- We removed the Bloom compactor in favor of the Bloom planner and builder components, which are now configured in Helm and enabled in the simple scale deployment backend target.
- We introduced a new build strategy that split tasks based on the TSDB stats for the chunks size of the series. The strategy is configurable via the
bloom_planning_strategyper-tenant limit. We haven’t changed the default planning strategy, which splits the series key-space by a configurable factor. But we encourage users to try the new strategy, which should result in more consistently sized Bloom building tasks. - Another addition to the Bloom planner component is a disk-backed queue. By default, the queue keeps all tasks in memory, but you can enable the disk-backed queue if you have a large set of tenants or a memory-contrained environment.
Start using Loki 3.3 today
Loki 3.3 ships a whole set of new experimental features, improvements and bug fixes that opens the door to make Loki more scalable, faster, and easy to use in the near future. If you are a GEL customer, the same functionality that’s in Loki 3.3 is also part of our GEL 3.3 release.
Consult the Loki and GEL release notes and changelog for more:
And if you are new to Loki, please consult our Getting Started section to get up and running as quickly as possible. In the meantime, give the new features a spin, explore the benefits of structured metadata Blooms, and let us know your feedback. Your insights help us make Loki better with every release.
Happy logging!
Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!
