New in Grafana Loki 2.6: multi-tenant queries and targeted log line deletion

Grafana Loki 2.6 is here! 

In addition to improvements in query performance, we are excited to introduce two key features in Grafana Loki: cross-tenant query federation and targeted deletes. 

What are multi-tenant queries in Grafana Loki?

Since we introduced the ability to simultaneously query multiple tenants in our open source TSDB, Grafana Mimir, and Grafana Enterprise Metrics, the Grafana Loki community has had one persistent query of its own: When can we do that? 

With the release of Grafana Loki 2.6, the answer is now. Cross-tenant query federation is now available in Loki, allowing users to access data from multiple tenants using one query to return a single, consolidated result. 

Grafana Loki is natively multi-tenant, which means one database can be operated to serve many different teams or lines of businesses within an organization. Each tenant gets its own slice of the database and is not affected by the others within the shared Loki infrastructure. So if there are five different groups to support, instead of running five different databases, an administrator can consolidate the groups’ logs into one Loki database, putting each in their own tenant, which keeps each group’s data compartmentalized within the cluster. Not only does this lower overhead costs; maintaining a single database also simplifies workflows for operators within an organization.

While providing an independent experience for each customer or team is important, there are use cases in which an administrator or operator needs a more global view of all the systems at work. For example, while Team A has access to the logs in their tenant and Team B has access to the logs in their tenant, an admin may need to query and analyze log data from both Team A and Team B. 

This is where cross-tenant query federation comes into play.

How multi-tenant queries work

Instead of the standard model of one query to one tenant, with cross-tenant queries, users can access multiple tenants at once, and the Loki database will merge the results and return one composite result.

Here is a standard LogQL query to Tenant1:

Copy

curl -H 'X-Scope-OrgID:Tenant1 -G -s  "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

Getting to a multi-tenant query is simple! Just make a small change to the X-Scope-OrgID header, which identifies the tenants you want to query. Specify as many tenants as you need, separating their names by the | character.

In this example, this cross-tenant query is written to target Tenant1, Tenant2, and Tenant3 for the requested data:

Copy

curl -H 'X-Scope-OrgID:Tenant1|Tenant2|Tenant3' \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

Multi-tenant queries in Grafana Loki are especially useful to an operator or administrator who may need to identify log lines with important or sensitive data across all tenants within a single database. Instead of running individual queries to search for a specific string in each tenant, users can now run a single query to search all tenants within their cluster. Also for those use cases in which you need to build metrics from logs, you can now easily calculate a metric that requires data from multiple tenants with one query, like a global error rate across all your services.

But it’s not all about accessing information in multiple places. Cross-tenant query federation also reduces the context switching for a user who, in the past, had to flip between Query A to Tenant A and Query B to Tenant B to compare or consolidate information. With multi-tenant querying, a user can now search all of the data for both tenants in one place by default all the time.

Cross-tenant query federation in Grafana Enterprise Logs

The ability to query multiple tenants at once is also available in Grafana Enterprise Logs 1.5. The big difference for enterprise users is that the feature is integrated with GEL’s access control model so users cannot run cross-tenant queries without proper permissions. To execute these queries in GEL, users must provide a username, which is comprised of the tenants you want to query, and a password in the form of an API token. GEL will validate that the given token provides the user read access to all the respective tenants in the query. If it does, GEL will return the query results. If not, the database will reject the query.

Deleting targeted log lines in Grafana Loki

Many organizations have safeguards in place to make sure sensitive or unwanted information doesn’t leak into log lines. We have a great webinar on managing privacy in log data with Grafana Loki available on demand about some best practices that you can put in place with Promtail, our preferred log collector for Grafana Loki.

But even with the best laid plans, unwanted log lines — for example, those with personally identifiable information (PII) or protected health information (PHI) — can get into your logging database. Other times, it is less about an unwanted data leak, but about having a way to service a request from a customer or end user who wants their data removed from your systems. GDPR, for example, requires businesses to erase an individual’s personal data upon request when certain circumstances apply.

How deleting log lines used to work

In any of these cases, you need a way to eliminate those unwanted log lines. Before this release, Grafana Loki had limited options to do this. One way to triage the issue would be to reset the retention period for the specific tenant(s) in your cluster. Users can set specific retention periods for log data for every tenant, so the solution would be to shorten the retention time for whatever tenants had the unwanted log lines. For example, if there are three months worth of logs that may have questionable data in TenantA, you could change TenantA’s retention period to one day, deleting everything older than that, regardless if they were good or bad log lines. Essentially, you would have to start from scratch.  

Users could also selectively delete all log lines that matched a specific label or set of labels within a given time range. So if a customer asked you to delete all the information about Topic X because of a GDPR request and you knew that all data related to Topic X was tagged with the label {application=app123}, you could delete every log line over a 2-week period tagged with that label. Again, a suboptimal solution because you’re deleting all data with that label, including some lines that may have no relation to Topic X.

How to delete specific log lines in Grafana Loki

In Grafana Loki 2.6, we introduced the functionality to perform more targeted deletions. Users are now empowered to craft your own LogQL query (or you can also leverage the new Grafana Loki query builder introduced in Grafana 9.0!) to identify specific log lines to delete.

Once the user has completed crafted a query that matches all the lines they want to delete, the user can then issue a request to delete those lines. The example below would delete all lines in Tenant1 that have the label {foo=bar} and contain the string “other” between the timestamps 1591616227 and 1591619692:

Copy

curl -g -X POST \
'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}|= “other”&start=1591616227&end=1591619692' \
  -H 'X-Scope-OrgID: Tenant1'

Any query that you can write in LogQL can now be used to mark log lines for deletion. Added bonus: On the backend, it takes up to 48 hours for the data to be officially erased from the store. On the frontend, as soon as you issue a delete request, Grafana Loki will immediately filter out the identified log lines so that they are no longer available to query. This makes the deletion request appear almost instantaneous to any end user querying Loki — an improvement over our previous deletion capabilities, which only stopped returning the data once it had been deleted from the store.

Targeted deletions in Grafana Enterprise Logs

As with cross tenant queries, targeted deletions in GEL are natively integrated with the database’s authNZ capabilities. When issuing the delete request the user must supply the proper credentials in the form of a username and password. The username is the tenant for which you want to perform the deletion, and the password is an API token. GEL will validate that the API token grants delete access to the tenant specified in the username. These enterprise access controls eliminate the chances of anyone accidentally deleting information from a tenant.

Learn more about Grafana Loki

Thanks to all the Grafana Loki users and contributors out there who continue to help grow the project. Our team is growing fast to keep up with the demand, and we’re all incredibly excited about the future of Loki. 

If you run Grafana Loki, be sure to check out the release notes and upgrading guide for more details on getting started with Grafana Loki 2.6 today!

If you run Grafana Enterprise Logs, be sure to check out the release notes and documentation for GEL 1.5, which is based off of Loki 2.6, so you can enjoy all the amazing changes made to the upstream OSS project! If you are interested in learning more about Grafana Enterprise Logs, please contact our team. 

Grafana Cloud is the easiest way to get started with metrics, logs, traces, and dashboards. We have a generous free forever tier and plans for every use case. Sign up for free now!