Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Enterprise Metrics Manage a tenant Monitor GEM usage

Monitor GEM usage by access policy and token

Monitor usage based on access policy and token to gain insight into which users in a tenant are running expensive queries in GEM. GEM reports these usage details in the query-frontend component’s query stats log.

Set up usage details logging

Set up usage details logging to capture a user’s access policy and token when they run a query in GEM.

1. Use the following CLI flags to send request headers containing a query author’s corresponding access policy and token to the downstream query services:

   • header_x_access_policy_name: The query author’s access policy
   • header_x_token_name: The query author’s token

2. Use the query-frontend.log-query-request-headers setting in the configuration file to capture this information in the query stats log.

View usage details

To view the log for a GEM query, run the following LogQL query:

logql Copy

{namespace="_`<NAMESPACE>`_", cluster="_`<CLUSTER>`_", container="query-frontend"}
|= `query stats`

The returned log file includes the header_x_access_policy_name and header_x_token_name headers.

You can use this information to run additional queries that aggregate the usage statistics for a specified access policy or token. For example, the following LogQL query measures the time that queries spent in-queue based on the query author’s token:

logql Copy

avg by (header_x_token_name) (
  avg_over_time(
    {namespace="_`<NAMESPACE>`_", cluster="_`<CLUSTER>`_", container="query-frontend"}
     |= `query stats`
     | logfmt
     | unwrap queue_time_seconds [$__auto]
))