Documentation Grafana Enterprise Metrics Manage a tenant Monitor GEM usage

Monitor GEM usage by access policy and token

Monitor usage based on access policy and token to gain insight into which users in a tenant are running expensive queries in GEM. GEM reports these usage details in the query-frontend component’s query stats log.

Set up usage details logging

Set up usage details logging to capture a user’s access policy and token when they run a query in GEM.

1. Use the following CLI flags to send request headers containing a query author’s corresponding access policy and token to the downstream query services:

   • header_x_access_policy_name: The query author’s access policy
   • header_x_token_name: The query author’s token

2. Use the query-frontend.log-query-request-headers setting in the configuration file to capture this information in the query stats log.

View usage details

To view the log for a GEM query, run the following LogQL query:

logql Copy

{namespace="_`<NAMESPACE>`_", cluster="_`<CLUSTER>`_", container="query-frontend"}
|= `query stats`

The returned log file includes the header_x_access_policy_name and header_x_token_name headers.

You can use this information to run additional queries that aggregate the usage statistics for a specified access policy or token. For example, the following LogQL query measures the time that queries spent in-queue based on the query author’s token:

logql Copy

avg by (header_x_token_name) (
  avg_over_time(
    {namespace="_`<NAMESPACE>`_", cluster="_`<CLUSTER>`_", container="query-frontend"}
     |= `query stats`
     | logfmt
     | unwrap queue_time_seconds [$__auto]
))