Documentation Grafana Enterprise Metrics Manage a tenant Setting per-tenant resource usage limits

Overview

Per-tenant resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Grafana Mimir limits configuration.

Using the admin API of Grafana Enterprise Metrics, you can easily set a subset of these per-tenant limits.

The following limits are supported via the admin API:

• ingestion_rate
• ingestion_burst_size
• max_series_per_query
• max_global_series_per_user
• max_global_series_per_metric
• max_global_exemplars_per_user
• ruler_max_rules_per_rule_group
• ruler_max_rule_groups_per_tenant
• max_fetched_chunks_per_query
• max_fetched_series_per_query
• max_fetched_chunk_bytes_per_query
• compactor_blocks_retention_period

Setting limits via the admin API

Limits can be set using the admin API when creating a new tenant or when modifying an existing one.

Create a JSON payload, test1.json, for creating a new tenant that includes a subset of per-tenant limits.

json Copy

{
  "name": "test1",
  "display_name": "Test Instance 1",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "max_series_per_query": 42000,
    "max_global_series_per_user": 42000,
    "compactor_blocks_retention_period": "24h"
  }
}

Use curl to POST the data to the admin API.

console Copy

$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants \
    -X POST --data @test1.json

Read the newly created tenant.

console Copy

curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_burst_size": 350000,
    "max_series_per_query": 42000,
    "max_global_series_per_user": 42000,
    "max_global_series_per_metric": 300000,
    "max_global_exemplars_per_user": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "max_fetched_chunks_per_query": 0,
    "max_fetched_series_per_query": 0,
    "max_fetched_chunk_bytes_per_query": 0,
    "compactor_blocks_retention_period": "24h"
  }
}

Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new tenant is created with some limit settings are populated from the global default limits. This also applies when limits are added to a tenant that previously did not have them: any limit settings that are not included will be populated from the global default limits.

Removing limits via the admin API

Limits can be removed from a tenant using the admin API by modifying the tenant.

Create a JSON payload, test1-update.json, that doesn’t include the limits field.

json Copy

{
  "display_name": "Test Instance 1",
  "status": "active",
  "cluster": "enterprise-metrics-dev"
}

Use curl to PUT the data to the admin API.

console Copy

$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 \
    -X PUT --data @test1-update.json | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev"
}

The modified tenant object is included in the update response.

Limit sources

Limits applied to a tenant are only taken from a single location at a time:

• Per-tenant limits set by the admin API, backfilled with global defaults.
• Per-tenant limits set by a runtime configuration file, backfilled with global defaults.
• Global default limits set by a configuration file.

These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for a tenant using the admin API, any limit settings for that tenant in a runtime configuration file will be ignored.

Implementation

Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each tenant. This means that based on the period you have configured to refresh tenant-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of tenants in your Grafana Enterprise Metrics cluster.