Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Overview
Per-tenant resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Grafana Mimir limits configuration.
Using the admin API of Grafana Enterprise Metrics, you can easily set a subset of these per-tenant limits.
The following limits are supported via the admin API:
ingestion_rate
ingestion_burst_size
max_series_per_query
max_global_series_per_user
max_global_series_per_metric
max_global_exemplars_per_user
ruler_max_rules_per_rule_group
ruler_max_rule_groups_per_tenant
max_fetched_chunks_per_query
max_fetched_series_per_query
max_fetched_chunk_bytes_per_query
compactor_blocks_retention_period
Setting limits via the admin API
Limits can be set using the admin API when creating a new tenant or when modifying an existing one.
Create a JSON payload, test1.json
, for creating a new tenant that includes a subset of per-tenant limits.
{
"name": "test1",
"display_name": "Test Instance 1",
"status": "active",
"cluster": "enterprise-metrics-dev",
"limits": {
"max_series_per_query": 42000,
"max_global_series_per_user": 42000,
"compactor_blocks_retention_period": "24h"
}
}
Use curl
to POST the data to the admin API.
$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants \
-X POST --data @test1.json
Read the newly created tenant.
curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 | jq
{
"name": "test1",
"display_name": "Test Instance 1",
"created_at": "2020-07-13T17:37:59.341728283Z",
"status": "active",
"cluster": "enterprise-metrics-dev",
"limits": {
"ingestion_rate": 350000,
"ingestion_burst_size": 350000,
"max_series_per_query": 42000,
"max_global_series_per_user": 42000,
"max_global_series_per_metric": 300000,
"max_global_exemplars_per_user": 0,
"ruler_max_rules_per_rule_group": 0,
"ruler_max_rule_groups_per_tenant": 0,
"max_fetched_chunks_per_query": 0,
"max_fetched_series_per_query": 0,
"max_fetched_chunk_bytes_per_query": 0,
"compactor_blocks_retention_period": "24h"
}
}
Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new tenant is created with some limit settings are populated from the global default limits. This also applies when limits are added to a tenant that previously did not have them: any limit settings that are not included will be populated from the global default limits.
Removing limits via the admin API
Limits can be removed from a tenant using the admin API by modifying the tenant.
Create a JSON payload, test1-update.json
, that doesn’t include the limits
field.
{
"display_name": "Test Instance 1",
"status": "active",
"cluster": "enterprise-metrics-dev"
}
Use curl
to PUT the data to the admin API.
$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 \
-X PUT --data @test1-update.json | jq
{
"name": "test1",
"display_name": "Test Instance 1",
"created_at": "2020-07-13T17:37:59.341728283Z",
"status": "active",
"cluster": "enterprise-metrics-dev"
}
The modified tenant object is included in the update response.
Limit sources
Limits applied to a tenant are only taken from a single location at a time:
- Per-tenant limits set by the admin API, backfilled with global defaults.
- Per-tenant limits set by a runtime configuration file, backfilled with global defaults.
- Global default limits set by a configuration file.
These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for a tenant using the admin API, any limit settings for that tenant in a runtime configuration file will be ignored.
Implementation
Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each tenant. This means that based on the period you have configured to refresh tenant-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of tenants in your Grafana Enterprise Metrics cluster.