Grafana Mimir version 2.4 release notes

Grafana Labs is excited to announce version 2.4 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Note: If you are upgrading from Grafana Mimir 2.3, review the list of important changes that follow.

Features and enhancements

Query-scheduler ring-based service discovery: The query-scheduler is an optional, stateless component that retains a queue of queries to execute, and distributes the workload to available queriers. The use the query-scheduler, query-frontends and queriers are required to discover the addresses of the query-scheduler instances.
In addition to DNS-based service discovery, Mimir 2.4 introduces the ring-based service discovery for the query-scheduler. When enabled, the query-schedulers join their own hash ring (similar to other Mimir components), and the query-frontends and queriers discover query-scheduler instances via the ring.
Ring-based service discovery makes it easier to set up the query-scheduler in environments where you can’t easily define a DNS entry that resolves to the running query-scheduler instances. For more information, refer to query-scheduler configuration.
New API endpoint exposes per-tenant limits: Mimir 2.4 introduces a new API endpoint, which is available on all Mimir components that load the runtime configuration. The endpoint exposes the limits of the authenticated tenant. You can use this new API endpoint when developing custom integrations with Mimir that require looking up the actual limits that are applied on a given tenant. For more information, refer to Get tenant limits.
New TLS configuration options: Mimir 2.4 introduces new options to configure the accepted TLS cipher suites, and the minimum versions for the HTTP and gRPC clients that are used between Mimir components, or by Mimir to communicate to external services such as Consul or etcd.
You can use these new configuration options to override the default TLS settings and meet your security policy requirements. For more information, refer to Securing Grafana Mimir communications with TLS.
Maximum range query length limit: Mimir 2.4 introduces the new configuration option -query-frontend.max-total-query-length to limit the maximum range query length, which is computed as the query’s end minus start timestamp. This limit is enforced in the query-frontend and defaults to -store.max-query-length if unset.
The new configuration option allows you to set different limits between the received query maximum length (-query-frontend.max-total-query-length) and the maximum length of partial queries after splitting and sharding (-store.max-query-length).

The following experimental features have been promoted to stable:

Helm chart improvements

The mimir-distributed Helm chart is the best way to install Mimir on Kubernetes. As part of the Mimir 2.4 release, we’re also releasing version 3.2 of the mimir-distributed Helm chart.

Notable enhancements follow. For the full list of changes, see the Helm chart changelog.

Added support for topologySpreadContraints.
Replaced the default anti-affinity rules with topologySpreadContraints for all components which puts less restrictions on where Kubernetes can run pods.

Important: if you are not using the sizing plans (small.yaml, large.yaml, capped-small.yaml, capped-large.yaml) in production, you must reintroduce pod affinity rules for the ingester and store-gateway. This also fixes a missing label selector for the ingester. Merge the following with your custom values file:

ingester:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - ingester
          topologyKey: "kubernetes.io/hostname"
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: "kubernetes.io/hostname"
store_gateway:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - store-gateway
          topologyKey: "kubernetes.io/hostname"
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: "kubernetes.io/hostname"

Updated the anti affinity rules in the sizing plans (small.yaml, large.yaml, capped-small.yaml, capped-large.yaml). The sizing plans now enforce that no two pods of the ingester, store-gateway, or alertmanager StatefulSets are scheduled on the same Node. Pods from different StatefulSets can share a Node.
Support for Openshift Route resource for nginx has been added.

Important changes

In Grafana Mimir 2.4, the default values of the following configuration options have changed:

-distributor.remote-timeout has changed from 20s to 2s.
-distributor.forwarding.request-timeout has changed from 10s to 2s.
-blocks-storage.tsdb.head-compaction-concurrency has changed from 5 to 1.
The hash-ring heartbeat period for distributors, ingesters, rulers, and compactors has increased from 5s to 15s.

In Grafana Mimir 2.4, the following deprecated configuration options have been removed:

The YAML configuration option limits.active_series_custom_trackers_config.
The CLI flag -ingester.ring.join-after and its respective YAML configuration option ingester.ring.join_after.
The CLI flag -querier.shuffle-sharding-ingesters-lookback-period and its respective YAML configuration option querier.shuffle_sharding_ingesters_lookback_period.

With Grafana Mimir 2.4, the anonymous usage statistics tracking is enabled by default. Mimir maintainers use this anonymous information to learn more about how the open source community runs Mimir and what the Mimir team should focus on when working on the next features and documentation improvements. If possible, we ask you to keep the usage reporting feature enabled. In case you want to opt-out from anonymous usage statistics reporting, refer to Disable the anonymous usage statistics reporting.

Bug fixes

PR 2979: Fix remote write HTTP response status code returned by Mimir when failing to write only to one ingester (the quorum is still honored when running Mimir with the default replication factor of 3) and some series are not ingested because of validation errors or some limits being reached.
PR 3005: Fix the querier to re-balance its workers connections when a query-frontend or query-scheduler instance is terminated.
PR 2963: Fix the remote read endpoint to correctly support the Accept-Encoding: snappy HTTP request header.