Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Grafana Enterprise Metrics Documentationbreadcrumb arrow Release notesbreadcrumb arrow Grafana Enterprise Metrics v1.6 release notes
Enterprise

Version 1.6 release notes

The Grafana Enterprise Metrics (GEM) team is excited to announce the release of GEM 1.6.

Features and enhancements

  • GEM 1.6 adds 2 new API endpoints to help users explore the metrics cardinality of tenants within their cluster. These endpoints make it possible to identify the highest cardinality metrics in a tenant, as well as the label names with the highest value count. The GEM plug-in (starting with version 3.3.0) includes a set of cardinality dashboards built on these endpoints to allow users to explore the cardinality data in a visual way. For more information, see the Cardinality Analysis page.
  • GEM 1.6 includes experimental support for query sharding, which allows a single query to be split up into groups of smaller series and parallelized. This improves the performance of high cardinality queries which touch a lot of series. This feature is still very much under active development, so take care when testing.
  • GEM 1.6 includes experimental support for a new compaction strategy known as “split and merge compaction”. This strategy enables users to parallelize the compaction of a large tenant horizontally across multiple machines. It also enables users to overcome the 64GiB block index limit inherent in Prometheus, raising the ceiling on how many time series a single tenant can support. As with query sharding, this feature is still under active development and should be enabled with caution. For more information, see the relevant section in Configuration.
  • GEM 1.6 eliminates the use of the word “instance,” which had been used as a synonym for “tenant”. Switching to “tenant” allows GEM to more closely align with terminology used in Cortex and avoids confusion caused by the fact that “instance” can have multiple meanings besides “tenant” (e.g., an “instance” of an ingester).
    • All references to “instance” in the GEM documentation and GEM Grafana plug-in have been replaced with the word “tenant”.
    • We’ve added new v2 API endpoints which specify “tenant” in the request and response bodies. We have deprecated and will eventually delete the original endpoints that use the word ‘instance’. For more detail, see the admin-api page.
  • GEM 1.6 renames the default auth option to trust. This reflects the fact that the trust option does not actually do any authentication, but simply ‘trusts’ the value of the HTTP header X-Scope-Org-Id to determine what tenant to send a request to. GEM users are highly recommended to use auth type enterprise, not trust.
  • We’ve made some efficiency improvements, including reduced CPU and memory utilization on the ingesters and queries and when seeing large volumes of out-of-order samples.

Upgrade considerations

  • Please upgrade your Grafana GEM administrative plug-in to version 3.3.0 in order to get the dashboards associated with the Cardinality Analysis feature.
    • To ensure you receive the most up-to-date dashboards, you must disable and then re-enable the plug-in. Therefore, once GEM plug-in version 3.3.0 is installed, navigate to its Plugin configuration page, and hit the Disable plugin button. After Grafana reloads, hit the Enable plugin.
  • See the note under “Changes” about the auth option being renamed from the default to trust. GEM 1.6 sets the default max number of inflight ingester push requests (-ingester.instance-limits.max-inflight-push-requests) to 30000. We don’t anticipate this impacting the steady state operation of any GEM user; it is in place as a safeguard against clusters getting overwhelmed by a spike in requests.
  • The option -querier.ingester-streaming has been removed. The querier and ruler now always use streaming method to query ingesters.
  • GEM 1.6 has removed support for block deletion marks migration (-compactor.block-deletion-marks-migration-enabled). If you’re upgrading to GEM 1.6 from GEM < 1.1.0, then you need to upgrade to GEM 1.5 first and run the migration at least once. Then you can upgrade to 1.6.

Bug fixes

1.6.2

  • GEM 1.6.2 fixes a bug that caused tenant limits set via the Admin API to be invalidated when upgrading to GEM 1.6.0 or GEM 1.6.1 from previous versions of GEM.

1.6.1

1.6.1 fixes one important bug in the 1.6.0 release:

  • GEM 1.6.1 fixes an issue with the way default tenant limits were applied when setting tenant limits using the Admin API.

1.6.0

  • GEM 1.6 changes the way per-tenant limits are stored to avoid breaking changes between version updates.
  • GEM 1.6 has a bugfix to ensure that the recording rules executed by the __system__ tenant (part of the self-monitoring feature) are properly split across rulers. This should eliminate out-of-order errors on the __system__ tenant.
  • GEM now enforces that a tenant must exist even when a wildcard access policy is used. Writes to non-existent tenants with a wildcard write access policy will fail. Similarly, if there are any read requests to a non-existent tenant using a wildcard write access policy, they will fail.
  • If a GEM tenant is deleted but the access policies to that GEM tenant are not, GEM will now fail read and write requests to that tenant.