Menu
Documentationbreadcrumb arrow Grafana Enterprise Metricsbreadcrumb arrow Migrating from Grafana Enterprise Metrics 1.7
Enterprise

Migrating from Grafana Enterprise Metrics 1.7

As an operator, you can migrate a deployment of Grafana Enterprise Metrics 1.7 to Grafana Enterprise Metrics 2.3. The steps to do so apply to any deployment environment, and there are specific instructions for environments deployed with Helm.

Grafana Enterprise Metrics 2.0 extends Grafana Mimir. Grafana Enterprise Metrics 1.7 extends Cortex. Grafana Mimir includes significant changes that simplify the deployment and continuous operation of a horizontally scalable, multi-tenant time series database with long-term storage.

The following changes make Grafana Mimir easy to run:

  • Removed configuration parameters that don’t require tuning
  • Renamed some parameters so that they’re more easy to understand
  • Updated the default values of some existing parameters to work out of the box

The Grafana Mimirtool (mimirtool) automates configuration conversion for you. It provides a simple migration by generating a Grafana Mimir configuration from a Cortex configuration. It can also generate a Grafana Enterprise Metrics 2.3 configuration from a Grafana Enterprise Metrics 1.7 configuration.

Before you begin

  • Ensure that you are running Grafana Enterprise Metrics 1.7.X.

    If you are running an older version of Grafana Enterprise Metrics, upgrade to 1.7.0 before proceeding with the migration.

  • Ensure that you have installed the Grafana Mimir monitoring mixin rules.

    The monitoring mixin has both alerting and recording rules to install in either Prometheus or Grafana Enterprise Metrics. The rules replace the Cortex monitoring mixin rules and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. To upload rules to the ruler using Grafana Mimirtool, refer to Grafana Mimirtool rules.

  • Ensure that you have installed the Grafana Mimir monitoring mixin dashboards.

    The dashboards replace the Cortex monitoring mixin dashboards and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. Alternatively, update the Grafana Enterprise Metrics plugin to automatically install the latest self monitoring dashboards.

Notable changes

Note: For a full list of changes, refer to the Grafana Mimir CHANGELOG.

  • By default, Grafana Enterprise Metrics 2.0 enables enterprise authentication. To continue using the Grafana Enterprise Metrics 1.7 default named trust, use -auth.type=trust.

  • The Grafana Enterprise Metrics 2.0 HTTP server defaults to listening on port 8080; previous versions defaulted to listening on port 80. The enterprise-metrics Helm chart already used port 8080 and is unaffected by this change.

  • Grafana Enterprise Metrics 2.0 uses anonymous as the default tenant ID where -auth.multitenancy=false. Previous versions used fake as the default tenant ID where -auth.enabled=false. Use -auth.no-auth-tenant=fake where -auth.multitenancy=false, to match the default tenant ID of previous versions.

  • Grafana Enterprise Metrics 2.0 removes the legacy HTTP prefixes.

    • Query endpoints

      LegacyCurrent
      /<legacy-http-prefix>/api/v1/query<prometheus-http-prefix>/api/v1/query
      /<legacy-http-prefix>/api/v1/query_range<prometheus-http-prefix>/api/v1/query_range
      /<legacy-http-prefix>/api/v1/query_exemplars<prometheus-http-prefix>/api/v1/query_exemplars
      /<legacy-http-prefix>/api/v1/series<prometheus-http-prefix>/api/v1/series
      /<legacy-http-prefix>/api/v1/labels<prometheus-http-prefix>/api/v1/labels
      /<legacy-http-prefix>/api/v1/label/{name}/values<prometheus-http-prefix>/api/v1/label/{name}/values
      /<legacy-http-prefix>/api/v1/metadata<prometheus-http-prefix>/api/v1/metadata
      /<legacy-http-prefix>/api/v1/read<prometheus-http-prefix>/api/v1/read
      /<legacy-http-prefix>/api/v1/cardinality/label_names<prometheus-http-prefix>/api/v1/cardinality/label_names
      /<legacy-http-prefix>/api/v1/cardinality/label_values<prometheus-http-prefix>/api/v1/cardinality/label_values
      /api/prom/user_stats/api/v1/user_stats

    • Distributor endpoints

      Legacy endpointCurrent
      /<legacy-http-prefix>/push/api/v1/push
      /all_user_stats/distributor/all_user_stats
      /ha-tracker/distributor/ha_tracker

    • Ingester endpoints

      LegacyCurrent
      /ring/ingester/ring
      /shutdown/ingester/shutdown
      /flush/ingester/flush
      /push/ingester/push

    • Ruler endpoints

      LegacyCurrent
      /<legacy-http-prefix>/api/v1/rules<prometheus-http-prefix>/api/v1/rules
      /<legacy-http-prefix>/api/v1/alerts<prometheus-http-prefix>/api/v1/alerts
      /<legacy-http-prefix>/rules<prometheus-http-prefix>/config/v1/rules
      /<legacy-http-prefix>/rules/{namespace}<prometheus-http-prefix>/config/v1/rules/{namespace}
      /<legacy-http-prefix>/rules/{namespace}/{groupName}<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}
      /<legacy-http-prefix>/rules/{namespace}<prometheus-http-prefix>/config/v1/rules/{namespace}
      /<legacy-http-prefix>/rules/{namespace}/{groupName}<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}
      /<legacy-http-prefix>/rules/{namespace}<prometheus-http-prefix>/config/v1/rules/{namespace}
      /ruler_ring/ruler/ring

    • Alertmanager endpoints

      LegacyCurrent
      /<legacy-http-prefix>/alertmanager
      /status/multitenant_alertmanager/status

Generating configuration for Grafana Enterprise Metrics 2.3

Grafana Mimirtool provides the mimirtool config convert --gem command for converting a Grafana Enterprise Metrics 1.7 configuration into a Grafana Enterprise Metrics 2.0 configuration. It can convert both flags and configuration files.

Downloading Grafana Mimirtool

Download the appropriate release asset for your operating system and architecture, and make it executable. For Linux with the AMD64 architecture:

bash 
curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool

Using Grafana Mimirtool

Note: To ensure that the Grafana Enterprise Metrics specific configuration isn’t removed from the output, you must use the --gem flag.

The tool removes any configuration parameters that are no longer available in Grafana Enterprise Metrics 2.3, and it renames any configuration parameter that has a new name. If you have explicitly set configuration parameters to a value that matches the Grafana Enterprise Metrics 1.7 default, by default, mimirtool config convert doesn’t update the value. To have mimirtool config convert --gem update explicitly set values from the Grafana Enterprise Metrics 1.7 defaults to the new Grafana Enterprise Metrics 2.3 defaults, provide the --update-defaults flag. For more information about using mimirtool for configuration conversion, refer to the Grafana Mimirtool configuration command named convert.

Migrating to Grafana Enterprise Metrics 2.3 with Helm

Grafana Enterprise Metrics 2.3 uses the Grafana Mimir mimir-distributed Helm chart version 3.1.0. You can update to the mimir-distributed Helm chart from the enterprise-metrics Helm chart.

Before you begin

  • Ensure that you are running the v1.9.0 release of the enterprise-metrics Helm chart.
  • Install yq v4.

To migrate to the mimir-distributed Helm chart:

  1. Update your Helm repositories.

    bash 
    helm repo update

  2. Convert your existing Grafana Enterprise Metrics configuration in your Helm values file.

    a. Extract the Grafana Enterprise Metrics configuration from the deployed Grafana Enterprise Metrics Secret and write the output to the enterprise-metrics.yaml file.

    bash 
    kubectl get secret <RELEASE>-enterprise-metrics -n <namespace> -o json | jq -r '.data."enterprise-metrics.yaml"' | base64 -d > enterprise-metrics-1.7.yaml

    If you are using an external configuration Secret, replace <RELEASE>-enterprise-metrics with the name of your Secret.

    b. Use mimirtool to update the configuration.

    bash 
    mimirtool config convert --gem --yaml-file enterprise-metrics-1.7.yaml --yaml-out enterprise-metrics-2.3.yaml

    c. Clean up the generated YAML configuration.

    You have to remove some fields that either the command mimirtool config convert generates or that are coming from an old configuration, because the mimir-distributed Helm chart has already set the default value for them. Use the following script to clean up those fields:

    bash 
    yq -i 'del(.activity_tracker.filepath,.alertmanager.data_dir,.compactor.data_dir,.frontend_worker.frontend_address,.ingester.ring.tokens_file_path,.ruler.alertmanager_url,.ruler.rule_path,.runtime_config.file)' mimir.yaml

    d. Make sure querier.query_ingester_within is greater than querier.query_store_after default value in mimir 2.3.0 which is set to 12 hours by default.

    Note: The Grafana Enterprise Metrics Helm chart added some configuration validation that prevents the installation to start if the value of a configuration (or combination of configurations) is invalid. Fix the configuration based on the validation errors.

    e. At the top level of your Helm values file, put the updated configuration under the mimir.structuredConfig key.

    Note: The mimir.structuredConfig field, which is added in v3.0.0, allows you to override a specific configuration without needing to rewrite the whole block string literal, such as in mimir.config.

    In your Helm values file:

    yaml 
    mimir:
  structuredConfig: <CONFIGURATION FILE CONTENTS>

    Example:

    yaml 
    mimir:
  structuredConfig: 
    ingester:
      ring:
        num_tokens: 512

    f. Enable enterprise configuration by setting enterprise.enabled to true and enterprise.legacyLabels to true. The value enterprise.legacyLabels installs resources using the same labels as the enterprise-metrics Helm chart.

    In your Helm values file:

    yaml 
    enterprise:
  enabled: true
  legacyLabels: true

    g. Latest version MinIO changes its credential fields from combination of accessKey and secretKey to rootUser and rootPassword. If you’re using the enterprise-metrics chart MinIO for object storage, set the minio.rootUser to enterprise-metrics and set the secret key or password to rootPassword.

    In your Helm values file:

    yaml 
    minio:
  rootUser: "enterprise-metrics"
  rootPassword: <password>

  3. Run the Helm upgrade with the mimir-distributed chart.

    During the upgrade, requests to the gateway fail for a short period of time. Prometheus and Grafana Agent retry failed write requests to avoid data loss.

    Note: The name of the release must match your enterprise-metrics Helm chart release.

    bash 
    helm upgrade <RELEASE> grafana/mimir-distributed --version 3.1.0 -f <VALUES FILE> --set-file 'license.contents=<LICENSE FILE>' [-n <NAMESPACE>]

To verify that the cluster is operating correctly, use the self monitoring or monitoring mixin dashboards installed in Grafana.