Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Grafana Enterprise Metrics documentationbreadcrumb arrow Grafana Enterprise Metrics on AWS Marketplace
Enterprise RSS

Getting started with Grafana Enterprise Metrics on AWS Marketplace

Grafana Enterprise Metrics (GEM) on AWS Marketplace offers a simple and easy way to get production grade GEM clusters up and running in your own Amazon Elastic Kubernetes Service (EKS) cluster.

This page will guide you through the process of deploying GEM through the AWS Marketplace.

Prerequisites

In order to complete the steps described in this guide we require that:

  • You have access to an AWS account with sufficient permissions to subscribe to products on the AWS Marketplace as well as permissions to create an IAM role and policy for your service account. For more information, refer to Creating an IAM role and policy for your service account.
  • You have a properly sized, and correctly configured and running EKS cluster (Kubernetes 1.18 or higher).
  • You have Helm (3.6.0 or newer), the latest version of the AWS Command Line Interface and kubectl installed on your local machine pointing to the correct EKS cluster.

Step 1. Subscribe to GEM on the AWS Marketplace

  1. Log in to your AWS account.
  2. Subscribe to the Grafana Enterprise Metrics product on the AWS Marketplace; go to the product overview page, and click the Continue to Subscribe button in the top-right corner of the page.
  3. Accept the terms and conditions by selecting the Accept Terms button.
  4. After the process of subscribing completes, from the top-right corner of the page, select Continue to Configuration.
  5. From the Delivery Method drop-down menu on Configure this software page, select “Helm Chart”.
  6. From the Software Version drop-down menu, select a version.
  7. In the upper-right corner, select Continue to Launch. The Launch this software page displays and your subscription set-up process is complete for the Grafana Enterprise Metrics product.
  8. Select Usage Instructions.

Step 2. Prepare your Kubernetes cluster

Note: Replace any <PLACEHOLDER_VALUES> (including the <> brackets) with your own values:

  1. Create a Kubernetes namespace for your application using the kubectl command.

    sh 
    kubectl create namespace <NAMESPACE>

  2. Create the image pull secret for the Amazon Elastic Container Registry (ECR).

    sh 
    # get login credentials from AWS and store them in config.json
aws ecr get-login-password --region us-east-1 | docker --config . login --username AWS --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com
# use config.json to create a Kubernetes image pull secret
kubectl --namespace <NAMESPACE> create secret generic ecr --from-file=.dockerconfigjson=config.json --type=kubernetes.io/dockerconfigjson

  3. Set up a Kubernetes ServiceAccount, to allow containers in your EKS cluster to communicate to the relevant AWS services.

    For that, create an IAM policy that specifies the required permissions, and attach the policy to a role that is associated with the Kubernetes ServiceAccount under which the Pod runs.

    There are multiple ways to create the policy and the role. We recommend to follow the steps of the Amazon EKS User Guide using eksctl. Use marketplace-service-account as value for the --name parameter of the eksctl create iamserviceaccount command. This will be referenced later when configuring the Helm chart.

GEM utilizes the Marketplace Metering APIs to check whether your AWS account is subscribed to the Marketplace product and to report usage for billing. Therefore, the Pod requires permissions for RegisterUsage and MeterUsage.

GEM also uses Amazon S3 as the storage backend for TSDB blocks and other meta information.

An example policy contains the required permissions:

json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowMeteringAccess",
      "Effect": "Allow",
      "Action": ["aws-marketplace:RegisterUsage", "aws-marketplace:MeterUsage"],
      "Resource": "*"
    },
    {
      "Sid": "AllowBucketAccess",
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::<bucket-admin>",
        "arn:aws:s3:::<bucket-admin>/*",
        "arn:aws:s3:::<bucket-alertmanager>",
        "arn:aws:s3:::<bucket-alertmanager>/*",
        "arn:aws:s3:::<bucket-ruler>",
        "arn:aws:s3:::<bucket-ruler>/*",
        "arn:aws:s3:::<bucket-tsdb>",
        "arn:aws:s3:::<bucket-tsdb>/*"
      ]
    },
    {
      "Sid": "AllowListBuckets",
      "Effect": "Allow",
      "Action": ["s3:ListAllMyBuckets"],
      "Resource": "*"
    }
  ]
}

Warning: If you do not grant the permissions above your Grafana Enterprise Metrics installation will not function! Even though some services will continue to run, others won’t start and any enterprise features are disabled and only Grafana Mimir functionality is available.

Step 3. Prepare a Helm chart

Grafana Metrics Enterprise uses the same Helm chart as Grafana Mimir

  • grafana/mimir-distributed. Before continuing, be aware that as part of the mimir-distributed Helm chart pods and services will be installed which all have varying resource requirements. Please make sure that you have sufficient resources (both CPU and memory) available in your EKS cluster or auto-scaling enabled before installing the software stack.

First, add the official grafana Helm repository and update, so you have the latest versions of the charts.

sh 
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

In the following instructions we omit the --kube-context option from helm commands and assume that the correct context pointing to your desired EKS cluster is set. You can verify the context by invoking

sh 
kubectl config current-context

The output should match your EKS cluster environment.

Step 4. Configure your Helm chart

a. You need a local file that contains the configuration for the Helm chart. To do this start by creating a new file custom.yaml with the following contents:

yaml 
image:
  pullSecrets:
    # name of the image pull secret created in "Step 2. Prepare your Kubernetes cluster"
    - <IMAGE_PULL_SECRET>

serviceAccount:
  create: false
  name: marketplace-service-account

enterprise:
  enabled: true
  image:
    # Version of Grafana Enterprise Metrics
    tag: <IMAGE_TAG>
    repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/grafana-labs/grafana-enterprise-metrics

b. You also need to provide configuration for GEM itself. Start by extracting the default configuration from the mimir-distributed Helm chart into the mimir-distributed.yaml file. The following command requires the Go yq tool.

bash 
helm template <RELEASE NAME> -f <VALUES FILE> grafana/mimir-distributed | yq 'select(.metadata.name == "<RELEASE NAME>-enterprise-metrics-config") | .data."mimir.yaml"' | base64 -d > mimir-distributed.yaml

c. You need to override the storage settings to use S3. Create a new file with the S3 storage configuration storage.yaml:

yaml 
license:
  type: aws-marketplace
# https://grafana.com/docs/enterprise-metrics/v2.6.x/config/reference/#blocks_storage_config
blocks_storage:
  backend: "s3"
  s3:
    endpoint: "s3.<AWS_REGION>.amazonaws.com"
    bucket_name: "<S3_BUCKET_TSDB>"
# https://grafana.com/docs/enterprise-metrics/v2.6.x/config/reference/#admin_client_config
admin_client:
  storage:
    type: "s3"
    s3:
      endpoint: "s3.<AWS_REGION>.amazonaws.com"
      bucket_name: "<S3_BUCKET_ADMIN>"
# https://grafana.com/docs/enterprise-metrics/v2.6.x/config/reference/#alertmanager_storage_config
alertmanager_storage:
  backend: "s3"
  s3:
    endpoint: "s3.<AWS_REGION>.amazonaws.com"
    bucket_name: "<S3_BUCKET_ALERTMANAGER>"
# https://grafana.com/docs/enterprise-metrics/v2.6.x/config/reference/#ruler_storage_config
ruler_storage:
  backend: "s3"
  s3:
    endpoint: "s3.<AWS_REGION>.amazonaws.com"
    bucket_name: "<S3_BUCKET_RULER>"

d. Merge the files containing the default GEM configuration and the storage configuration. The resulting configuration is the configuration used for GEM itself. The following command requires the Go yq tool.

bash 
yq '. *= load("storage.yaml")' mimir-distributed.yaml > config.yaml

e. In custom.yaml, add the contents of the config.yaml file as the value for the mimir.config key.

Note: The Grafana Mimir Helm chart expects the configuration as a string value. You can provide a literal block string with the | symbol. In your Helm values file:

yaml 
mimir:
  config: |
    <CONFIG.YAML CONTENTS>

If you need to configure the deployment further, see Supported contents and default values of the config file.

f. Replace S3_BUCKET_ADMIN, S3_BUCKET_ALERTMANAGER, S3_BUCKET_TSDB, S3_BUCKET_RULER, AWS_REGION, and IMAGE_PULL_SECRET. Remember to use different buckets for storing administration objects (admin_client), TSDB blocks (blocks_storage), rules (ruler_storage) and alertmanager objects (alertmanager_storage).

Also, replace <IMAGE_TAG> key with the GEM version that you selected in the final step of the Marketplace subscription flow.

Step 5. Install your Helm chart

To account for different ingestion and querying requirements, the Helm chart contains configurations for different scale values for different cluster sizes. You can use one of the following configurations as is or as a starting point to configure your cluster size:

If you are unsure about sizing, use the capped-small configuration. However, use the capped-large configuration for production workloads. Only adjust values if you already have experience with GEM.

Run:

bash 
helm install <NAME> \
    grafana/mimir-distributed \
    --namespace <NAMESPACE> \
    --values https://raw.githubusercontent.com/grafana/mimir/main/operations/helm/charts/mimir-distributed/capped-large.yaml \
    --values custom.yaml \
    --no-hooks

After deploying the resources using the helm install command it can take a few minutes before all the services are running. To verify that your installation succeeded, watch the rollout of the pods of Deployments and StatefulSets.

console 
$ kubectl get deployments
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
<NAME>-mimir-gateway              2/2     2            2           72s
<NAME>-mimir-admin-api            1/1     1            1           72s
<NAME>-mimir-distributor          2/2     2            2           72s
<NAME>-mimir-overrides-exporter   1/1     1            1           72s
<NAME>-mimir-querier              1/1     1            1           72s
<NAME>-mimir-query-frontend       1/1     1            1           72s
<NAME>-mimir-ruler                1/1     1            1           72s
console 
$ kubectl get statefulsets
NAME                           READY   AGE
<NAME>-memcached-queries     2/2     9m34s
<NAME>-memcached-metadata    2/2     9m34s
<NAME>-memcached             2/2     9m34s
<NAME>-mimir-alertmanager    1/1     100s
<NAME>-mimir-compactor       1/1     100s
<NAME>-mimir-store-gateway   1/1     100s
<NAME>-mimir-ingester        3/3     100s

Note: Depending on your cluster size, the number of replicas might differ from the preceding output.

After all Deployment and StatefulSet instances are ready, you can verify the integrity of the GEM cluster by querying metrics that are emitted from self-monitoring. Refer to self-monitoring verification.

The AWS Marketplace version of Grafana Enterprise Metrics does not contain a web-based frontend. If you need one, install Grafana Enterprise and the GEM plugin. Refer to Set up the GEM plugin for Grafana.

Billing

The GEM product is billed by consumption with a single usage dimension “CPU time”. CPU time is defined by the amount of provisioned CPU cores (value of GOMAXPROCS as exposed by Prometheus metric cortex_quota_gomaxprocs) times the hours that the product is running. Each Pod that runs as part of the product contributes to the overall usage, meaning that scaling up will also result in higher consumption.

The minimum CPU per container is 1, even if Kubernetes resource limits are set to a fraction of a CPU (below 1000m) for a Pod. If you don’t set any CPU resource limits on containers, GEM will use the number of physical CPU cores as reported by Golang’s GOMAXPROCS. If Kubernetes resource limits are set, GOMAXPROCS is automatically set to the next highest integer value of the CPU limit. If you do not want to set CPU limits on Kubernetes (kernel level), but still want to limit CPU usage, you can still override the GOMAXPROCS value by providing the GOMAXPROCS environment variable in the Helm chart deployment.