Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Grafana Enterprise Metrics Documentationbreadcrumb arrow Grafana Enterprise Metrics on AWS Marketplace
Enterprise RSS

Getting started with Grafana Enterprise Metrics on AWS Marketplace

Grafana Enterprise Metrics (GEM) on AWS Marketplace offers a simple and easy way to get production grade GEM clusters up and running in your own Amazon Elastic Kubernetes Service (EKS) cluster.

This page will guide you through the process of deploying GEM through the AWS Marketplace.

Prerequisites

In order to complete the steps described in this guide we require that:

  • You have access to an AWS account with sufficient permissions to subscribe to products on the AWS Marketplace as well as permissions to create an IAM role and policy for your service account. For more information, refer to Creating an IAM role and policy for your service account.
  • You have a properly sized, and correctly configured and running EKS cluster (Kubernetes 1.18 or higher).
  • You have Helm (3.6.0 or newer), the latest version of the AWS Command Line Interface and kubectl installed on your local machine pointing to the correct EKS cluster.

Step 1. Subscribe to GEM on the AWS Marketplace

  1. Log in to your AWS account.
  2. Subscribe to the Grafana Enterprise Metrics product on the AWS Marketplace; go to the product overview page, and click the Continue to Subscribe button in the top-right corner of the page.
  3. Accept the terms and conditions by selecting the Accept Terms button.
  4. After the process of subscribing completes, from the top-right corner of the page, select Continue to Configuration.
  5. From the Delivery Method drop-down menu on Configure this software page, select “Helm Chart”.
  6. From the Software Version drop-down menu, select a version.
  7. In the upper-right corner, select Continue to Launch. The Launch this software page displays and your subscription set-up process is complete for the Grafana Enterprise Metrics product.
  8. Select Usage Instructions.

Step 2. Prepare your Kubernetes cluster

Note: Replace any <PLACEHOLDER_VALUES> (including the <> brackets) with your own values:

  1. Create a Kubernetes namespace for your application using the kubectl command.

    sh 
    kubectl create namespace <NAMESPACE>

  2. Create the image pull secret for the Amazon Elastic Container Registry (ECR).

    sh 
    # get login credentials from AWS and store them in config.json
aws ecr get-login-password --region us-east-1 | docker --config . login --username AWS --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com
# use config.json to create a Kubernetes image pull secret
kubectl --namespace <NAMESPACE> create secret generic ecr --from-file=.dockerconfigjson=config.json --type=kubernetes.io/dockerconfigjson

  3. Set up a Kubernetes ServiceAccount, to allow containers in your EKS cluster to communicate to the relevant AWS services.

    For that, create an IAM policy that specifies the required permissions, and attach the policy to a role that is associated with the Kubernetes ServiceAccount under which the Pod runs.

    There are multiple ways to create the policy and the role. We recommend to follow the steps of the Amazon EKS User Guide using eksctl. Use marketplace-service-account as value for the --name parameter of the eksctl create iamserviceaccount command. This will be referenced later when configuring the Helm chart.

GEM utilizes the Marketplace Metering APIs to check whether your AWS account is subscribed to the Marketplace product and to report usage for billing. Therefore the Pod requires permissions for RegisterUsage and MeterUsage.

GEM also uses Amazon S3 as the storage backend for TSDB blocks and other meta information.

An example policy contains the required permissions:

json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowMeteringAccess",
      "Effect": "Allow",
      "Action": [
        "aws-marketplace:RegisterUsage",
        "aws-marketplace:MeterUsage"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-admin>",
        "arn:aws:s3:::<bucket-admin>/*",
        "arn:aws:s3:::<bucket-alertmanager>",
        "arn:aws:s3:::<bucket-alertmanager>/*",
        "arn:aws:s3:::<bucket-ruler>",
        "arn:aws:s3:::<bucket-ruler>/*",
        "arn:aws:s3:::<bucket-tsdb>",
        "arn:aws:s3:::<bucket-tsdb>/*"
      ]
    },
    {
      "Sid": "AllowListBuckets",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": "*"
    }
  ]
}

Warning: If you do not grant the permissions above your Grafana Enterprise Metrics installation will not function! Even though some services will continue to run, others won’t start and any enterprise features are disabled and only Cortex functionality is available.

Step 3. Prepare a Helm chart

Before continuing, be aware that as part of the Grafana Enterprise Metrics Helm chart many different pods and services are installed which all have varying resource requirements. Please make sure that you have sufficient resources (both CPU and memory) available in your EKS cluster or auto-scaling enabled before installing the software stack.

First, add the official grafana Helm repository and update so you have the latest versions of the charts.

sh 
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

The generic command to install GEM using Helm is:

sh 
helm [--kube-context CONTEXT] install [NAME] grafana/enterprise-metrics [--namespace NAMESPACE] [--values VALUES]

In the following instructions we omit the --kube-context option and assume that the correct context pointing to your desired EKS cluster is set. You can verify the context by invoking

sh 
kubectl config current-context

The output should match your EKS cluster environment.

Step 4. Configure your Helm chart

To account for different ingestion and querying requirements, the Helm chart contains configurations for different scale values for different cluster sizes. You can use one of the following configurations as is or as a starting point to configure your cluster size:

If you are unsure about sizing, use the capped-small configuration. However, use the capped-large configuration for production workloads. Only adjust values if you already have experience with GEM.

In addition to the scale values file, you need a local file that contains the AWS Marketplace license configuration and the custom S3 configuration for the storage backends. The contents of this file must contain all of the following configuration settings:

yaml 
image:
  # Version of Grafana Enterprise Metrics
  tag: "v1.7.0"
  repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/grafana-labs/grafana-enterprise-metrics
  pullSecrets:
    # name of the image pull secret created in "Step 2. Prepare your Kubernetes cluster"
    - ecr
serviceAccount:
  create: false
  name: marketplace-service-account
config:
  license:
    type: aws-marketplace
  # https://grafana.com/docs/enterprise-metrics/v1.7.x/config/reference/#admin_client_config
  admin_client:
    storage:
      type: "s3"
      s3:
        endpoint: "s3.<AWS_REGION>.amazonaws.com"
        bucket_name: "<S3_BUCKET_ADMIN>"
  # https://grafana.com/docs/enterprise-metrics/v1.7.x/config/reference/#alertmanager_storage_config
  alertmanager_storage:
    backend: "s3"
    s3:
      endpoint: "s3.<AWS_REGION>.amazonaws.com"
      bucket_name: "<S3_BUCKET_ALERTMANAGER>"
  # https://grafana.com/docs/enterprise-metrics/v1.7.x/config/reference/#blocks_storage_config
  blocks_storage:
    backend: "s3"
    s3:
      endpoint: "s3.<AWS_REGION>.amazonaws.com"
      bucket_name: "<S3_BUCKET_TSDB>"
  # https://grafana.com/docs/enterprise-metrics/v1.7.x/config/reference/#ruler_storage_config
  ruler_storage:
    backend: "s3"
    s3:
      endpoint: "s3.<AWS_REGION>.amazonaws.com"
      bucket_name: "<S3_BUCKET_RULER>"

You need to provide the S3_BUCKET_ADMIN, S3_BUCKET_ALERTMANAGER, S3_BUCKET_TSDB, S3_BUCKET_RULER and AWS_REGION configuration. Configure GEM to use different buckets for storing administration objects (admin_client), TSDB blocks (blocks_storage), rules (ruler_storage) and alertmanager objects (alertmanager_storage).

Also, replace the image.tag with the GEM version that you selected in the final step of the Marketplace subscription flow.

If you need to configure the deployment further, see Supported contents and default values of the config file.

Step 5. Install your Helm chart

Store the following YAML configuration in a local file, such as custom.yaml, so that it can be passed as values to the helm install command:

sh 
helm install <NAME> \
    grafana/enterprise-metrics \
    --namespace <NAMESPACE> \
    --values https://raw.githubusercontent.com/grafana/helm-charts/main/charts/enterprise-metrics/capped-large.yaml \
    --values custom.yaml \
    --no-hooks

Hint: You can download the capped-large.yaml or capped-small.yaml to make sure they are safe, and use a local path with the --values argument.

After deploying the resources using the helm install command it take a few minutes before all of the services are running. To verify that your installation succeeded, watch the rollout of the pods of deployments and statefulsets.

console 
$ kubectl get deployments
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
dev-enterprise-metrics-admin-api        1/1     1            1           13m
dev-enterprise-metrics-distributor      3/3     3            3           13m
dev-enterprise-metrics-gateway          1/1     1            1           13m
dev-enterprise-metrics-querier          1/1     1            1           13m
dev-enterprise-metrics-query-frontend   1/1     1            1           13m
dev-enterprise-metrics-ruler            1/1     1            1           13m
console 
$ kubectl get statefulsets
NAME                                   READY   AGE
dev-enterprise-metrics-alertmanager    1/1     14m
dev-enterprise-metrics-compactor       1/1     14m
dev-enterprise-metrics-ingester        3/3     14m
dev-enterprise-metrics-store-gateway   1/1     14m
dev-memcached                          2/2     14m
dev-memcached-metadata                 1/1     14m
dev-memcached-queries                  3/3     14m

Note: Depending on your cluster size, the number of replicas might differ from the preceding output.

After all Deployment and StatefulSet instances are ready, you can verify the integrity of the GEM cluster by querying metrics that are emitted from self-monitoring. Refer to self-monitoring verification.

The AWS Marketplace version of Grafana Enterprise Metrics does not contain a web-based frontend. If you need one, install Grafana Enterprise and the GEM plugin. Refer to Set up the GEM plugin for Grafana.

Billing

The GEM product is billed by consumption with a single usage dimension “CPU time”. CPU time is defined by the amount of provisioned CPU cores (value of GOMAXPROCS as exposed by Prometheus metric cortex_quota_gomaxprocs) times the hours that the product is running. Each Pod that runs as part of the product contributes to the overall usage, meaning that scaling up will also result in higher consumption.

The minimum CPU per container is 1, even if Kubernetes resource limits are set to a fraction of a CPU (below 1000m) for a Pod. If you don’t set any CPU resource limits on containers, GEM will use the number of physical CPU cores as reported by Golang’s GOMAXPROCS. If Kubernetes resource limits are set, GOMAXPROCS is automatically set to the next highest integer value of the CPU limit. If you do not want to set CPU limits on Kubernetes (kernel level), but still want to limit CPU usage, you can still override the GOMAXPROCS value by providing the GOMAXPROCS environment variable in the Helm chart deployment.