Install Prometheus Operator with Grafana Cloud for Kubernetes
You can install Prometheus Operator in a Kubernetes Cluster, configure it to scrape an endpoint, and send scraped metrics to Grafana Cloud.
Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and Kubernetes controller code. Together, these abstract away the management and implementation details of running a given service on Kubernetes. For more about Kubernetes Operators, refer to the Operator pattern.
The Prometheus Operator installs a set of Kubernetes custom resources that simplify Prometheus deployment and configuration. For example, when you use the ServiceMonitor
custom resource, you can configure how to monitor Kubernetes services in Kubernetes YAML manifests instead of Prometheus configuration code. The Operator controller then communicates with the Kubernetes API server to add service /metrics
endpoints and automatically generate the required Prometheus scrape configurations for the configured services. To learn more, refer to Prometheus Operator.
Configuration steps
The steps for configuration are:
- Install Prometheus Operator into the Kubernetes Cluster.
- Configure RBAC permissions for Prometheus.
- Deploy Prometheus into the Cluster using the Operator.
- Expose the Prometheus server as a service.
- Create a ServiceMonitor.
Before you begin
Before you begin, have the following items available to you:
- A Kubernetes
>=1.16.0
Cluster with role-based access control (RBAC) enabled - A Grafana Cloud account
- A Grafana Cloud access policy token with the
metrics:write
scope. To create a Grafana Cloud access policy, refer to Create a Grafana Cloud Access Policy. - The
kubectl
command-line tool installed on your local machine and configured to connect to your Cluster. Refer to installkubectl
.
Install Prometheus Operator
Install Prometheus Operator into the Kubernetes Cluster. This includes all of Prometheus Operator’s Kubernetes custom resource definitions (CRDs) that define the Prometheus, Alertmanager, and ServiceMonitor abstractions used to configure the monitoring stack. In this step, you also deploy a Prometheus Operator controller into the Cluster.
Note
With these instructions, you install everything into thedefault
Namespace. To install Prometheus Operator into another Namespace, refer to the Prometheus Operator.
Install the Operator using the
bundle.yaml
file in the Prometheus Operator GitHub repository:kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
You should see the following output:
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created clusterrole.rbac.authorization.k8s.io/prometheus-operator created deployment.apps/prometheus-operator created serviceaccount/prometheus-operator created service/prometheus-operator created
bundle.yaml
installs CRDs for Prometheus objects as well as a Prometheus Operator controller and service.Note: With these instructions, you install everything into the
default
Namespace. To install Prometheus Operator into another Namespace, refer to the Prometheus Operator documentation.Verify that the Prometheus Operator installation succeeded by using
kubectl
:kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 3m21s
Configure Prometheus RBAC Permissions
Configure RBAC privileges for the Prometheus server using a ClusterRole. Then bind this ClusterRole to a ServiceAccount using a ClusterRoleBinding object.
Prometheus needs Kubernetes API access to discover targets and pull ConfigMaps. For more about permissions granted in this section, refer to RBAC.
Create a directory to store any Kubernetes manifests used in these instructions, and
cd
into it:mkdir operator_k8s cd operator_k8s
Create a manifest file called
prom_rbac.yaml
using an editor, and paste in the following Kubernetes manifest:apiVersion: v1 kind: ServiceAccount metadata: name: prometheus --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/metrics - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: [""] resources: - configmaps verbs: ["get"] - apiGroups: - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: default
This creates a ServiceAccount called
prometheus
and binds it to theprometheus
ClusterRole. The manifest grants the ClusterRoleget
,list
, andwatch
Kubernetes API privileges.Save and close the manifest when editing is complete.
Create the objects using
kubectl
:kubectl apply -f
serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created
Now that Prometheus has Kubernetes API access, you can deploy it into the Cluster.
Deploy Prometheus
In this step, you launch a 2-replica HA Prometheus deployment into your Kubernetes Cluster using a Prometheus
resource defined by Prometheus Operator. This Prometheus
resource encodes domain-specific Prometheus configuration into a set of readily configurable YAML fields. Instead of managing Prometheus configuration files and learn Prometheus configuration syntax, you can toggle many important configuration parameters by modifying Prometheus
object variables in a Kubernetes manifest.
Create a file called
prometheus.yaml
, and paste in the following manifest:apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {}
Notice that
kind
is set toPrometheus
and notDeployment
orPod
. ThePrometheus
resource abstracts away the underlying controllers and ConfigMaps into streamlined objects used to manipulate Prometheus infrastructure components.The manifest:
- Names the
Prometheus
resourceprometheus
. - Gives it an
app: prometheus
label. - Sets the container image used to run Prometheus.
- Restricts its deployment to Linux nodes.
- Ensures each replica has
400Mi
of memory available to it. - Configures its Security Context. To learn more, refer to Configure a Security Context for a Pod or Container.
- Sets the ServiceAccount it will use.
- Sets the Prometheus version.
- Instructs Prometheus to automatically pick up all configured
ServiceMonitor
resources using{}
. You’ll create aServiceMonitor
to get Prometheus to scrape its own metrics.
- Names the
Save and close the file.
Deploy the manifest into your Cluster using
kubectl apply -f
:kubectl apply -f
prometheus.monitoring.coreos.com/prometheus created
Note: You deployed Prometheus into the
default
Namespace. To deploy Prometheus in another Namespace, use the-n namespace_name
flag withkubectl
or set thenamespace
field for the resource in a Kubernetes manifest file.Verify the deployment using
kubectl get
:kubectl get prometheus
NAME VERSION REPLICAS AGE prometheus v2.22.1 2 32s
Check the underlying Pods using
get pod
:kubectl get pod
NAME READY STATUS RESTARTS AGE prometheus-operator-79cd654746-mdfp6 1/1 Running 0 33m prometheus-prometheus-0 2/2 Running 1 57s prometheus-prometheus-1 2/2 Running 1 57s
Prometheus is up and running in the Cluster.
Create a Prometheus service
Expose Prometheus using a service. To create the Prometheus service:
Open a manifest file called
prom_svc.yaml
and paste in the following definitions:apiVersion: v1 kind: Service metadata: name: prometheus labels: app: prometheus spec: ports: - name: web port: 9090 targetPort: web selector: app.kubernetes.io/name: prometheus sessionAffinity: ClientIP
This manifest configures:
- Sets the service name to
prometheus
. - Creates an
app: prometheus
Service
label. - Exposes port
9090
on a Cluster-wide stable IP address and forwards it to the two Prometheus Pods at their defaultweb
port (9090
). - Selects the Prometheus Pods as targets using the
app: prometheus
label. - Uses
sessionAffinity: ClientIP
to ensure that connections from a particular client get forwarded to the same Pod.
- Sets the service name to
Save and close the file.
Create the service using
kubectl apply -f
:kubectl apply -f prom_svc.yaml
service/prometheus created
Check your work using
kubectl get
:kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 27h prometheus ClusterIP 10.245.106.105 <none> 9090/TCP 26h prometheus-operated ClusterIP None <none> 9090/TCP 8m52s prometheus-operator ClusterIP None <none> 8080/TCP 41m
The
prometheus
service is up and running.To access the Prometheus server, forward a local port to the Prometheus service running inside of the Kubernetes Cluster:
kubectl port-forward svc/prometheus 9090
Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090
Navigate to
http://localhost:9090
to access the Prometheus interface:Click Status, then Targets to see any configured scrape targets.
This should be empty, as nothing is configured to scrape.
Create a Prometheus ServiceMonitor
Create a ServiceMonitor so that the Prometheus server scrapes its own metrics endpoint. A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource.
Instead of modifying a Prometheus configuration file, update a ConfigMap object and deploy the new configuration so that Prometheus Operator automatically hooks in new ServiceMonitors to your running Prometheus deployment.
To create the ServiceMonitor:
Create a file called
prometheus_servicemonitor.yaml
, and paste in the following ServiceMonitor resource definition:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: prometheus-self labels: app: prometheus spec: endpoints: - interval: 30s port: web selector: matchLabels: app: prometheus
The resource definition configures the following:
- Names the ServiceMonitor
prometheus-self
. - Gives it an
app: prometheus
label. - Sets the scrape interval to
30s
, and scrapes theweb
port (defined in theprometheus
Service
). - Selects the
prometheus
Service
to scrape using thematchLabels
selector withapp: prometheus
.
- Names the ServiceMonitor
Save and close the file.
Deploy the using
kubectl apply -f
:kubectl apply -f prometheus_servicemonitor.yaml
servicemonitor.monitoring.coreos.com/prometheus-self created
Prometheus Operator should update your Prometheus configuration immediately. In some cases, you may need to wait a minute or so for changes to propagate.
Verify your updates
Forward a port to your Prometheus server and check its configuration for verification.
And replace with this one?
Find the name of the Prometheus service. This will be in the format <your_release_name>-kube-prometheus-stack-prometheus, but could be truncated to fit naming length restrictions.
kubectl --namespace default get service
Use port-forward with that Prometheus service by running the following command:
kubectl --namespace default port-forward svc/<prometheus service name> 9090
Navigate to Status, and then Targets in the Prometheus interface:
You should see the two Prometheus replicas as scrape targets.
Navigate to Graph to test metrics collection:
In the Expression box, type
prometheus_http_requests_total
, and pressENTER
.You should see a list of scraped metrics and their values. These are HTTP request counts for various Prometheus server endpoints.
If you’re having trouble configuring a ServiceMonitor, refer to Troubleshooting ServiceMonitor Changes.
Now you have configured Prometheus to scrape itself and store metrics locally.
Create a Kubernetes Secret to store Grafana Cloud credentials
Configure Prometheus to send the scraped metrics to Grafana Cloud. Before configuring Prometheus’s remote_write
feature to send metrics to Grafana Cloud, you must create a Kubernetes Secret to store your Grafana Cloud Metrics username and password.
To create the Secret:
Navigate to your stack in the Cloud Portal, and click Details next to the Prometheus panel to find your username.
Your password corresponds to a Cloud Access Policy token that you can generate by clicking on Generate now in this same panel. To create a Cloud Access Policy, refer to Create a Grafana Cloud Access Policy.
After noting your Cloud Prometheus username and password, you can create the Kubernetes Secret using a manifest file or directly using
kubectl
. With these instructions, you create it directly usingkubectl
. To learn more about Kubernetes Secrets, refer to Secrets.Run the following command to create a Secret called
kubepromsecret
:kubectl create secret generic kubepromsecret \ --from-literal=username=<your_grafana_cloud_prometheus_username>\ --from-literal=password='<your_grafana_cloud_access_policy_token>'
Note: If you deployed your monitoring stack in a namespace other than
default
, append the-n
flag with the appropriate namespace to the above command.For more about this command, refer to Managing Secrets using kubectl from the official Kubernetes documentation.
Configure Prometheus remote_write and metrics deduplication
Configure remote_write
to send Cluster metrics to Grafana Cloud and to deduplicate metrics. The remote_write
Prometheus feature allows you to send metrics to remote endpoints for long-term storage and aggregation. Grafana Cloud’s deduplication feature allows you to deduplicate metrics sent from high-availability Prometheus pairs, which reduces your active series usage. You enable remote_write
by modifying the prometheus
resource you created earlier.
To configure Prometheus remote_write and metrics deduplication:
Open the
prometheus.yaml
manifest using your editor:apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {}
Add the following section to the end of the resource definition:
. . . remoteWrite: - url: "<Your Metrics instance remote_write endpoint>" basicAuth: username: name: kubepromsecret key: username password: name: kubepromsecret key: password replicaExternalLabelName: "__replica__" externalLabels: cluster: "<choose_a_prom_cluster_name>"
The resource definition configures the following:
- Sets the
remote_write
URL corresponding to Grafana Cloud’s Prometheus metrics endpoint. To find the/api/prom/push
URL, username, and password for your metrics endpoint, click Details in the Prometheus card of the Cloud Portal. - Configures a
basicAuth
username and password that references the Secret created in the earlier step namedkubepromsecret
- Selects the
username
andpassword
keys of this Secret - Configures Grafana Cloud metrics deduplication using the
replicaExternalLabelName
andexternalLabels
fields - Sets
cluster
to a value that identifies your Prometheus HA Cluster. To learn more, refer to Deduplicating metrics data sent from high-availability Prometheus pairs.
- Sets the
Save and close the file.
Apply the changes using
kubectl apply -f
:kubectl apply -f prometheus.yaml
prometheus.monitoring.coreos.com/prometheus configured
At this point, you’ve successfully configured your Prometheus instances to
remote_write
scraped metrics to Grafana Cloud.Verify that your changes have propagated to your running Prometheus instances using
port-forward
:kubectl port-forward svc/prometheus 9090
Navigate to
http://localhost:9090
in your browser, and then Status and Configuration. Verify that theremote_write
andexternal_labels
blocks you appended earlier have propagated to your running Prometheus instances.It may take a minute or two for Prometheus Operator to pick up the new configuration.
In the final step, you’ll query your Cluster metrics in Grafana Cloud.
Access your Prometheus metrics in Grafana Cloud
Now that your Prometheus instances send their scraped metrics to Grafana Cloud, you can query and visualize these metrics from the Grafana Cloud platform.
From the Cloud Portal, click Log In next to the Grafana card to log in to Grafana.
Click Explore in the left-side menu.
In the PromQL query box, enter the same metric you tested earlier,
prometheus_http_requests_total
, and pressSHIFT + ENTER
.You should see a graph of time-series data corresponding to different labels of the
prometheus_http_requests_total
metric. Grafana queries this data from the Grafana Cloud Metrics data store, not your local Cluster.Navigate to Kubernetes Monitoring, and click Configuration on the main menu.
Click the Metrics status tab to view the data status. Your data begins populating in the view as the system components begin scraping and sending data to Grafana Cloud.
Next steps
Now that you have installed and configured Prometheus Operator, you can:
- Create dashboards and panels to visualize and alert on this data. To learn more, refer to Dashboard overview and Panel overview.
- Add additional endpoints to scrape, such as the Kubernetes API or
kubelet
metrics from the Kubernetes nodes. To see a fully configured Prometheus Kubernetes stack in action, refer to kube-prometheus.