Transforming application logs into metrics with Istio and Grafana Cloud

Antonio Berben is a Field Engineer working for solo.io on Gloo Mesh (with Istio) and Gloo Edge (with Envoy). He is a Kubernetes enthusiast and his favorite quote is: “I will go anywhere, provided it be forward” from David Livingstone. You can find and follow Antonio on Twitter.

Kim Uyttebrouck is an experienced leader who has a proven track record of working with successful teams in both professional and personal life. He’s a DevOps enthusiast based in Leuven, Belgium, who likes to learn about new technologies. In his free time he trains for short distance triathlons.

Do you actually know what your customers are looking for?

A way to uncover new business opportunities is to analyze your system, collect what you really need, and visualize it through a comprehensive graph!

Log traces are a great place to start because they usually contain useful information on your customers’ interests. You just need to transform them.

This blog post will show you how to transform raw data from Istio access logs collected by Promtail into valuable metrics visualized through Grafana Cloud.

The scenario

Your e-commerce platform is selling different pairs of socks. Customers can filter the socks catalog based on their personal interest. You want to find out which pair of socks is most filtered by “formal” or “geek.”

Your architecture will look like this:

Hands-on setup!

Prerequisites:

• Kubernetes cluster - e.g. Kind
• Istioctl (this example uses version 1.12.1)
• Helm
• Grafana Cloud free account

First, let’s set up your Grafana Cloud stack:

• Create a free Grafana Cloud account if you don’t already have one.
• Sign up with existing credentials (Google, Github, etc), create your account, and obtain your own stack on the Grafana Cloud Portal.

• You will also need to create a Grafana Cloud API key with metrics push privileges!
• Click on “Send Metrics” next to Prometheus to navigate to Grafana Cloud’s Prometheus Configuration page.
• Scroll to find the Password/API key section.

• Click on the blue text “Generate now.” In the new window, make sure to select “MetricsPublisher” from the dropdown Role list when creating your API key.

• IMPORTANT note: Grafana Cloud’s Prometheus Configuration page also contains the following required information for being able to send Prometheus metrics to Grafana Cloud:

  • Remote Write Endpoint
  • Username / Instance ID

• Navigate back to the Grafana Cloud Portal and click on “Send Logs” next to Loki. 

On the “Using Grafana with Logs” page, scroll to find the Grafana Data Source settings. You will need the User value when deploying Promtail.

Now, on the Kubernetes cluster, install Helm repositories.

Copy

helm repo add grafana https://grafana.github.io/helm-charts
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Install Istio operator.

Copy

istioctl operator init
kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator
  namespace: istio-system
spec:
 profile: default
 meshConfig:
   accessLogFile: /dev/stdout
   accessLogFormat: |
     accesslog [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
EOF

Wait until Istio is up and running.

Install the sock-shop application. You can read more about the application design here.

Copy

kubectl create ns sock-shop
kubectl label namespace sock-shop istio-injection=enabled --overwrite
kubectl -n sock-shop apply -f  
https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/kubernetes/complete-demo.yaml 

Verify if all pods have Istio injected. (Each microservice pod should show an extra container which is the Istio sidecar.)

Copy

kubectl get pods -n sock-shop

Deploy an Istio gateway and virtual service.

Copy

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: sock-gateway
 namespace: sock-shop
spec:
 selector:
   istio: ingressgateway 
 servers:
 - port:
     number: 80
     name: http
     protocol: HTTP
   hosts:
   - "*" 
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: sock
 namespace: sock-shop
spec:
 hosts:
 - "*"
 gateways:
 - sock-gateway
 http:
 - match:
   - uri:
       prefix: "/"
   route:
   - destination:
       host: front-end.sock-shop.svc.cluster.local
       port:
         number: 80
EOF

Open a tunnel to the Istio ingress gateway (= service mesh entry-point) to access the sock-shop application.

Copy

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

The sock-shop application is now available at http://localhost:8080

Install Prometheus with remote_write configuration.

The required details can be found in Grafana Cloud’s Prometheus metrics documentation (see also Prerequisites section).

Copy

export GC_PROMETHEUS_USERNAME=<my-grafana-cloud-prometheus-username>
export GC_API_KEY=<my-grafana-cloud-api-key>
export REMOTE_WRITE_ENDPOINT=<my-grafana-cloud-remote-write-endpoint>

kubectl create ns logging

kubectl create secret generic kubepromsecret \
  --from-literal=username=${GC_PROMETHEUS_USERNAME} \
  --from-literal=password=${GC_API_KEY} \
  -n logging

helm upgrade --install prometheus -n logging bitnami/kube-prometheus --version 6.6.0 -f - <<EOF
prometheus:
  remoteWrite:
  - url: ${REMOTE_WRITE_ENDPOINT}
    basicAuth:
      username:
        name: kubepromsecret
        key: username
      password:
        name: kubepromsecret
        key: password
EOF

Install Promtail as daemonset.

Copy

export GC_LOKI_USER=<my-grafana-cloud-loki-user>
export GC_API_KEY=<my-grafana-cloud-api-key>
export GC_LOKI_URL=<my-grafana-cloud-loki-url>

kubectl create secret generic kubelokisecret \
  --from-literal=username=${GC_LOKI_USER} \
  --from-literal=password=${GC_API_KEY} \
  -n logging

helm upgrade --install promtail -n logging grafana/promtail --version 3.8.1 -f - <<EOF
extraArgs:
  - -config.expand-env=true 
extraEnvFrom:
  - secretRef:
      name: kubelokisecret
config:
  logLevel: debug
  serverPort: 9080 
  lokiAddress: "https://${username}:${password}@${GC_LOKI_URL}/loki/api/v1/push" 
  snippets:
    pipelineStages:
      - match:
          selector: '{app="istio-ingressgateway"}'
          stages:
          # sample: accesslog [xxxx-xx-xxxxx:xxxxxxxxxx] "GET /category.html?tags=formal"
          # sample: accesslog [xxxx-xx-xxxxx:xxxxxxxxxx] "GET /category.html?tags=geek"
          - regex:
              expression: ^.*GET \/category.html\?tags=(?P<filter>geek)"
          - regex:
              expression: ^.*GET \/category.html\?tags=(?P<filter>formal)"
          - labels:
              filter:
          - metrics:
              filter:
                prefix: 'analytics_'
                type: Counter
                description: "Total count of filtered results"
                source: filter
                config:
                  action: inc               
serviceMonitor:
  enabled: true
  namespaceSelector: {}
  annotations: {}
  labels: {}
  interval: 10s
  scrapeTimeout: 5s
EOF

Port-forward both Prometheus and Promtail.

Copy

kubectl -n logging port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
kubectl -n logging port-forward ds/promtail 9080:9080

The Grafana dashboard

Let’s have a look at some pairs of socks by browsing through the catalog on the website while filtering on either “formal” or “geek” (this will create the required traffic).

In your browser open http://localhost:9080/metrics to consult Promtail metrics. Search for analytics_filter related entries.

On http://localhost:9090 Prometheus will visualize the metrics in the graph section when executing analytics_filter. These entries are forwarded to Grafana Cloud.

Side note: An alternate way of creating the setup in this post would be to use the Grafana Agent, which bundles the Prometheus discovery, WAL and remote_write capabilities without the local TSDB, and also bundles Promtail. Check out the Grafana Agent Quickstart guide.

Let’s consult the data on Grafana Cloud! On the Grafana Cloud Portal, you can log into Grafana to start using it by clicking “Log In.” On the Grafana General/Home page, you can create a new dashboard to visualize the generated metrics.

Let’s add a panel to the newly created dashboard by clicking “Add a new panel.”

Next, you define a query. From the “Data source” pulldown menu select grafanacloud-< your-grafana-cloud-stack-name >-prom to set Prometheus as data source.

Nearly done. Define your query and possibly give the panel a title. Here you will use:

Copy

sum(rate(analytics_filter[30s])) by (filter)

Click on “Apply” (top right corner of the page). Here you go!

You can add another panel to visualize the logs. Select grafanacloud-< your-grafana-cloud-stack-name >-logs to set Loki as the data source. Choose either Table or Logs as Visualization and define your query e.g. {filter=”geek”}.

Final thoughts

So the end result allows you to understand how both “formal” and “geek” socks are trending. The visualization of these metrics will surely help you to better understand what customers are looking for and guide you towards better business decisions that generate more revenue.

How to turn application logs into useful metrics with Gloo Edge and Grafana is a real use case story where Promtail is leveraged by Gloo Edge to find corner cases on one of its components: ExtAuth. The metric is used to signal and alert about a misconfiguration.

Knowing this approach, you can leverage the same setup to get metrics from your logs and not just to obtain some sort of analytics, but to understand better how your system works for DevOps purposes.