Set up Go profiling in pull mode
In pull mode, the collector, Grafana Alloy, periodically retrieves profiles from Golang applications, specifically targeting the
/debug/pprof/*
endpoints.
To set up Golang profiling in pull mode, you need to:
- Expose pprof endpoints.
- Install a collector, such as Grafana Alloy.
- Prepare the collector’s configuration file.
- Start the collector.
Expose pprof endpoints
Ensure your Golang application exposes pprof endpoints.
Get
godeltaprof
packagego get github.com/grafana/pyroscope-go/godeltaprof@latest
Import
net/http/pprof
andgodeltaprof/http/pprof
packages at the start of your application.import _ "net/http/pprof" import _ "github.com/grafana/pyroscope-go/godeltaprof/http/pprof"
Install the collector
You can use the sample Alloy collector configuration file to send data to Pyroscope.
To install Alloy, refer to Grafana Alloy installation.
Prepare the collector configuration file
In the Alloy configuration file, you need to add at least two blocks: pyroscope.write
and pyroscope.scrape
.
Add
pyroscope.write
block.pyroscope.write "write_job_name" { endpoint { url = "http://localhost:4040" } }
Add
pyroscope.scrape
block.pyroscope.scrape "scrape_job_name" { targets = [{"__address__" = "localhost:4040", "service_name" = "example_service"}] forward_to = [pyroscope.write.write_job_name.receiver] profiling_config { profile.process_cpu { enabled = true } profile.godeltaprof_memory { enabled = true } profile.memory { // disable memory, use godeltaprof_memory instead enabled = false } profile.godeltaprof_mutex { enabled = true } profile.mutex { // disable mutex, use godeltaprof_mutex instead enabled = false } profile.godeltaprof_block { enabled = true } profile.block { // disable block, use godeltaprof_block instead enabled = false } profile.goroutine { enabled = true } } }
Save the changes to the file.
Start the collector
Start a local Pyroscope instance for testing purposes:
docker run -p 4040:4040 grafana/pyroscope
To start Alloy v1.2 and later: Replace
configuration.alloy
with your configuration filename:alloy run configuration.alloy
Open a browser to
http://localhost:4040
. The page should list profiles.
Examples
Send data to Grafana Cloud
Your Grafana Cloud URL, username, and password can be found on the “Details Page” for Pyroscope from your stack on grafana.com. On this same page, create a token and use it as the Basic authentication password.
pyroscope.write "write_job_name" {
endpoint {
url = "<Grafana Cloud URL>"
basic_auth {
username = "<Grafana Cloud User>"
password = "<Grafana Cloud Password>"
}
}
}
Discover Kubernetes targets
- Select all pods
discovery.kubernetes "all_pods" {
role = "pod"
}
Drop not running pods, create
namespace
,pod
,node
andcontainer
labels. Composeservice_name
label based onnamespace
andcontainer
labels. Select only services matching regex pattern(ns1/.*)|(ns2/container-.*0)
.discovery.relabel "specific_pods" { targets = discovery.kubernetes.all_pods.targets rule { action = "drop" regex = "Succeeded|Failed|Completed" source_labels = ["__meta_kubernetes_pod_phase"] } rule { action = "replace" source_labels = ["__meta_kubernetes_namespace"] target_label = "namespace" } rule { action = "replace" source_labels = ["__meta_kubernetes_pod_name"] target_label = "pod" } rule { action = "replace" source_labels = ["__meta_kubernetes_node_name"] target_label = "node" } rule { action = "replace" source_labels = ["__meta_kubernetes_pod_container_name"] target_label = "container" } rule { action = "replace" regex = "(.*)@(.*)" replacement = "${1}/${2}" separator = "@" source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"] target_label = "service_name" } rule { action = "keep" regex = "(ns1/.*)|(ns2/container-.*0)" source_labels = ["service_name"] } }
Use
discovery.relabel.specific_pods.output
as a target forpyroscope.scrape
block.pyroscope.scrape "scrape_job_name" { targets = discovery.relabel.specific_pods.output ... }
Exposing pprof endpoints
If you don’t use http.DefaultServeMux
, you can register /debug/pprof/*
handlers to your own http.ServeMux
:
var mux *http.ServeMux
mux.Handle("/debug/pprof/", http.DefaultServeMux)
Or, if you use gorilla/mux:
var router *mux.Router
router.PathPrefix("/debug/pprof").Handler(http.DefaultServeMux)