This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.
Profile Grafana Alloy resource consumption
Alloy is written in the Go programming language, which offers built-in support for profiling. Like other applications written in Go, you can profile Alloy by sending an HTTP request, which returns a pprof Go profile.
Once you have the pprof file, visualize it as a flame graph in Grafana Pyroscope. Alternatively, you could visualize it on Grafana Play or locally by using Go’s built-in pprof tool.
Note
A profile may contain sensitive information about your environment. You may not want to upload your profiles to a public location.
The port you use to send the HTTP request is controlled by the --server.http.listen-addr
command line argument.
It is set to 127.0.0.1:12345
by default.
Obtaining a single profile
Different types of HTTP requests will retrieve different profiles.
Memory consumption
Goroutine leaks often cause memory leaks. This is why obtaining a goroutine profile is usually necessary when investigating memory issues. For example:
curl localhost:12345/debug/pprof/heap -o heap.pprof
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof
It is often helpful to collect profiles both when memory usage is low and when it is high. You can compare the profiles, and it may be easier to identify what caused the memory consumption to increase.
CPU consumption
If you are experiencing high CPU consumption, you can collect a CPU profile:
curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof
The ?seconds=30
part of the URL above means the profiling will continue for 30 seconds.
Continuous profiling
You don’t have to send manual curl
commands each time you want to collect profiles.
You can also profile continuously using the pyroscope components in Alloy.
If you have very few Alloy instances, you can even configure them to profile themselves. However, if you have a large cluster of collectors, it is best to set up Alloy instances whose sole job is to profile other Alloy instances.
The following is an example of an Alloy instance profiling itself:
pyroscope.scrape "default" {
targets = [{"__address__" = "localhost:12345", "service_name"="alloy"}]
forward_to = [pyroscope.write.default.receiver]
}
pyroscope.write "default" {
endpoint {
url = "https://profiles-prod-014.grafana.net"
basic_auth {
username = sys.env("PYROSCOPE_USERNAME")
password = sys.env("PYROSCOPE_PASSWORD")
}
}
}
Expected resource consumption
Refer to Estimate resource usage for more information about the expected resource consumption in Alloy.
Alloy consumes an abnormally large amount of resources
If Alloy consumes an abnormally large amount of resources, you can open an issue in the Alloy repository. Attach your pprof files and your Alloy configuration file. Make sure you redact any secrets in the attachments.