Application Observability with Grafana Alloy
Grafana Alloy (formerly Grafana Agent) is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector and the recommended way to send OpenTelemetry data to Grafana Cloud. Grafana Alloy will act as an outbound gateway providing reliability and scalability.
Prerequisites
To set up Grafana Alloy as a data collector and send data to Grafana Cloud:
- Create and/or login to a Grafana Cloud account
- Install Grafana Alloy
We require that the Alloy is run on every host for Application Observability to function optimally. This also allows us to have a seamless correlation between Application and Infrastructure observability.
Configuration
An alloy-config.river
configuration file is needed for the Grafana Alloy to run successfully in flow mode. It is recommended to use the OpenTelemetry (OTLP) integration to generate a configuration file.
Navigate to the OpenTelemetry Application page and click the Add service button. Choose the OpenTelemetry (OTLP) integration from the list.
The OpenTelemetry (OTLP) integration will generate a Grafana Alloy configuration to use with Grafana Application Observability:
otelcol.receiver.otlp "default" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/
// configures the default grpc endpoint "0.0.0.0:4317"
grpc { }
// configures the default http/protobuf endpoint "0.0.0.0:4318"
http { }
output {
metrics = [otelcol.processor.resourcedetection.default.input]
logs = [otelcol.processor.resourcedetection.default.input]
traces = [otelcol.processor.resourcedetection.default.input]
}
}
otelcol.processor.resourcedetection "default" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.resourcedetection/
detectors = ["env", "system"] // add "gcp", "ec2", "ecs", "elastic_beanstalk", "eks", "lambda", "azure", "aks", "consul", "heroku" if you want to use cloud resource detection
system {
hostname_sources = ["os"]
}
output {
metrics = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
logs = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
traces = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
}
}
otelcol.processor.transform "drop_unneeded_resource_attributes" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
error_mode = "ignore"
trace_statements {
context = "resource"
statements = [
"delete_key(attributes, \"k8s.pod.start_time\")",
"delete_key(attributes, \"os.description\")",
"delete_key(attributes, \"os.type\")",
"delete_key(attributes, \"process.command_args\")",
"delete_key(attributes, \"process.executable.path\")",
"delete_key(attributes, \"process.pid\")",
"delete_key(attributes, \"process.runtime.description\")",
"delete_key(attributes, \"process.runtime.name\")",
"delete_key(attributes, \"process.runtime.version\")",
]
}
metric_statements {
context = "resource"
statements = [
"delete_key(attributes, \"k8s.pod.start_time\")",
"delete_key(attributes, \"os.description\")",
"delete_key(attributes, \"os.type\")",
"delete_key(attributes, \"process.command_args\")",
"delete_key(attributes, \"process.executable.path\")",
"delete_key(attributes, \"process.pid\")",
"delete_key(attributes, \"process.runtime.description\")",
"delete_key(attributes, \"process.runtime.name\")",
"delete_key(attributes, \"process.runtime.version\")",
]
}
log_statements {
context = "resource"
statements = [
"delete_key(attributes, \"k8s.pod.start_time\")",
"delete_key(attributes, \"os.description\")",
"delete_key(attributes, \"os.type\")",
"delete_key(attributes, \"process.command_args\")",
"delete_key(attributes, \"process.executable.path\")",
"delete_key(attributes, \"process.pid\")",
"delete_key(attributes, \"process.runtime.description\")",
"delete_key(attributes, \"process.runtime.name\")",
"delete_key(attributes, \"process.runtime.version\")",
]
}
output {
metrics = [otelcol.processor.transform.add_resource_attributes_as_metric_attributes.input]
logs = [otelcol.processor.batch.default.input]
traces = [
otelcol.processor.batch.default.input,
otelcol.connector.host_info.default.input,
]
}
}
otelcol.connector.host_info "default" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.host_info/
host_identifiers = ["host.name"]
output {
metrics = [otelcol.processor.batch.default.input]
}
}
otelcol.processor.transform "add_resource_attributes_as_metric_attributes" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
error_mode = "ignore"
metric_statements {
context = "datapoint"
statements = [
"set(attributes[\"deployment.environment\"], resource.attributes[\"deployment.environment\"])",
"set(attributes[\"service.version\"], resource.attributes[\"service.version\"])",
]
}
output {
metrics = [otelcol.processor.batch.default.input]
}
}
otelcol.processor.batch "default" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
output {
metrics = [otelcol.exporter.prometheus.grafana_cloud_prometheus.input]
logs = [otelcol.exporter.loki.grafana_cloud_loki.input]
traces = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
}
}
otelcol.exporter.loki "grafana_cloud_loki" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.loki/
forward_to = [loki.write.grafana_cloud_loki.receiver]
}
otelcol.exporter.prometheus "grafana_cloud_prometheus" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.prometheus/
add_metric_suffixes = false
forward_to = [prometheus.remote_write.grafana_cloud_prometheus.receiver]
}
prometheus.remote_write "grafana_cloud_prometheus" {
// https://grafana.com/docs/alloy/latest/reference/components/prometheus.remote_write/
endpoint {
url = env("GRAFANA_CLOUD_PROMETHEUS_URL")
basic_auth {
username = env("GRAFANA_CLOUD_PROMETHEUS_USERNAME")
password = env("GRAFANA_CLOUD_API_KEY")
}
}
}
loki.write "grafana_cloud_loki" {
// https://grafana.com/docs/alloy/latest/reference/components/loki.write/
endpoint {
url = env("GRAFANA_CLOUD_LOKI_URL")
basic_auth {
username = env("GRAFANA_CLOUD_LOKI_USERNAME")
password = env("GRAFANA_CLOUD_API_KEY")
}
}
}
otelcol.exporter.otlp "grafana_cloud_tempo" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlp/
client {
endpoint = env("GRAFANA_CLOUD_TEMPO_ENDPOINT")
auth = otelcol.auth.basic.grafana_cloud_tempo.handler
}
}
otelcol.auth.basic "grafana_cloud_tempo" {
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
username = env("GRAFANA_CLOUD_TEMPO_USERNAME")
password = env("GRAFANA_CLOUD_API_KEY")
}
The configuration file requires several environmental variable to be set:
Environment Variable | Description | Example |
---|---|---|
GRAFANA_CLOUD_API_KEY | API key generated above | eyJvSomeLongStringJ9fQ== |
GRAFANA_CLOUD_PROMETHEUS_URL | Remote Write Endpoint from Grafana Cloud > Prometheus > Details | https://prometheus-prod-***.grafana.net/api/prom/push |
GRAFANA_CLOUD_PROMETHEUS_USERNAME | Username/Instance ID from the Grafana Cloud > Prometheus > Details | 11111 |
GRAFANA_CLOUD_LOKI_URL | url (without the username and password) from Grafana Cloud > Loki > Details > From a standalone host | https://logs-prod-***.grafana.net/loki/api/v1/push |
GRAFANA_CLOUD_LOKI_USERNAME | User from Grafana Cloud > Loki > Details | 11112 |
GRAFANA_CLOUD_TEMPO_ENDPOINT | endpoint from the Grafana Cloud > Tempo > Details > Sending Data to Tempo | tempo-***.grafana.net:443 |
GRAFANA_CLOUD_TEMPO_USERNAME | User from Grafana Cloud > Tempo > Details | 11113 |
Run Grafana Alloy
Create the alloy-config.river
file, set the necessary environment variables, and start Grafana Alloy.
Language Specific Guides
Language specific guides that show how to instrument your application can be found in the production instrumentation guides.