Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.
Release notes
The release notes provide information about deprecations and breaking changes in Grafana Agent static mode.
For a complete list of changes to Grafana Agent, with links to pull requests and related issues when available, refer to the Changelog.
Note: These release notes are specific to Grafana Agent static mode. Other release notes for the different Grafana Agent variants are contained on separate pages:
v0.38
Breaking change: support for exporting Jaeger traces removed
The deprecated support for exporting Jaeger-formatted traces has been removed. To send traces to Jaeger, export OTLP-formatted data to a version of Jaeger that supports OTLP.
v0.37
Breaking change: The default value of retry_on_http_429
is overriden to true
for the queue_config
in remote_write
in metrics
config.
Note
The default set by Grafana Agent Static Mode is different than the default set by Prometheus.
The Prometheus default value for retry_on_http_429
is set to true
for the queue_config
in remote_write
.
This changed default setting allows the agent to retry sending data when it receives an HTTP 429 error and helps avoid losing data in metric pipelines.
- If you explicitly set the
retry_on_http_429
, no action is required. - If you do not explicitly set
retry_on_http_429
and you do not want to retry on HTTP 429, make sure you set it tofalse
when you upgrade to this new version.
Breaking change: Renamed non_indexed_labels
Loki processing stage to structured_metadata
.
If you use the Loki processing stage in your Agent configuration, you must rename the non_indexed_labels
pipeline stage definition to structured_metadata
.
Old configuration example:
pipeline_stages:
- logfmt:
mapping:
app:
- non_indexed_labels:
app:
New configuration example:
pipeline_stages:
- logfmt:
mapping:
app:
- structured_metadata:
app:
v0.35
Breaking change: Jaeger remote sampling no longer configurable using the Jaeger receiver
Jaeger remote sampling used to be configured using the Jaeger receiver configuration. This receiver was updated to a new version, where support for remote sampling in the receiver was removed.
Jaeger remote sampling is available as a separate configuration field starting in v0.35.3.
Old configuration example:
receivers:
jaeger:
protocols:
grpc:
remote_sampling:
strategy_file: /etc/agent/strategies.json
strategy_file_reload_interval: 1s
New configuration example:
jaeger_remote_sampling:
- source:
file: /etc/agent/strategies.json
reload_interval: 1s
Breaking change: auth
and version
attributes from walk_params
block of SNMP integration have been removed
The SNMP integrations (both v1 and v2) wrap a new version of SNMP exporter which introduces a new configuration file format. This new format separates the walk and metric mappings from the connection and authentication settings. This allows for easier configuration of different auth params without having to duplicate the full walk and metric mapping.
Old configuration example:
snmp_targets:
- name: network_switch_1
address: 192.168.1.2
module: if_mib
walk_params: public
auth: public
walk_params:
public:
retries: 2
version: 2
auth:
community: public
New configuration example:
snmp_targets:
- name: network_switch_1
address: 192.168.1.2
module: if_mib
walk_params: public
auth: public
walk_params:
public:
retries: 2
See Module and Auth Split Migration for more details.
Removal of Dynamic Configuration
The experimental feature Dynamic Configuration has been removed. The use case of dynamic configuration will be replaced with Modules in Grafana Agent Flow.
Breaking change: Removed and renamed tracing metrics
In the traces subsystem for Static mode some metrics are removed and others are renamed. The reason for the removal is a bug which caused the metrics to be incorrect if more than one instance of a traces configuration is specified.
Removed metrics:
- “blackbox_exporter_config_last_reload_success_timestamp_seconds” (gauge)
- “blackbox_exporter_config_last_reload_successful” (gauge)
- “blackbox_module_unknown_total” (counter)
- “traces_processor_tail_sampling_count_traces_sampled” (counter)
- “traces_processor_tail_sampling_new_trace_id_received” (counter)
- “traces_processor_tail_sampling_sampling_decision_latency” (histogram)
- “traces_processor_tail_sampling_sampling_decision_timer_latency” (histogram)
- “traces_processor_tail_sampling_sampling_policy_evaluation_error” (counter)
- “traces_processor_tail_sampling_sampling_trace_dropped_too_early” (counter)
- “traces_processor_tail_sampling_sampling_traces_on_memory” (gauge)
- “traces_receiver_accepted_spans” (counter)
- “traces_receiver_refused_spans” (counter)
- “traces_exporter_enqueue_failed_log_records” (counter)
- “traces_exporter_enqueue_failed_metric_points” (counter)
- “traces_exporter_enqueue_failed_spans” (counter)
- “traces_exporter_queue_capacity” (gauge)
- “traces_exporter_queue_size” (gauge)
Renamed metrics:
- “traces_receiver_refused_spans” is renamed to “traces_receiver_refused_spans_total”
- “traces_receiver_accepted_spans” is renamed to “traces_receiver_refused_spans_total”
- “traces_exporter_sent_metric_points” is renamed to “traces_exporter_sent_metric_points_total”
v0.33
Symbolic links in Docker containers removed
We’ve removed the deprecated symbolic links to /bin/agent*
in Docker
containers, as planned in v0.31. In case you’re setting a custom entrypoint,
use the new binaries that are prefixed with /bin/grafana*
.
Deprecation of Dynamic Configuration
Dynamic Configuration will be removed in v0.34. The use case of dynamic configuration will be replaced with Modules in Grafana Agent Flow.
v0.32
Breaking change: node_exporter
configuration options changed
With the update of the node_exporter
integration to use v1.5.0, configuration
options for the diskstats
collector have changed names:
diskstats_ignored_devices
is nowdiskstats_device_exclude
in the static mode configuration.ignored_devices
is nowdevice_exclude
in the Flow component configuration.
v0.31.1
Breaking change: all Windows executables are now zipped
All release Windows .exe
files are now zipped. Prior to v0.31, only
grafana-agent-installer.exe
was unzipped.
This fixes an issue from v0.31.0 where all .exe
files were accidentally left
unzipped.
v0.31
Breaking change: binary names are now prefixed with grafana-
As first announced in v0.29, release binary names are now prefixed with
grafana-
:
agent
is nowgrafana-agent
.agentctl
is nowgrafana-agentctl
.
For the grafana/agent
Docker container, the entrypoint is now
/bin/grafana-agent
. A symbolic link from /bin/agent
to the new binary has
been added.
For the grafana/agentctl
Docker container, the entrypoint is now
/bin/grafana-agentctl
. A symbolic link from /bin/agentctl
to the new binary
has been added.
These symbolic links will be removed in v0.33. Custom entrypoints must be updated prior to v0.33 to use the new binaries before the symbolic links get removed.
v0.30
Breaking change: ebpf_exporter
integration removed
The ebpf_exporter
version bundled in the Agent used bcc to compile eBPF
programs at runtime. This made it hard to run successfully, as the
dynamic linking approach required a compiler, the correct kernel headers, as
well as an exact match of the libbpf toolchain on the host system. For these
reasons, we’ve decided to remove the ebpf_exporter
integration.
Running the ebpf_exporter
integration is now deprecated and will result in
configuration errors. To continue using the same configuration file, remove the
ebpf
block.
v0.29
Breaking change: JSON-encoded traces from OTLP versions below 0.16.0 are no longer supported
Grafana Agent’s OpenTelemetry Collector dependency has been updated from v0.55.0 to v0.61.0. OpenTelemetry Collector v0.58.0 no longer translates from InstrumentationLibrary to Scope.
This means that JSON-encoded traces that still use InstrumentationLibrary will be dropped. To work around this issue, either send traces using protobuf or update your OTLP protocol version to v0.16.0 or newer.
Deprecation: binary names will be prefixed with grafana-
in v0.31.0
The binary names agent
and agentctl
have been deprecated
and will be renamed to grafana-agent
and grafana-agentctl
respectively in
the v0.31.0 release.
As part of this change, the Docker containers for the v0.31.0 release will include symbolic links from the old binary names to the new binary names.
There is no action to take at this time.
v0.26
Breaking change: Deprecated YAML fields in server
block removed
The YAML fields which were first deprecated in the v0.24.0 release have now been removed, replaced by equivalent command line flags. Please refer to the original deprecation notice for instructions for how to migrate to the command line flags.
Breaking change: Reconcile sampling policies between Agent and OTel
Configuring sampling policies in the tail_sampling
block of the traces
block has been changed to be equal with the upstream configuration of the OTel
processor. It now requires that the policy type
is specified.
Old configuration:
traces:
configs:
- name: default
...
tail_sampling:
policies:
- latency:
threshold_ms: 100
New configuration:
traces:
configs:
- name: default
...
tail_sampling:
policies:
- type: latency
latency:
threshold_ms: 100
v0.24
Breaking change: Integrations renamed when integrations-next
feature flag is used
This change only applies to users utilizing the integrations-next
feature
flag. Nothing is changed for configuring integrations when the feature flag is
not used.
Most integrations-next
integrations have been renamed to describe what
telemetry data they generate instead of the projects they are powered by.
consul_exporter
is nowconsul
dnsmasq_exporter
is nowdnsmasq
elasticsearch_exporter
is nowelasticsearch
github_exporter
is nowgithub
kafka_exporter
is nowkafka
memcached_exporter
is nowmemcached
mongodb_exporter
is nowmongodb
mysqld_exporter
is nowmysql
- Note that it is
mysql
and notmysqld
- Note that it is
postgres_exporter
is nowpostgres
process_exporter
is nowprocess
redis_exporter
is nowredis
statsd_exporter
is nowstatsd
windows_exporter
is nowwindows
Keys in the integrations
config block have changed to match the above:
integrations.consul_exporter_configs
is nowintegrations.consul_configs
integrations.dnsmasq_exporter_configs
is nowintegrations.dnsmasq_configs
integrations.elasticsearch_exporter_configs
is nowintegrations.elasticsearch_configs
integrations.github_exporter_configs
is nowintegrations.github_configs
integrations.kafka_exporter_configs
is nowintegrations.kafka_configs
integrations.memcached_exporter_configs
is nowintegrations.memcached_configs
integrations.mongodb_exporter_configs
is nowintegrations.mongodb_configs
integrations.mysqld_exporter_configs
is nowintegrations.mysql_configs
integrations.postgres_exporter_configs
is nowintegrations.postgres_configs
integrations.process_exporter
is nowintegrations.process
integrations.redis_exporter_configs
is nowintegrations.redis_configs
integrations.statsd_exporter
is nowintegrations.statsd
integrations.windows_exporter
is nowintegrations.windows
Integrations not listed here have not changed; node_exporter
still has the
same name.
This change propagates to the label values generated by these integrations. For
example, job="integrations/redis_exporter
will now be job="redis"
.
Change: Separating YAML and command line flags
As of this release, we are starting to separate what can be configured within the YAML file, and what can be configured by command line flag. Previously, there was a lot of overlap: many things could be set by both command line flag and configuration file, with command line flags taking precedence.
The configuration file will be used for settings that can be updated at runtime
using the /-/reload
endpoint or sending SIGHUP. Meanwhile, command line flags
will be used for settings that must remain consistent throughout the process
lifetime, such as the HTTP listen port.
This conceptual change will require some number of breaking changes. This
release focuses on the server
block of the YAML, which has historically
caused the most issues with the /-/reload
endpoint working correctly.
There may be more breaking changes in the future as we identify more settings that must be static and moved to flags. These changes will either be moving a YAML field to a flag or moving a flag to a YAML field. After we are done with this migration, there will be no overlap between flags and the YAML file.
Deprecation on YAML fields in server
block that have flags
The server
block is the most impacted by the separation of flags/fields.
Instead of making a breaking change immediately, we are deprecating these
fields.
NOTE: These deprecated fields will be removed in the v0.26.0 release. We will communicate when other deprecated features will be removed when a timeline is established.
The following fields are now deprecated in favor of command line flags:
server.register_instrumentation
server.graceful_shutdown_timeout
server.log_source_ips_enabled
server.log_source_ips_header
server.log_source_ips_regex
server.http_listen_network
server.http_listen_address
server.http_listen_port
server.http_listen_conn_limit
server.http_server_read_timeout
server.http_server_write_timout
server.http_server_idle_timeout
server.grpc_listen_network
server.grpc_listen_address
server.grpc_listen_port
server.grpc_listen_conn_limit
server.grpc_server_max_recv_msg_size
server.grpc_server_max_send_msg_size
server.grpc_server_max_concurrent_streams
server.grpc_server_max_connection_idle
server.grpc_server_max_connection_age
server.grpc_server_max_connection_age_grace
server.grpc_server_keepalive_time
server.grpc_server_keepalive_timeout
server.grpc_server_min_time_between_pings
server.grpc_server_ping_without_stream_allowed
This is most of the fields; the remaining non-deprecated fields are
server.log_level
, server.log_format
, server.http_tls_config
, and
server.grpc_tls_config
, which support dynamic updating.
Breaking change: Removing support for dynamically updating deprecated server fields
/-/reload
will now fail if any of the deprecated server block fields have
changed. It is still valid to change a non-deprecated field (i.e., changing the
log level).
Breaking change: Server-specific command line flags have changed
The following flags are new:
-server.http.enable-tls
-server.grpc.enable-tls
-server.http.address
-server.grpc.address
The following flags have been removed:
-log.level
(replacement: use YAML fieldserver.log_level
)-log.format
(replacement: use YAML fieldserver.log_format
)-server.http-tls-cert-path
(replacement: use YAML fieldserver.http_tls_config
)-server.http-tls-key-path
(replacement: use YAML fieldserver.http_tls_config
)-server.http-tls-client-auth
(replacement: use YAML fieldserver.http_tls_config
)-server.http-tls-ca-path
(replacement: use YAML fieldserver.http_tls_config
)-server.grpc-tls-cert-path
(replacement: use YAML fieldserver.grpc_tls_config
)-server.grpc-tls-key-path
(replacement: use YAML fieldserver.grpc_tls_config
)-server.grpc-tls-client-auth
(replacement: use YAML fieldserver.grpc_tls_config
)-server.grpc-tls-ca-path
(replacement: use YAML fieldserver.grpc_tls_config
)-server.http-listen-address
(replacement: use the new-server.http.address
flag, which combines host and port)-server.http-listen-port
(replacement: use the new-server.http.address
flag, which combines host and port)-server.grpc-listen-address
(replacement: use the new-server.grpc.address
flag, which combines host and port)-server.grpc-listen-port
(replacement: use the new-server.grpc.address
flag, which combines host and port)-server.path-prefix
(no replacement; this flag was unsupported and caused undefined behavior when set)
The following flags have been renamed:
-server.log-source-ips-enabled
has been renamed to-server.log.source-ips.enabled
-server.log-source-ips-header
has been renamed to-server.log.source-ips.header
-server.log-source-ips-regex
has been renamed to-server.log.source-ips.regex
-server.http-listen-network
has been renamed to-server.http.network
-server.http-conn-limit
has been renamed to-server.http.conn-limit
-server.http-read-timeout
has been renamed to-server.http.read-timeout
-server.http-write-timeout
has been renamed to-server.http.write-timeout
-server.http-idle-timeout
has been renamed to-server.http.idle-timeout
-server.grpc-listen-network
has been renamed to-server.grpc.network
-server.grpc-conn-limit
has been renamed to-server.grpc.conn-limit
-server.grpc-max-recv-msg-size-bytes
has been renamed to-server.grpc.max-recv-msg-size-bytes
-server.grpc-max-send-msg-size-bytes
has been renamed to-server.grpc.max-send-msg-size-bytes
-server.grpc-max-concurrent-streams
has been renamed to-server.grpc.max-concurrent-streams
Breaking change: New TLS flags required for enabling TLS
The two new flags, -server.http.enable-tls
and -server.grpc.enable-tls
now
must be provided for TLS support to be enabled.
This is a change over the previous behavior where TLS was automatically enabled when a certificate pair was provided.
Breaking change: Default HTTP/gRPC address changes
The HTTP and gRPC listen addresses now default to 127.0.0.1:12345
and
127.0.0.1:12346
respectively.
If running inside of a container, you must change these to 0.0.0.0
to
externally communicate with the agent’s HTTP server.
The listen addresses may be changed via -server.http.address
and
-server.grpc.address
respectively.
Breaking change: Removal of -reload-addr
and -reload-port
flags
The -reload-addr
and -reload-port
flags have been removed. They were
initially added to workaround an issue where reloading a changed server block
would cause the primary HTTP server to restart. As the HTTP server settings are
now static, this can no longer happen, and as such the flags have been removed.
Change: In-memory autoscrape for integrations-next
This change is only relevant to those using the integrations-next
feature flag.
In-memory connections will now be used for autoscraping-enabled integrations.
This is a change over the previous behavior where autoscraping integrations
would connect to themselves over the network. As a result of this change, the
integrations.client_config
field is no longer necessary and has been removed.
v0.22
node_exporter
integration deprecated field names
The following field names for the node_exporter
integration are now deprecated:
netdev_device_whitelist
is deprecated in favor ofnetdev_device_include
.netdev_device_blacklist
is deprecated in favor ofnetdev_device_exclude
.systemd_unit_whitelist
is deprecated in favor ofsystemd_unit_include
.systemd_unit_blacklist
is deprecated in favor ofsystemd_unit_exclude
.filesystem_ignored_mount_points
is deprecated in favor offilesystem_mount_points_exclude
.filesystem_ignored_fs_types
is deprecated in favor offilesystem_fs_types_exclude
.
This change aligns with the equivalent flag names also being deprecated in the upstream node_exporter.
Support for the old field names will be removed in a future version. A warning will be logged if using the old field names when the integration is enabled.
v0.21.2, v0.20.1
Disabling of config retrieval endpoints
These two patch releases, as part of a fix for
CVE-2021-41090,
disable the /-/config
and /agent/api/v1/configs/{name}
endpoints by
default. Pass the --config.enable-read-api
flag at the command line to
re-enable them.
v0.21
Integrations: Change in how instance labels are handled (Breaking change)
Integrations will now use a SUO-specific instance
label value. Integrations
that apply to a whole machine or agent will continue to use <agent machine hostname>:<agent listen port>
, but integrations that connect to an external
system will now infer an appropriate value based on the config for that specific
integration. Please refer to the documentation for each integration for which
defaults are used.
Note: In some cases, a default value for instance
cannot be inferred. This
is the case for mongodb_exporter and postgres_exporter if more than one SUO is
being connected to. In these cases, the instance value can be manually set by
configuring the instance
field on the integration. This can also be useful if
two agents infer the same value for instance for the same integration.
As part of this change, the agent_hostname
label is permanently affixed to
self-scraped integrations and cannot be disabled. This disambiguates multiple
agents using the same instance label for an integration, and allows users to
identify which agents need to be updated with an override for instance
.
Both use_hostname_label
and replace_instance_label
are now both deprecated
and ignored from the YAML file, permanently treated as true. A future release
will remove these fields, causing YAML errors on load instead of being silently
ignored.
v0.20
Traces: Changes to receiver’s TLS config (Breaking change).
Upgrading to OpenTelemetry v0.36.0 contains a change in the receivers TLS config.
TLS params have been changed from being squashed to being in its own block.
This affect the jaeger receiver’s remote_sampling
config.
Example old config:
receivers:
jaeger:
protocols:
grpc: null,
remote_sampling:
strategy_file: <file_path>
insecure: true
Example new config:
receivers:
jaeger:
protocols:
grpc: null,
remote_sampling:
strategy_file: <file_path>
tls:
insecure: true
Traces: push_config is no longer supported (Breaking change)
push_config
was deprecated in favor of remote_write
in v0.14.0, while
maintaining backwards compatibility.
Refer to the deprecation announcement for how to upgrade.
Traces: legacy OTLP gRPC port no longer default port
OTLP gRPC receivers listen at port 4317
by default, instead of at port 55680
.
This goes in line with OTLP legacy port deprecation.
To upgrade, point the client instrumentation push endpoint to :4317
if using
the default OTLP gRPC endpoint.
v0.19
Traces: Deprecation of “tempo” in config and metrics. (Deprecation)
The term tempo
in the config has been deprecated of favor of traces
. This
change is to make intent clearer.
Example old config:
tempo:
configs:
- name: default
receivers:
jaeger:
protocols:
thrift_http:
Example of new config:
traces:
configs:
- name: default
receivers:
jaeger:
protocols:
thrift_http:
Any tempo metrics have been renamed from tempo_*
to traces_*
.
Tempo: split grouping by trace from tail sampling config (Breaking change)
Load balancing traces between agent instances has been moved from an embedded functionality in tail sampling to its own configuration block. This is done due to more processor benefiting from receiving consistently receiving all spans for a trace in the same agent to be processed, such as service graphs.
As a consequence, tail_sampling.load_balancing
has been deprecated in favor of
a load_balancing
block. Also, port
has been renamed to receiver_port
and
moved to the new load_balancing
block.
Example old config:
tail_sampling:
policies:
- always_sample:
port: 4318
load_balancing:
exporter:
insecure: true
resolver:
dns:
hostname: agent
port: 4318
Example new config:
tail_sampling:
policies:
- always_sample:
load_balancing:
exporter:
insecure: true
resolver:
dns:
hostname: agent
port: 4318
receiver_port: 4318
Metrics: Deprecation of “prometheus” in config. (Deprecation)
The term prometheus
in the config has been deprecated of favor of metrics
. This
change is to make it clearer when referring to Prometheus or another
Prometheus-like database, and configuration of Grafana Agent to send metrics to
one of those systems.
Old configs will continue to work for now, but support for the old format will
eventually be removed. To migrate your config, change the prometheus
key to
metrics
.
Example old config:
prometheus:
configs:
- name: default
host_filter: false
scrape_configs:
- job_name: local_scrape
static_configs:
- targets: ['127.0.0.1:12345']
labels:
cluster: 'localhost'
remote_write:
- url: http://localhost:9009/api/prom/push
Example new config:
metrics:
configs:
- name: default
host_filter: false
scrape_configs:
- job_name: local_scrape
static_configs:
- targets: ['127.0.0.1:12345']
labels:
cluster: 'localhost'
remote_write:
- url: http://localhost:9009/api/prom/push
Tempo: prom_instance rename (Breaking change)
As part of prometheus
being renamed to metrics
, the spanmetrics
prom_instance
field has been renamed to metrics_instance
. This is a breaking
change, and the old name will no longer work.
Example old config:
tempo:
configs:
- name: default
spanmetrics:
prom_instance: default
Example new config:
tempo:
configs:
- name: default
spanmetrics:
metrics_instance: default
Logs: Deprecation of “loki” in config. (Deprecation)
The term loki
in the config has been deprecated of favor of logs
. This
change is to make it clearer when referring to Grafana Loki, and
configuration of Grafana Agent to send logs to Grafana Loki.
Old configs will continue to work for now, but support for the old format will
eventually be removed. To migrate your config, change the loki
key to logs
.
Example old config:
loki:
positions_directory: /tmp/loki-positions
configs:
- name: default
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets: ['localhost']
labels:
job: varlogs
__path__: /var/log/*log
Example new config:
logs:
positions_directory: /tmp/loki-positions
configs:
- name: default
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets: ['localhost']
labels:
job: varlogs
__path__: /var/log/*log
Tempo: Deprecation of “loki” in config. (Deprecation)
As part of the loki
to logs
rename, parts of the automatic_logging component
in Tempo have been updated to refer to logs_instance
instead.
Old configurations using loki_name
, loki_tag
, or backend: loki
will
continue to work as of this version, but support for the old config format
will eventually be removed.
Example old config:
tempo:
configs:
- name: default
automatic_logging:
backend: loki
loki_name: default
spans: true
processes: true
roots: true
overrides:
loki_tag: tempo
Example new config:
tempo:
configs:
- name: default
automatic_logging:
backend: logs_instance
logs_instance_name: default
spans: true
processes: true
roots: true
overrides:
logs_instance_tag: tempo
v0.18
Tempo: Remote write TLS config
Tempo remote_write
now supports configuring TLS settings in the trace
exporter’s client. insecure_skip_verify
is moved into this setting’s block.
Old configurations with insecure_skip_verify
outside tls_config
will continue
to work as of this version, but support will eventually be removed.
If both insecure_skip_verify
and tls_config.insecure_skip_verify
are used,
then the latter take precedence.
Example old config:
tempo:
configs:
- name: default
remote_write:
- endpoint: otel-collector:55680
insecure: true
insecure_skip_verify: true
Example new config:
tempo:
configs:
- name: default
remote_write:
- endpoint: otel-collector:55680
insecure: true
tls_config:
insecure_skip_verify: true
v0.15
Tempo: automatic_logging
changes
Tempo automatic logging previously assumed that the operator wanted to log to a Loki instance. With the addition of an option to log to stdout a new field is required to maintain the old behavior.
Example old config:
tempo:
configs:
- name: default
automatic_logging:
loki_name: <some loki instance>
Example new config:
tempo:
configs:
- name: default
automatic_logging:
backend: loki
loki_name: <some loki instance>
v0.14
Scraping Service security change
v0.14.0 changes the default behavior of the scraping service config management API to reject all configuration files that read credentials from a file on disk. This prevents malicious users from crafting an instance config file that read arbitrary files on disk and send their contents to remote endpoints.
To revert to the old behavior, add dangerous_allow_reading_files: true
in your
scraping_service
config.
Example old config:
prometheus:
scraping_service:
# ...
Example new config:
prometheus:
scraping_service:
dangerous_allow_reading_files: true
# ...
SigV4 config change
v0.14.0 updates the internal Prometheus dependency to 2.26.0, which includes native support for SigV4, but uses a slightly different configuration structure than the Grafana Agent did.
To migrate, remove the enabled
key from your sigv4
configs. If enabled
was
the only key, define sigv4 as an empty object: sigv4: {}
.
Example old config:
sigv4:
enabled: true
region: us-east-1
Example new config:
sigv4:
region: us-east-1
Tempo: push_config
deprecation
push_config
is now deprecated in favor of a remote_write
array which allows for sending spans to multiple endpoints.
push_config
will be removed in a future release, and it is recommended to migrate to remote_write
as soon as possible.
To migrate, move the batch options outside the push_config
block.
Then, add a remote_write
array and move the remaining of your push_config
block inside it.
Example old config:
tempo:
configs:
- name: default
receivers:
otlp:
protocols:
gpc:
push_config:
endpoint: otel-collector:55680
insecure: true
batch:
timeout: 5s
send_batch_size: 100
Example migrated config:
tempo:
configs:
- name: default
receivers:
otlp:
protocols:
gpc:
remote_write:
- endpoint: otel-collector:55680
insecure: true
batch:
timeout: 5s
send_batch_size: 100
v0.12
v0.12 had two breaking changes: the tempo
and loki
sections have been changed to require a list of tempo
/loki
configs rather than just one.
Tempo Config Change
The Tempo config (tempo
in the config file) has been changed to store
configs within a configs
list. This allows for defining multiple Tempo
instances for collecting traces and forwarding them to different OTLP
endpoints.
To migrate, add a configs:
array and move your existing config inside of it.
Give the element a name: default
field.
Each config must have a unique non-empty name. default
is recommended for users
that don’t have other configs. The name of the config will be added as a
tempo_config
label for metrics.
Example old config:
tempo:
receivers:
jaeger:
protocols:
thrift_http:
attributes:
actions:
- action: upsert
key: env
value: prod
push_config:
endpoint: otel-collector:55680
insecure: true
batch:
timeout: 5s
send_batch_size: 100
Example migrated config:
tempo:
configs:
- name: default
receivers:
jaeger:
protocols:
thrift_http:
attributes:
actions:
- action: upsert
key: env
value: prod
push_config:
endpoint: otel-collector:55680
insecure: true
batch:
timeout: 5s
send_batch_size: 100
Loki Promtail Config Change
The Loki Promtail config (loki
in the config file) has been changed to store
configs within a configs
list. This allows for defining multiple Loki
Promtail instances for collecting logs and forwarding them to different Loki
servers.
To migrate, add a configs:
array and move your existing config inside of it.
Give the element a name: default
field.
Each config must have a unique non-empty name. default
is recommended for users
that don’t have other configs. The name of the config will be added as a
loki_config
label for Loki Promtail metrics.
Example old config:
loki:
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
Example migrated config:
loki:
configs:
- name: default
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log