This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.
Community
otelcol.exporter.splunkhec
Community: This component is developed, maintained, and supported by the Alloy user community. Grafana doesn’t offer commercial support for this component. To enable and use community components, you must set the
--feature.community-components.enabled
flag totrue
.
otelcol.exporter.splunkhec
accepts metrics and traces telemetry data from other otelcol
components and sends it to Splunk HEC.
Note
otelcol.exporter.splunkhec
is a wrapper over the upstream OpenTelemetry Collectorsplunkhec
exporter from theotelcol-contrib
distribution. Bug reports or feature requests will be redirected to the upstream repository, if necessary.
You can specify multiple otelcol.exporter.splunkhec
components by giving them different labels.
Usage
otelcol.exporter.splunkhec "LABEL" {
splunk {
token = "YOUR_SPLUNK_TOKEN"
}
client {
endpoint = "http://splunk.yourdomain.com:8088"
}
}
Arguments
The otelcol.exporter.splunkhec
component does not support any arguments, and is configured
fully through child blocks.
Blocks
The following blocks are supported inside the definition of otelcol.exporter.splunkhec
:
Hierarchy | Block | Description | Required |
---|---|---|---|
splunk | splunk | Configures the Splunk HEC exporter. | yes |
splunk->otel_to_hec_fields | otel_to_hec_fields | Configures mapping of Open Telemetry to HEC Fields. | no |
splunk->telemetry | telemetry | Configures the exporters telemetry. | no |
splunk->heartbeat | heartbeat | Configures the exporters heartbeat settings. | no |
splunk->batcher | batcher | Configures batching requests based on a timeout and a minimum number of items. | no |
client | client | Configures the HTTP client used to send data to Splunk HEC. | yes |
retry_on_failure | retry_on_failure | Configures retry mechanism for failed requests. | no |
queue | queue | Configures batching of data before sending. | no |
debug_metrics | debug_metrics | Configures the metrics that this component generates to monitor its state. | no |
splunk block
The splunk
block configures Splunk HEC specific settings.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
token | secret | Splunk HEC Token. | yes | |
log_data_enabled | bool | Enable sending logs from the exporter. One of log_data_enabled or profiling_data_enabled must be true . | true | no |
profiling_data_enabled | bool | Enable sending profiling data from the exporter. One of log_data_enabled or profiling_data_enabled must be true . | true | no |
source | string | Splunk source. | "" | no |
source_type | string | Splunk source sype. | "" | no |
index | string | Splunk index name. | "" | no |
disable_compression | bool | Disable GZip compression. | false | no |
max_content_length_logs | uint | Maximum log payload size in bytes. Must be less than 838860800 (~800MB). | 2097152 | no |
max_content_length_metrics | uint | Maximum metric payload size in bytes. Must be less than 838860800 (~800MB). | 2097152 | no |
max_content_length_traces | uint | Maximum trace payload size in bytes. Must be less than 838860800 (~800MB). | 2097152 | no |
max_event_size | uint | Maximum event payload size in bytes. Must be less than 838860800 (~800MB). | 5242880 | no |
splunk_app_name | string | Used to track telemetry for Splunk Apps by name. | Alloy | no |
splunk_app_version | string | Used to track telemetry by App version. | "" | no |
health_path | string | Path for the health API. | /services/collector/health' | no |
health_check_enabled | bool | Used to verify Splunk HEC health on exporter startup. | true | no |
export_raw | bool | Send only the logs body when targeting HEC raw endpoint. | false | no |
use_multi_metrics_format | bool | Use multi-metrics format to save space during ingestion. | false | no |
otel_to_hec_fields block
Name | Type | Description | Default | Required |
---|---|---|---|---|
severity_text | string | Maps severity text field to a specific HEC field. | "" | no |
severity_number | string | Maps severity number field to a specific HEC field. | "" | no |
heartbeat block
Name | Type | Description | Default | Required |
---|---|---|---|---|
interval | time.Duration | Time interval for the heartbeat interval, in seconds. | 0s | no |
startup | bool | Send heartbeat events on exporter startup. | false | no |
telemetry block
Name | Type | Description | Default | Required |
---|---|---|---|---|
enabled | bool | Enable telemetry inside the exporter. | false | no |
override_metrics_names | map(string) | Override metrics for internal metrics in the exporter. | no |
batcher block
Name | Type | Description | Default | Required |
---|---|---|---|---|
enabled | bool | Whether to not enqueue batches before sending to the consumerSender. | false | no |
flush_timeout | time.Duration | The time after which a batch will be sent regardless of its size. | 200ms | no |
min_size_items | uint | The number of items at which the batch is sent regardless of the timeout. | 8192 | no |
max_size_items | uint | Maximum number of batch items, if the batch exceeds this value, it will be broken up into smaller batches. Must be greater than or equal to min_size_items. Setting this value to zero disables the maximum size limit. | 0 | no |
client block
The client
block configures the HTTP client used by the component.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
endpoint | string | The Splunk HEC endpoint to use. | yes | |
read_buffer_size | int | Size of the read buffer the HTTP client uses for reading server responses. | 0 | no |
write_buffer_size | int | Size of the write buffer the HTTP client uses for writing requests. | 0 | no |
timeout | duration | Time to wait before marking a request as failed. | "15s" | no |
max_idle_conns | int | Limits the number of idle HTTP connections the client can keep open. | 100 | no |
max_idle_conns_per_host | int | Limits the number of idle HTTP connections the host can keep open. | 2 | no |
max_conns_per_host | int | Limits the total (dialing,active, and idle) number of connections per host. Zero means no limit | 0 | no |
idle_conn_timeout | duration | Time to wait before an idle connection closes itself. | "45s" | no |
disable_keep_alives | bool | Disable HTTP keep-alive. | false | no |
insecure_skip_verify | bool | Ignores insecure server TLS certificates. | false | no |
retry_on_failure block
The retry_on_failure
block configures how failed requests to splunkhec are retried.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
enabled | boolean | Enables retrying failed requests. | true | no |
initial_interval | duration | Initial time to wait before retrying a failed request. | "5s" | no |
max_elapsed_time | duration | Maximum time to wait before discarding a failed batch. | "5m" | no |
max_interval | duration | Maximum time to wait between retries. | "30s" | no |
multiplier | number | Factor to grow wait time before retrying. | 1.5 | no |
randomization_factor | number | Factor to randomize wait time before retrying. | 0.5 | no |
When enabled
is true
, failed batches are retried after a given interval.
The initial_interval
argument specifies how long to wait before the first retry attempt.
If requests continue to fail, the time to wait before retrying increases by the factor specified by the multiplier
argument, which must be greater than 1.0
.
The max_interval
argument specifies the upper bound of how long to wait between retries.
The randomization_factor
argument is useful for adding jitter between retrying Alloy instances.
If randomization_factor
is greater than 0
, the wait time before retries is multiplied by a random factor in the range [ I - randomization_factor * I, I + randomization_factor * I]
, where I
is the current interval.
If a batch hasn’t been sent successfully, it is discarded after the time specified by max_elapsed_time
elapses.
If max_elapsed_time
is set to "0s"
, failed requests are retried forever until they succeed.
queue block
The queue
block configures an in-memory buffer of batches before data is sent to the HTTP server.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
enabled | boolean | Enables an in-memory buffer before sending data to the client. | true | no |
num_consumers | number | Number of readers to send batches written to the queue in parallel. | 10 | no |
queue_size | number | Maximum number of unwritten batches allowed in the queue at the same time. | 1000 | no |
When enabled
is true
, data is first written to an in-memory buffer before sending it to the configured server.
Batches sent to the component’s input
exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size
.
queue_size
determines how long an endpoint outage is tolerated.
Assuming 100 requests/second, the default queue size 1000
provides about 10 seconds of outage tolerance.
To calculate the correct value for queue_size
, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.
The num_consumers
argument controls how many readers read from the buffer and send data in parallel.
Larger values of num_consumers
allow data to be sent more quickly at the expense of increased network traffic.
debug_metrics block
The debug_metrics
block configures the metrics that this component generates to monitor its state.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
disable_high_cardinality_metrics | boolean | Whether to disable certain high cardinality metrics. | true | no |
level | string | Controls the level of detail for metrics emitted by the wrapped collector. | "detailed" | no |
disable_high_cardinality_metrics
is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics
feature gate in the OpenTelemetry Collector.
It removes attributes that could cause high cardinality metrics.
For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.
Note
If configured,disable_high_cardinality_metrics
only applies tootelcol.exporter.*
andotelcol.receiver.*
components.
level
is the Alloy equivalent to the telemetry.metrics.level
feature gate in the OpenTelemetry Collector.
Possible values are "none"
, "basic"
, "normal"
and "detailed"
.
Exported fields
The following fields are exported and can be referenced by other components:
Name | Type | Description |
---|---|---|
input | otelcol.Consumer | A value other components can use to send telemetry data to. |
input
accepts otelcol.Consumer
data for any telemetry signal (metrics, logs, or traces).
Component health
otelcol.exporter.splunkhec
is only reported as unhealthy if given an invalid configuration.
Debug information
otelcol.exporter.splunkhec
does not expose any component-specific debug information.
Example
Open Telemetry Receiver
This example forwards metrics, logs and traces send to the otelcol.receiver.otlp.default
receiver to the Splunk HEC exporter.
otelcol.receiver.otlp "default" {
grpc {
endpoint = "localhost:4317"
}
http {
endpoint = "localhost:4318"
compression_algorithms = ["zlib"]
}
output {
metrics = [otelcol.exporter.splunkhec.default.input]
logs = [otelcol.exporter.splunkhec.default.input]
traces = [otelcol.exporter.splunkhec.default.input]
}
}
otelcol.exporter.splunkhec "default" {
client {
endpoint = "https://splunkhec.domain.com:8088/services/collector"
timeout = "10s"
max_idle_conns = 200
max_idle_conns_per_host = 200
idle_conn_timeout = "10s"
}
splunk {
token = "SPLUNK_TOKEN"
source = "otel"
sourcetype = "otel"
index = "metrics"
splunk_app_name = "OpenTelemetry-Collector Splunk Exporter"
splunk_app_version = "v0.0.1"
z
otel_to_hec_fields {
severity_text = "otel.log.severity.text"
severity_number = "otel.log.severity.number"
}
heartbeat {
interval = "30s"
}
telemetry {
enabled = true
override_metrics_names = {
otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
otelcol_exporter_splunkhec_heartbeats_sent = "app_heartbeats_success_total",
}
extra_attributes = {
custom_key = "custom_value",
dataset_name = "SplunkCloudBeaverStack",
}
}
}
}
Forward Prometheus Metrics
This example forwards Prometheus metrics from Alloy through a receiver for conversion to Open Telemetry format before finally sending them to splunkhec.
prometheus.exporter.self "default" {
}
prometheus.scrape "metamonitoring" {
targets = prometheus.exporter.self.default.targets
forward_to = [otelcol.receiver.prometheus.default.receiver]
}
otelcol.receiver.prometheus "default" {
output {
metrics = [otelcol.exporter.splunkhec.default.input]
}
}
otelcol.exporter.splunkhec "default" {
splunk {
token = "SPLUNK_TOKEN"
}
client {
endpoint = "http://splunkhec.domain.com:8088"
}
}
Forward Loki logs
This example watches for files ending with .log
in the path /var/log
, tails these logs with Loki and forwards the logs to the configured Splunk HEC endpoint. The Splunk HEC exporter component is setup to send an heartbeat every 5 seconds.
local.file_match "local_files" {
path_targets = [{"__path__" = "/var/log/*.log"}]
sync_period = "5s"
}
otelcol.receiver.loki "default" {
output {
logs = [otelcol.processor.attributes.default.input]
}
}
otelcol.processor.attributes "default" {
action {
key = "host"
action = "upsert"
value = "myhost"
}
action {
key = "host.name"
action = "upsert"
value = "myhost"
}
output {
logs = [otelcol.exporter.splunkhec.default.input]
}
}
loki.source.file "log_scrape" {
targets = local.file_match.local_files.targets
forward_to = [otelcol.receiver.loki.default.receiver]
tail_from_end = false
}
otelcol.exporter.splunkhec "default" {
retry_on_failure {
enabled = false
}
client {
endpoint = "http://splunkhec.domain.com:8088"
timeout = "5s"
max_idle_conns = 200
max_idle_conns_per_host = 200
idle_conn_timeout = "10s"
write_buffer_size = 8000
}
sending_queue {
enabled = false
}
splunk {
token = "SPLUNK_TOKEN"
source = "otel"
sourcetype = "otel"
index = "devnull"
log_data_enabled = true
heartbeat {
interval = "5s"
}
batcher {
flush_timeout = "200ms"
}
telemetry {
enabled = true
override_metrics_names = {
otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
otelcol_exporter_splunkhec_heartbeats_sent = "app_heartbeats_success_total",
}
extra_attributes = {
host = "myhost",
dataset_name = "SplunkCloudBeaverStack",
}
}
}
}
Compatible components
otelcol.exporter.splunkhec
has exports that can be consumed by the following components:
- Components that consume OpenTelemetry
otelcol.Consumer
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.