Menu

This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.

Open source

Community

otelcol.exporter.splunkhec

Community: This component is developed, maintained, and supported by the Alloy user community. Grafana doesn’t offer commercial support for this component. To enable and use community components, you must set the --feature.community-components.enabled flag to true.

otelcol.exporter.splunkhec accepts metrics and traces telemetry data from other otelcol components and sends it to Splunk HEC.

Note

otelcol.exporter.splunkhec is a wrapper over the upstream OpenTelemetry Collector splunkhec exporter from the otelcol-contrib distribution. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

You can specify multiple otelcol.exporter.splunkhec components by giving them different labels.

Usage

alloy
otelcol.exporter.splunkhec "LABEL" {
    splunk {
        token = "YOUR_SPLUNK_TOKEN"
    }
    client {
        endpoint = "http://splunk.yourdomain.com:8088"
    }
}

Arguments

The otelcol.exporter.splunkhec component does not support any arguments, and is configured fully through child blocks.

Blocks

The following blocks are supported inside the definition of otelcol.exporter.splunkhec:

HierarchyBlockDescriptionRequired
splunksplunkConfigures the Splunk HEC exporter.yes
splunk->otel_to_hec_fieldsotel_to_hec_fieldsConfigures mapping of Open Telemetry to HEC Fields.no
splunk->telemetrytelemetryConfigures the exporters telemetry.no
splunk->heartbeatheartbeatConfigures the exporters heartbeat settings.no
splunk->batcherbatcherConfigures batching requests based on a timeout and a minimum number of items.no
clientclientConfigures the HTTP client used to send data to Splunk HEC.yes
retry_on_failureretry_on_failureConfigures retry mechanism for failed requests.no
queuequeueConfigures batching of data before sending.no
debug_metricsdebug_metricsConfigures the metrics that this component generates to monitor its state.no

splunk block

The splunk block configures Splunk HEC specific settings.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
tokensecretSplunk HEC Token.yes
log_data_enabledboolEnable sending logs from the exporter. One of log_data_enabled or profiling_data_enabled must be true.trueno
profiling_data_enabledboolEnable sending profiling data from the exporter. One of log_data_enabled or profiling_data_enabled must be true.trueno
sourcestringSplunk source.""no
source_typestringSplunk source sype.""no
indexstringSplunk index name.""no
disable_compressionboolDisable GZip compression.falseno
max_content_length_logsuintMaximum log payload size in bytes. Must be less than 838860800 (~800MB).2097152no
max_content_length_metricsuintMaximum metric payload size in bytes. Must be less than 838860800 (~800MB).2097152no
max_content_length_tracesuintMaximum trace payload size in bytes. Must be less than 838860800 (~800MB).2097152no
max_event_sizeuintMaximum event payload size in bytes. Must be less than 838860800 (~800MB).5242880no
splunk_app_namestringUsed to track telemetry for Splunk Apps by name.Alloyno
splunk_app_versionstringUsed to track telemetry by App version.""no
health_pathstringPath for the health API./services/collector/health'no
health_check_enabledboolUsed to verify Splunk HEC health on exporter startup.trueno
export_rawboolSend only the logs body when targeting HEC raw endpoint.falseno
use_multi_metrics_formatboolUse multi-metrics format to save space during ingestion.falseno

otel_to_hec_fields block

NameTypeDescriptionDefaultRequired
severity_textstringMaps severity text field to a specific HEC field.""no
severity_numberstringMaps severity number field to a specific HEC field.""no

heartbeat block

NameTypeDescriptionDefaultRequired
intervaltime.DurationTime interval for the heartbeat interval, in seconds.0sno
startupboolSend heartbeat events on exporter startup.falseno

telemetry block

NameTypeDescriptionDefaultRequired
enabledboolEnable telemetry inside the exporter.falseno
override_metrics_namesmap(string)Override metrics for internal metrics in the exporter.no

batcher block

NameTypeDescriptionDefaultRequired
enabledboolWhether to not enqueue batches before sending to the consumerSender.falseno
flush_timeouttime.DurationThe time after which a batch will be sent regardless of its size.200msno
min_size_itemsuintThe number of items at which the batch is sent regardless of the timeout.8192no
max_size_itemsuintMaximum number of batch items, if the batch exceeds this value, it will be broken up into smaller batches. Must be greater than or equal to min_size_items. Setting this value to zero disables the maximum size limit.0no

client block

The client block configures the HTTP client used by the component.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
endpointstringThe Splunk HEC endpoint to use.yes
read_buffer_sizeintSize of the read buffer the HTTP client uses for reading server responses.0no
write_buffer_sizeintSize of the write buffer the HTTP client uses for writing requests.0no
timeoutdurationTime to wait before marking a request as failed."15s"no
max_idle_connsintLimits the number of idle HTTP connections the client can keep open.100no
max_idle_conns_per_hostintLimits the number of idle HTTP connections the host can keep open.2no
max_conns_per_hostintLimits the total (dialing,active, and idle) number of connections per host. Zero means no limit0no
idle_conn_timeoutdurationTime to wait before an idle connection closes itself."45s"no
disable_keep_alivesboolDisable HTTP keep-alive.falseno
insecure_skip_verifyboolIgnores insecure server TLS certificates.falseno

retry_on_failure block

The retry_on_failure block configures how failed requests to splunkhec are retried.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
enabledbooleanEnables retrying failed requests.trueno
initial_intervaldurationInitial time to wait before retrying a failed request."5s"no
max_elapsed_timedurationMaximum time to wait before discarding a failed batch."5m"no
max_intervaldurationMaximum time to wait between retries."30s"no
multipliernumberFactor to grow wait time before retrying.1.5no
randomization_factornumberFactor to randomize wait time before retrying.0.5no

When enabled is true, failed batches are retried after a given interval. The initial_interval argument specifies how long to wait before the first retry attempt. If requests continue to fail, the time to wait before retrying increases by the factor specified by the multiplier argument, which must be greater than 1.0. The max_interval argument specifies the upper bound of how long to wait between retries.

The randomization_factor argument is useful for adding jitter between retrying Alloy instances. If randomization_factor is greater than 0, the wait time before retries is multiplied by a random factor in the range [ I - randomization_factor * I, I + randomization_factor * I], where I is the current interval.

If a batch hasn’t been sent successfully, it is discarded after the time specified by max_elapsed_time elapses. If max_elapsed_time is set to "0s", failed requests are retried forever until they succeed.

queue block

The queue block configures an in-memory buffer of batches before data is sent to the HTTP server.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
enabledbooleanEnables an in-memory buffer before sending data to the client.trueno
num_consumersnumberNumber of readers to send batches written to the queue in parallel.10no
queue_sizenumberMaximum number of unwritten batches allowed in the queue at the same time.1000no

When enabled is true, data is first written to an in-memory buffer before sending it to the configured server. Batches sent to the component’s input exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size.

queue_size determines how long an endpoint outage is tolerated. Assuming 100 requests/second, the default queue size 1000 provides about 10 seconds of outage tolerance. To calculate the correct value for queue_size, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.

The num_consumers argument controls how many readers read from the buffer and send data in parallel. Larger values of num_consumers allow data to be sent more quickly at the expense of increased network traffic.

debug_metrics block

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
disable_high_cardinality_metricsbooleanWhether to disable certain high cardinality metrics.trueno
levelstringControls the level of detail for metrics emitted by the wrapped collector."detailed"no

disable_high_cardinality_metrics is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

Note

If configured, disable_high_cardinality_metrics only applies to otelcol.exporter.* and otelcol.receiver.* components.

level is the Alloy equivalent to the telemetry.metrics.level feature gate in the OpenTelemetry Collector. Possible values are "none", "basic", "normal" and "detailed".

Exported fields

The following fields are exported and can be referenced by other components:

NameTypeDescription
inputotelcol.ConsumerA value other components can use to send telemetry data to.

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.exporter.splunkhec is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.splunkhec does not expose any component-specific debug information.

Example

Open Telemetry Receiver

This example forwards metrics, logs and traces send to the otelcol.receiver.otlp.default receiver to the Splunk HEC exporter.

otelcol.receiver.otlp "default" {
	grpc {
		endpoint = "localhost:4317"
	}

	http {
		endpoint               = "localhost:4318"
		compression_algorithms = ["zlib"]
	}

	output {
		metrics = [otelcol.exporter.splunkhec.default.input]
		logs    = [otelcol.exporter.splunkhec.default.input]
		traces  = [otelcol.exporter.splunkhec.default.input]
	}
}

otelcol.exporter.splunkhec "default" {
	client {
		endpoint                = "https://splunkhec.domain.com:8088/services/collector"
		timeout                 = "10s"
		max_idle_conns          = 200
		max_idle_conns_per_host = 200
		idle_conn_timeout       = "10s"
	}

	splunk {
		token              = "SPLUNK_TOKEN"
		source             = "otel"
		sourcetype         = "otel"
		index              = "metrics"
		splunk_app_name    = "OpenTelemetry-Collector Splunk Exporter"
		splunk_app_version = "v0.0.1"
z
		otel_to_hec_fields {
			severity_text   = "otel.log.severity.text"
			severity_number = "otel.log.severity.number"
		}

		heartbeat {
			interval = "30s"
		}

		telemetry {
			enabled                = true
			override_metrics_names = {
				otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
				otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
			}
			extra_attributes = {
				custom_key   = "custom_value",
				dataset_name = "SplunkCloudBeaverStack",
			}
		}
	}
}

Forward Prometheus Metrics

This example forwards Prometheus metrics from Alloy through a receiver for conversion to Open Telemetry format before finally sending them to splunkhec.

alloy
prometheus.exporter.self "default" {
}

prometheus.scrape "metamonitoring" {
  targets    = prometheus.exporter.self.default.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

otelcol.receiver.prometheus "default" {
  output {
    metrics = [otelcol.exporter.splunkhec.default.input]
  }
}


otelcol.exporter.splunkhec "default" {
    splunk {
        token = "SPLUNK_TOKEN"
    }
    client {
        endpoint = "http://splunkhec.domain.com:8088"
    }
}

Forward Loki logs

This example watches for files ending with .log in the path /var/log, tails these logs with Loki and forwards the logs to the configured Splunk HEC endpoint. The Splunk HEC exporter component is setup to send an heartbeat every 5 seconds.

alloy
local.file_match "local_files" {
	path_targets = [{"__path__" = "/var/log/*.log"}]
	sync_period  = "5s"
}

otelcol.receiver.loki "default" {
	output {
		logs = [otelcol.processor.attributes.default.input]
	}
}

otelcol.processor.attributes "default" {
	action {
		key    = "host"
		action = "upsert"
		value  = "myhost"
	}

	action {
		key    = "host.name"
		action = "upsert"
		value  = "myhost"
	}

	output {
		logs = [otelcol.exporter.splunkhec.default.input]
	}
}

loki.source.file "log_scrape" {
	targets       = local.file_match.local_files.targets
	forward_to    = [otelcol.receiver.loki.default.receiver]
	tail_from_end = false
}

otelcol.exporter.splunkhec "default" {
	retry_on_failure {
		enabled = false
	}

	client {
		endpoint                = "http://splunkhec.domain.com:8088"
		timeout                 = "5s"
		max_idle_conns          = 200
		max_idle_conns_per_host = 200
		idle_conn_timeout       = "10s"
		write_buffer_size       = 8000
	}

	sending_queue {
		enabled = false
	}

	splunk {
		token            = "SPLUNK_TOKEN"
		source           = "otel"
		sourcetype       = "otel"
		index            = "devnull"
		log_data_enabled = true

		heartbeat {
			interval = "5s"
		}

		batcher {
			flush_timeout = "200ms"
		}

		telemetry {
			enabled                = true
			override_metrics_names = {
				otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
				otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
			}
			extra_attributes = {
				host   = "myhost",
				dataset_name = "SplunkCloudBeaverStack",
			}
		}
	}
}

Compatible components

otelcol.exporter.splunkhec has exports that can be consumed by the following components:

Note

Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.