Documentation Grafana Cloud Instrument and send data OpenTelemetry OTLP format considerations

OTLP: OpenTelemetry Protocol format considerations

OpenTelemetry doesn’t specify a data storage approach and leaves it to the observability backend to implement. Grafana Cloud converts metrics, logs, and traces sent to the Grafana Cloud OTLP endpoint to be compatible with Grafana databases.

The Grafana Cloud OTLP endpoint supports the OTLP/HTTP with binary protocol buffer encoding.

Metrics

Grafana Cloud stores metrics in Mimir, a Prometheus compatible database, and maps OpenTelemetry metrics following the official specification for OTLP Metric points to Prometheus.

Exponential histograms

By default, Grafana Cloud doesn’t accept OpenTelemetry exponential histograms.

To accept exponential histograms in Grafana Cloud, follow the send Prometheus native histograms documentation to enable Prometheus native histogram metrics for your environment.

When enabled, Grafana cloud converts OpenTelemetry exponential histograms into Prometheus native histograms following the official specifications for exponential histograms.

Metric and label name conversion

Prometheus metric and label names need to adhere to the regular expression [a-zA-Z_:][a-zA-Z0-9_:]* which contains alphanumeric, underscore, and colon characters. Grafana converts fullstops . and dashes - in OpenTelemetry metric or label names to underscores _ to store them in Prometheus.

Example

If you sent the following OTLP data:

json Copy

requests.duration{http.response.status_code=500, http.route="/api/orders"}

Grafana converts it for Prometheus storage:

json Copy

requests_duration{http_response_status_code="500", http_route="/api/orders"}

Resource attributes added to target_info metric

As described in the Resource Attributes specification, we add resource attributes to a special target_info metric, with the exception of service.instance.id, service.name and service.namespace.

The value <service.namespace>/<service.name> (or <service.name> if namespace is empty), is added as job label and service.instance.id is added as instance label to every metric as well as the target_info metric. The job and instance labels are sometimes called identifying labels.

Use the on(job, instance) operator to join any metric with the target_info metric to retrieve the resource attributes for that metric. For example to get the service.version attribute for a metric called requests_total, use the following PromQL:

PromQL Copy

requests_total * on(job, instance) group_left(service_version) target_info

Alternatively you can promote (copy) service.instance.id, service.name, service.namespace or any other resource attribute to the labels of metrics as show for example in A practical guide to data collection with OpenTelemetry and Prometheus.

Suffixes added to metric names

By default Prometheus adds suffixes to metric names to comply with the Prometheus metric types name conventions. Grafana Cloud follows the official specification for OTLP Metric points to Prometheus.

The suffix generation procedure is as follows:

1. Split the OTel unit into two parts, before and after any / character.
2. Add _<UNIT> to the suffix if the metric name doesn’t contain the first unit part.
3. Add _per_<UNIT> to the suffix if there’s a second unit part and the metric name doesn’t contain it.
4. Add _total to the suffix if the OTel metric type is monotonic sum.
5. Add ratio to the suffix if the OTel metric type is gauge and the unit is 1.

Example

Grafana Cloud converts the OpenTelemetry monotonic sum system.io with unit By into the Prometheus name system_io_bytes_total, and _bytes_total is the generated suffix.

Metrics ingestion limits

When you send batch metrics, Prometheus ingests only the metrics that don’t exceed limits and returns exceptions for the metrics that exceed limits.

Logs

Grafana Cloud converts OpenTelemetry logs and stores them in the Loki V3 format to leverage structured metadata and labels.

The conversion process is as follows:

Logs ingestion limits

When you send batch logs, Loki ingests only the logs that don’t exceed limits and returns exceptions for the logs that exceed limits.

Loki versions before V3

Before Loki V3 introduced structured metadata, Loki converted OpenTelemetry logs in Alloy otelcol.exporter.loki or in the OpenTelemetry Collector Loki Exporter and stored them as JSON in the message of log lines.

To promote more resource attributes and log attributes to Loki labels, use the hints loki.resource.labels and loki.attribute.labels as documented on OpenTelemetry Collector Loki Exporter and Alloy otelcol.exporter.loki.

Traces

Traces limits