Cardinality

Cardinality refers to the total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate. For more information on cardinality, see the What are cardinality spikes and why do they matter? blog post.

Because writes to a time-series database (TSDB) database are in series, high cardinality does not make a big difference to performance at ingest. However, cardinality can have a major impact on querying where, the higher the cardinality, the more items are required to be iterated over.

Traces collection and metrics

Tempo’s server-side metrics generation adds functionality to the collection of traces by creating Prometheus-based metrics that track a variety of metrics such as:

Total span call counts
Span latency histograms
Total span size count

The metrics-generator creates metrics which define the relationship between services via edges and nodes. Each of these metrics are queryable using a set of Prometheus labels (key/value pairs).

Each new value for a label increases the number of active series associated with a metric. (To learn more about active series, read the Trace active series documentation.)

This is also known as an increase in cardinality, and the number of active series generated for a metric is directly proportional to the number of labels that exist for that metrics alongside the number of values each label has added.

In a non-modified instance of the metrics generator, a small number of labels are added automatically. Because labels like span_kind and status_code only have a few valid values, the largest variable for the number of active series produced for each metric depends on the number of service names and span names associated with trace spans.

The metrics-generator can also be configured to also add extra labels on metrics, using span attribute key/value pairs which are mapped directly to these labels see the custom span attribute documentation.

Be careful when configuring custom attributes: the greater the number of values seen in a specific attribute, the greater the number of active series will be produced. For more information about active series, refer to the active series documentation

Let’s say that you are adding a custom attribute that includes unique customer IDs as a metrics label. If you have 100 customers, this could potentially multiply the number of active series generated by up to 100 (for example, going from 25,000 active series to 2.5M). Always consider which attributes will actually be useful as labels for querying metrics, as well as the cardinality that they will increase metrics by.

Dry-running the metrics-generator

A good practice, before fully enabling metrics-generator, is to run the metrics-generator in dry-run mode. Using the dry-run mode generates metrics but does not collect them, and thus will not write them to a metrics storage database. The override metrics_generator.disable_collection is defined for this use-case.

To get an estimate, set the override to true, and run the metrics-generator as you would normally. Then, check tempo_metrics_generator_registry_active_series for the calculated estimation of active series for your set-up.