Tag search
An issue occurs while searching for traces in Grafana Explore. The Service Name and Span Name drop down lists are empty, and there is a No options found
message.
HTTP requests to Tempo query frontend endpoint at /api/search/tag/service.name/values
would respond with an empty set.
Root cause
The introduction of a cap on the size of tags causes this issue.
Configuration parameter max_bytes_per_tag_values_query
causes the return of an empty result
when a query exceeds the configured value.
Solutions
There are two main solutions to this issue:
- Reduce the cardinality of tags pushed to Tempo. Reducing the number of unique tag values will reduce the size returned by a tag search query.
- Increase the
max_bytes_per_tag_values_query
parameter in the overrides block of your Tempo configuration to a value as high as 50MB.