Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Configuration

Polling

Open source

Polling

The polling cycle is controlled by a number of configuration options detailed here.

storage:
    trace:
        # How often to repoll the backend for new blocks. Default is 5m
        [blocklist_poll: <duration>] 

        # Number of blocks to process in parallel during polling. Default is 50.
        [blocklist_poll_concurrency: <int>]

        # By default components will pull the blocklist from the tenant index. If that fails the component can
        # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true.
        [blocklist_poll_fallback: <bool>]

        # Maximum number of compactors that should build the tenant index. All other components will download 
        # the index.  Default 2.
        [blocklist_poll_tenant_index_builders: <int>]

        # The oldest allowable tenant index. If an index is pulled that is older than this duration,
        # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here.
        # If fallback is true and a tenant index exceeds this duration, it will fall back to listing
        # the bucket contents.
        # Default 0 (disabled).
        [blocklist_poll_stale_tenant_index: <duration>]

Due to the mechanics of the tenant index the blocklist will be stale by at most 2 times the configured blocklist_poll duration. There are two configuration options that need to be balanced against the blockist_poll to handle this:

The ingester complete_block_timeout is used to hold a block in the ingester for a given period of time after it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware of the newly flushed blocks.

ingester:
  # How long to hold a complete block in the ingester after it has been flushed to the backend.  Default is 15m
  [complete_block_timeout: <duration>]

The compactor compacted_block_retention is used to keep a block in the backend for a given period of time after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the complete_block_timeout this should be at a minimum 2x the configurated blocklist_poll duration.

compactor:
  compaction:
    # How long to leave a block in the backend after it has been compacted successfully.  Default is 1h
    [compacted_block_retention: <duration>]

Additionally, it is important that the querier blocklist_poll duration is greater than or equal to the compactor blocklist_poll duration. Otherwise a querier may not correctly check all assigned blocks and incorrectly return 404. It is recommended to simply set both components to use the same poll duration.