Documentation Grafana Enterprise Logs Set up Migrate Migrate to Thanos storage clients

Migrate to Thanos storage clients

Loki release 3.4 introduces new object storage clients based on the Thanos Object Storage Client Go module.

One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki.

This is an opt-in feature with the Loki 3.4 release. In a future release, Thanos will become the default way of configuring storage and the existing storage clients will be deprecated.

Enable the new storage clients

1. Enable Thanos storage clients by setting use_thanos_objstore to true in the storage_config section or by setting the -use-thanos-objstore flag to true. When enabled, configuration under storage_config.object_store takes effect instead of existing storage configurations.

   yaml Copy

   # Uses the new storage clients for connecting to gcs backend
   storage_config:
     use_thanos_objstore: true # enable the new storage clients
     object_store:
       gcs:
         bucket_name: "example-bucket"

2. As an alternative, you can also configure the new clients in the common storage section if you prefer to use the common config section.

   yaml Copy

   storage_config:
      use_thanos_objstore: true # enable the new storage clients
   common:
     storage:
       object_store:
         gcs:
           bucket_name: "example-bucket"

3. Ruler storage should be configured under the ruler_storage section when using the new storage clients.

   yaml Copy

   storage_config:
      use_thanos_objstore: true # enable the new storage clients
   ruler_storage:
      backend: gcs
      gcs:
         bucket_name: "example-bucket"

4. If you are using store.object-prefix flag or the corresponding object_prefix YAML setting, you’ll need to update your configuration to use the new object_store.storage-prefix flag or the corresponding storage_prefix YAML setting.

   yaml Copy

   # Example configuration to prefix all objects with "prefix"
   storage_config:
      use_thanos_objstore: true # enable the new storage clients
      object_store:
         storage_prefix: "prefix"

GCS Storage Migration

When migrating from the existing Google Cloud Storage (GCS) storage client to the new Thanos-based client, you’ll need to update your configuration parameters as follows:

Example configuration migration (GCS):

Existing configuration:

yaml Copy

storage_config:
  gcs:
    bucket_name: example-bucket
    chunk_buffer_size: 10MB
    enable_retries: true

New configuration (Thanos-based):

yaml Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    gcs:
      bucket_name: example-bucket
      chunk_buffer_size: 10MB
      max_retries: 5

Amazon S3 Storage Migration

When migrating from the existing Amazon S3 storage client to the new Thanos-based client, update or remove parameters as follows:

Example configuration migration (S3):

Existing configuration

yaml Copy

storage_config:
  aws:
    bucket_names: my-bucket1,my-bucket2   # multiple buckets no longer supported
    endpoint: s3.amazonaws.com
    region: us-west-2
    access_key_id: example-key
    secret_access_key: example-secret
    signature_version: v4
    disable_dualstack: true
    storage_class: STANDARD
    http_config:
      timeout: 1m
      insecure_skip_verify: false
    # ...
    backoff_config:
      max_retries: 5
    sse:
      type: SSE-KMS
      kms_key_id: mySSEKey

New configuration (Thanos-based)

yaml Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    s3:
      bucket_name: my-bucket1                       # single bucket
      endpoint: s3.amazonaws.com
      region: us-west-2
      access_key_id: example-key
      secret_access_key: example-secret
      dualstack_enabled: false                        # was disable_dualstack: true
      storage_class: STANDARD
      max_retries: 5
      http:
        insecure_skip_verify: false
      sse:
        type: SSE-KMS
        kms_key_id: mySSEKey

For more advanced configuration options (such as list_objects_version, bucket_lookup_type, etc.), see the Thanos S3 configuration reference.

Azure Storage Migration

When migrating from the existing Azure storage client to the new Thanos-based client, no changes are required if you are using the following parameters:

If you are using an authentication method other than storage account key or user-assigned managed identity, you’ll have to pass the neccessary credetials using environment variables. For more details, refer to Azure Identity Client Module for Go.

Filesystem Storage Migration

When migrating from the existing Filesystem storage client to the new Thanos-based client, update or remove parameters as follows:

Example configuration migration (Filesystem):

Existing configuration (FSConfig)

yaml Copy

storage_config:
  filesystem:
    directory: /var/loki/chunks

New configuration (Thanos-based)

yaml Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    filesystem:
      dir: /var/loki/chunks