Configure Pyroscope object storage backend
Pyroscope can use different object storage services to persist blocks containing the profiles data. Blocks are flushed by ingesters on disk first then are uploaded to the object store.
The supported backends are:
- Amazon S3 (and compatible implementations like MinIO)
- Google Cloud Storage
- Azure Blob Storage
- Swift (OpenStack Object Storage)
Internally, Pyroscope uses Thanos’ object store client library, so their stated limitations apply.
Amazon S3
To use an AWS S3 or S3-compatible bucket for long term storage, you can find Pyroscope’s configuration parameters in the reference config. Apart from those parameters, it is also possible to supply configuration parameters using the well-known environment variables of the AWS SDK.
At a minimum, you will need to provide a values for the bucket_name
, endpoint
, access_key_id
, and secret_access_key
keys.
Example using an AWS Bucket
This how one would configure a bucket in the AWS region eu-west-2
:
storage:
backend: s3
s3:
bucket_name: #REPLACE_WITH_BUCKET_NAME
region: eu-west-2
endpoint: s3.eu-west-2.amazonaws.com
access_key_id: #REPLACE_WITH_ACCESS_KEY
secret_access_key: #REPLACE_WITH_SECRET_KEY
Example using a S3 compatible Bucket
This how one would configure a bucket on a locally running instance of MinIO:
storage:
backend: s3
s3:
bucket_name: grafana-pyroscope-data
endpoint: localhost:9000
insecure: true
access_key_id: grafana-pyroscope-data
secret_access_key: grafana-pyroscope-data
Google Cloud Storage
To use a Google Cloud Storage (GCS) bucket for long term storage, you can find Pyroscope’s configuration parameters in the reference config.
At a minimum, you will need to provide a values for the bucket_name
and a service account. To supply the service account there are two ways:
- Use the
GOOGLE_APPLICATION_CREDENTIALS
environment variable to locate your application credentials. - Provide the content the service account key within the
service_account
parameter.
Example using a Google Cloud Storage bucket
This how one would configure a GCS bucket using the service_account
parameter:
storage:
backend: gcs
gcs:
bucket_name: grafana-pyroscope-data
service_account: |
{
"type": "service_account",
"project_id": "PROJECT_ID",
"private_key_id": "KEY_ID",
"private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n",
"client_email": "SERVICE_ACCOUNT_EMAIL",
"client_id": "CLIENT_ID",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL"
}
Azure Blob Storage
To use an Azure Blob Storage bucket for long term storage, you can find Pyroscope’s configuration parameters in the reference config.
If user_assigned_id
is used, authentication is done via user-assigned managed identity.
Swift (OpenStack Object Storage)
To use a Swift (OpenStack Object Storage) bucket for long term storage, you can find Pyroscope’s configuration parameters in the reference config.
If the
name
of a user, project or tenant is used one must also specify its domain by ID or name. Various examples for OpenStack authentication can be found in the official documentation.