This is documentation for the next version of Loki. For the latest stable release, go to the latest version.

Documentation Grafana Loki Release notes v3.3

v3.3

Grafana Labs and the Loki team are excited to announce the release of Loki 3.3. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.3.0 include the following:

• Query acceleration with Bloom filters (experimental): The Bloom filters feature introduced in Loki 3.0 remains experimental. There has been a major shift in how Loki leverages bloom filters — pivoting from free-text search to harnessing the power of structured metadata (#14061) (a2fbaa8). This change leverages the growing adoption of OpenTelemetry in observability stacks. The Bloom Compactor component has been removed and replaced by the Bloom Planner and Bloom Builder components (#13997) (bf60455) and blooms has a new block schema (V3) (#14038) (5395daf) which is a breaking change.

• Explore Logs: Starting with Grafana v11.3, the plugin for the Explore Logs app is installed in Grafana by default. This release includes enhancements to improve pattern ingester tracing (#14707) (80aec25), to apply patterns line length limit to json message keys (#14296) (41fafd8), add configuration to set max-allowed-line-length in the pattern ingester (#14070) (0780456) and move metric aggregation to a per-tenant configuration (#14709) (c1fde26).

• helm: Multiple improvements to the Helm charts, including: Updating the chart with Loki version 3.2.0 (#14281) (11b92ee), replacing bloom compactor with bloom planner and builder (#14003) (08e61ca), adding the ability to configure client_max_body_size (#12924) (809a024), adding additional service annotations for components in distributed mode (#14131) (5978f13), adding configurable extraEnvFrom to admin-api and enterprise-gateway (#14533) (5d78a3a), adding kubeVersionOverride for Helm chart (#14434) (0935d77), adding persistence option to memcached on Helm chart (#13619) (ef1df0e), adding tolerations to pattern-ingester statefulset (#13605) (09530c0), allowing setting node attributes to tokengen and provisioner (#14311) (c708ae6), updating Loki Helm chart for restricted environments (#14440) (adc7538), and supporting ruler sidecar in singleBinary mode (#13572) (684baf7).

• mixins: Multiple improvements to the Loki mixins, including: adding a missing cluster label to mixins (#12870) (547ca70), adding support for partition ingester in dashboards (#14498) (70deebf), adding a Loki compaction not successful alert (#14239) (da04f50), allowing overriding of some labels by parameterizing mixin recording/alert rules (#11495) (f1425b6), allowing disabling of bloom dashboards (#14177) (ce2e6d5), allowing hiding of unused rows in loki-operational (#13646) (3aa4f22), and merging read-resources and write-resources dashboards for SSD into one dashboard that also includes the backend-resources(#13471) (45b8719).

• operator: Multiple updates to the Operator, including: Adding support for Loki OTLP limits configuration (#13446) (d02f435), providing default OTLP attribute configuration (#14410) (1b52387), adding a User-guide for OTLP configuration (#14620) (27b4071), updating the Loki operand to v3.1.1 (#14042) (7ae1588), updating the Loki operand to v3.2.1 (#14526) (5e970e5), renaming the Loki API go module (#14568) (976d8ab), introducing 1x.pico size (#14407) (57de81d), migrating the project layout to kubebuilder go/v4 (#14447) (dbb3b6e), declaring feature FIPS support for OpenShift only (#14308) (720c303).

• docs: New videos and tutorials, including a New Cloud deployment guide for Deploying Loki on AWS, a new Fluent Bit tutorial and a Getting Started with OpenTelemetry Collector tutorial. Also there are improvements to the Query documentation, including a new Query Best Practices topic, a reorganized Query Template, and a revised LogQL Simulator topic.

Other improvements include the following:

• api: Added endpoint to return detected field values (#14350) (7983f94).
• blooms: Add disk-backed queue for the bloom-planner (backport k227) (#14927) (1f6828b).
• blooms: Only write key and key=value to blooms (#14686) (3af0004).
• blooms: Do not add empty blooms to offsets (#14577) (51c42e8).
• blooms: Extract task computing into a strategy interface (#13690) (ab5e6ea).
• blooms: New bloom planning using chunk size TSDB stats (#14547) (673ede1).
• blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f).
• compactor: Add backoff mechanism to the retention process (#14182) (3136880).
• distributor: Track discarded data by usageTracker (#14081) (c65721e).
• distributor: Add ability to log stream selectors before service name detection (#14154) (d7ff426).
• distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a).
• fluentd: Support custom http headers (#14299) (e59035e).
• ingester: Implement owned streams calculation using Partition Ring (#14282) (3c36ba9).
• ingester: Implement partition shuffle sharding for ingester (#14304) (1a4436c).
• jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177).
• labels: Add app_name as a service label (#13660) (f2a16f4).
• logcli: Add gzip compression option (#14598) (4d3f9f5).
• logql: Introduce shardable probabilistic topk for instant queries. (backport k227) (#14765) (02eb024).
• logql: Micro-optimizations for IP filter (#14072) (c5083c7).
• logql: Add query user and query source to “executing query” log lines (#14320) (4d69929).
• promtail: Add structured metadata to the promtail push API (#14153) (66cffcb).
• querier: Move detected field logic to query frontend (#14212) (36ace66).
• ruler: Enable ruler store that uses clients from thanos-io/objstore pkg (#11713) (8bca2e7).
• storage: Add retries for s3 ObjectExists calls (#14062) (73cbbb0).
• storage: Implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63).
• storage: Introduce new ObjectExistsWithSize API to (#14268) (ac422b3).
• storage: AWS backend using thanos.io/objstore (#11221) (b872246).
• storage: Azure backend using thanos.io/objstore (#11315) (5824e3d).
• storage: GCS backend using thanos.io/objstore (#11132) (c059ace).
• structured metadata: Include structured_metadata size while asserting rate limit (#14571) (a962edb).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.

• Deprecated configuration options.

• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

• BREAKING CHANGE - blooms: Introduce a new block schema (V3) (#14038).
• BREAKING CHANGE - blooms: Index structured metadata into blooms (#14061).
• BREAKING CHANGE - operator: Migrate project layout to kubebuilder go/v4 (#14447).
• BREAKING CHANGE - operator: Rename Loki API go module (#14568).
• BREAKING CHANGE - operator: Provide default OTLP attribute configuration (#14410).

Bug fixes

3.3.0 (2024-11-19)

• blooms: Add tenant limits as dependency to pattern ingester (#14665) (31eea90).
• blooms: Check length of tasks before accessing first element in slice (#14634) (601f549).
• blooms: Copy chunks from ForSeries (backport k227) (#14864) (d10f79c).
• blooms: Do not restart builders when planner disconnects (backport k227) (#14922) (213e8ee).
• blooms: Exclude label filters where label name is part of the series labels. (#14661) (d1668f6).
• blooms: Fix panic in initialization of the bloom planner and builder (#14110) (8307c42).
• blooms: Fix strategy logger and add task test (backport k227) (#14921) (dc36a1e).
• blooms: Fix tenants slice on loadTenantTables (backport k227) (#14901) (540dd5a).
• blooms: Skip multi-tenant TSDBs during bloom planning (backport k227) (#14888) (631cff3).
• blooms: Improve docs for min and max table offsets (backport k227) (#14929) (3161fdc).
• blooms: Fix lint errors (#14574) (99ef900).
• build: Use Debian Bullseye base image for build image (#14368) (3beb8ff).
• canary: Reconnect immediately upon tail max duration (#14287) (9267ee3).
• ci: Fix Publish Rendered Helm Chart Diff workflow (#14365) (6de6420).
• ci: Update helm diff rendering workflow (#14424) (916e511).
• config: Copy Alibaba and IBM object storage configuration from common (#14297) (59ff1ec).
• config: Migrate renovate configuration (#14646) (a67d8ef).
• deps: Update aws-sdk-go-v2 monorepo (#13986) (6f49123).
• deps: update aws-sdk-go-v2 monorepo (#14742) (53a1ab7).
• deps: update github.com/grafana/dskit digest to 687ec48 (#14395) (c2f38e1).
• deps: update github.com/grafana/dskit digest to 7c41a40 (#14277) (f39cdbd).
• deps: update github.com/grafana/dskit digest to 931a021 (#14032) (7c18642).
• deps: update github.com/grafana/dskit digest to b69ac1b (#14355) (9d7a6ea).
• deps: update github.com/grafana/dskit digest to f52de24 (#14319) (a4f3edf).
• deps: update github.com/twmb/franz-go/pkg/kfake digest to cea7aa5 (#14590) (688c42a).
• deps: update k8s.io/utils digest to 702e33f (#14033) (b7eecc7).
• deps: Update module cloud.google.com/go/bigtable to v1.33.0 (#14580) (a0920ed).
• deps: Update module cloud.google.com/go/pubsub to v1.45.0 (#14361) (4351238).
• deps: Update module cloud.google.com/go/pubsub to v1.45.1 (#14650) (f173708).
• deps: Update module cloud.google.com/go/storage to v1.46.0 (#14744) (8e45116).
• deps: Update module github.com/alicebob/miniredis/v2 to v2.33.0 (#14721) (7bfda25).
• deps: Update module github.com/aws/aws-sdk-go to v1.55.5 (#14715) (03f0f5a).
• deps: Update module github.com/axiomhq/hyperloglog to v0.2.0 (#14722) (0167b22).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.189 (#14044) (7fb34b4).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.192 (#14337) (6f7cae2).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.196 (#14651) (478085a).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.197 (#14682) (b898294).
• deps: Update module github.com/coder/quartz to v0.1.2 (#14652) (7459e07).
• deps: Update module github.com/felixge/fgprof to v0.9.5 (#14338) (a2ad3aa).
• deps: Update module github.com/fsouza/fake-gcs-server to v1.50.2 (#14313) (275c97c).
• deps: Update module github.com/hashicorp/raft to v1.7.1 (#14005) (e9cec1d).
• deps: Update module github.com/ibm/go-sdk-core/v5 to v5.17.5 (#14045) (677d217).
• deps: Update module github.com/ibm/go-sdk-core/v5 to v5.18.1 (#14716) (8395acd).
• deps: Update module github.com/ibm/ibm-cos-sdk-go to v1.11.1 (#14342) (aa82a7c).
• deps: Update module github.com/klauspost/compress to v1.17.10 (#14352) (e23c5ed).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.76 (#14006) (51f9376).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.77 (#14353) (d0e3ef7).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.80 (#14654) (eec2513).
• deps: Update module github.com/ncw/swift/v2 to v2.0.3 (#14356) (c843288).
• deps: Update module github.com/prometheus/client_golang to v1.20.5 (#14655) (e12f843).
• deps: Update module github.com/schollz/progressbar/v3 to v3.17.0 (#14720) (4419d0f).
• deps: Update module github.com/shirou/gopsutil/v4 to v4.24.10 (#14719) (3280376).
• deps: Update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357) (c8e6a9d).
• deps: Update module github.com/shopify/sarama to v1.43.3 (#14059) (1cf4813).
• deps: Update module github.com/spf13/afero to v1.11.0 (#14060) (bbbd82b).
• deps: Update module go.etcd.io/bbolt to v1.3.11 (#14358) (b7bccfc).
• deps: Update module golang.org/x/net to v0.29.0 (#14341) (1b6b9da).
• deps: Downgrade grpc to fix regression (#14065) (8c38d46).
• deps: Fix missing dep PartitionRing for Ingester (#14292) (6354ded).
• deps: nix build, downgrade toolchain to go1.23.1 (#14442) (26dfd62).
• deps: Revert “fix(deps): Update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357)” (#14437) (d53955b).
• detected_fields: Return parsed labels when parsers are passed (#14047) (aa1ac99).
• detected_fields: Always return empty array as null (#14112) (93009d4).
• docs: Correct OTLP documentation typo (#14602) (063c590).
• distributor: Always write detected_level when enabled, even if unknown (#14464) (41c6b6c).
• distributor: Validate partition ring is kafka is enabled (#14303) (8438d41).
• distributor: Level detection for warning level (#14444) (242a852).
• explore logs: Correctly create logfmt string (#14124) (63e84b4).
• explore logs: Allow any level for aggregated metrics (#14255) (c001a1d).
• helm: Add missing loki.storage.azure.chunkDelimiter parameter to Helm chart (#14011) (08c70cc).
• helm: Check for rbac.namespaced condition before creating roles (#14201) (3f47f09).
• helm: Fix persistence configuration for Memcached (#14049) (ee6e1cf).
• helm: Fix wrong port name referenced for ingress NetworkPolicy (#12907) (963a25b).
• helm: Various fixes and enhancements for bloom components (#14128) (dc0cbd4).
• ingester: Report PSRL error message correctly (#14187) (a475153).
• ingester: Wait for OwnedStreams service in Ingester startup (#14208) (a4aee4f).
• lambda-promtail: Add s3 principal to iam policy attached to sqs in lambda-promtail terraform code (#14619) (db0889e).
• logcli: Check for errors before checking for exists when fetching data (backport k227) (#14906) (31b2a63).
• logcli: Create new tail response for every line (#14525) (bcfd0d1).
• logql: Fix panic in json parsing when using empty array index (#14393) (833bf0d).
• logql: Update JSONExpressionParser to not unescape extracted values if it is JSON object. (#14499) (08b1a90).
• logql: Allow rename of structured metadata labels (#13955) (2d4792a).
• logql: Renamed label should use ParsedLabel category (#14515) (82fb2f0).
• logql: Skipping label if it contains special symbol (#14068) (55e374e).
• mixin: Remove pod label from disk usage aggregation (#14180) (5d45c96).
• mixins: Fix loki-resources-overview panel layout (#14178) (8f54ec6).
• mixins: Add backend path section in loki-operational for single scalable deployment (#13023) (16881ab).
• mixins: Disk space utilization panels with latest KSM versions (#13486) (0ea7431).
• mixins: Retention dashboards fix metric name (#14617) (c762b9b).
• oltp: Enable service detection for otlp endpoint (#14036) (4f962ef).
• operator: Add 1x.pico OpenShift UI dropdown menu (#14660) (4687f37).
• operator: Add missing groupBy label for all rules on OpenShift (#14279) (ce7b2e8).
• operator: Correctly ignore again BlotDB dashboards (#14587) (4879d10).
• operator: Disable automatic discovery of service name (#14506) (3834c74).
• operator: Disable log level discovery for OpenShift tenancy modes (#14613) (5034d34).
• operator: Fix building the size-calculator image (#14573) (a79b8fe).
• operator: Fix make build target for size-calculator (#14551) (e727187).
• operator: Move OTLP attribute for statefulset name to stream labels (#14630) (5df3594).
• operator: Use empty initialized pod status map when no pods (#14314) (6f533ed).
• operator: Update renovate ignore for operator API with new module path (#14581) (c9b2907).
• pattern: Fixes latency metric namespace for tee to pattern (#14241) (ae955ed).
• promtail: Fix configuration unmarshalling (#14408) (a05431f).
• promtail: Fix parser for azureeventhubs message without time field (#14218) (2e62abb).
• promtail: Validate scrape_config job name, do not allow duplicate job names (#13719) (f2d3499).
• querier: Propagate query stats from quantile & topk queries (#13831) (78b275b).
• querier: Correct _extracted logic in detected fields (#14064) (1b3ba53).
• querier: Timestamp. Time should be called with milliseconds (#14196) (f8d9143).
• querier: Report correct status code for metric and log queries in metrics.go (#12102) (900751c).
• query: Fix bug in query result marshaling for invalid utf8 characters (#14585) (f411a07).
• Remove usage of unsafe string in label adapter unmarshall (#14216) (758364c).
• Rename misspelled filename (#14237) (cf1d4a3).
• sharding: Apply offset to both from and through in shard request (#14256) (17c472d).
• storage: Add additional validation for timeout while retrieving headers (#14217) (8322e51).
• storage: Do not retain span logger created with index set initialized at query time (#14027) (4e41744).
• storage: Expand matching for additional variations (#14221) (71d7291).
• storage: Disable client retries when congestion control is enabled in S3(#14588) (cff9f43).
• storage: Have GetObject check for canceled context (#14420) (5f325aa).
• storage: Transform ObjectExistsWithSize into GetAttributes (#14329) (2f56f50).
• storage: Update AWS storage timeout error for Go 1.23 behavior (#14226) (a4642b5).