logfmt

The logfmt stage is a parsing stage that reads the log line as logfmt and allows extraction of data into labels.

Schema

logfmt:
  # Set of key/value pairs for mapping of logfmt fields to extracted labels. The YAML key will be
  # the key in the extracted data, while the expression will be the YAML value. If the value
  # is empty, then the logfmt field with the same name is extracted.
  mapping:
    [ <string>: <string> ... ]

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

This stage uses the go-logfmt unmarshaler, which means non-string types like numbers or booleans will be unmarshaled into those types. The extracted data can hold non-string values, and this stage does not do any type conversions; downstream stages will need to perform correct type conversion of these values as necessary. Please refer to the template stage for how to do this.

If the value extracted is a complex type, its value is extracted as a string.

Examples

Using log line

For the given pipeline:

- logfmt:
    mapping:
      timestamp: time
      app:
      duration:
      unknown:

Given the following log line:

time=2012-11-01T22:08:41+00:00 app=loki level=WARN duration=125 message="this is a log line" extra="user=foo""

The following key-value pairs would be created in the set of extracted data:

timestamp: 2012-11-01T22:08:41+00:00
app: loki
duration: 125

Using extracted data

For the given pipeline:

- logfmt:
    mapping:
      extra:
- logfmt:
    mapping:
      user:
    source: extra

And the given log line:

time=2012-11-01T22:08:41+00:00 app=loki level=WARN duration=125 message="this is a log line" extra="user=foo"

The first stage would create the following key-value pairs in the set of extracted data:

extra: user=foo

The second stage will parse the value of extra from the extracted data as logfmt and append the following key-value pairs to the set of extracted data: