Log::Fmt(3pm)

The logfmt text format

Although quite a few tools exist for managing "logfmt", there is no spec-like document for it. Because you may require multiple implementations, a specification can be helpful.

Every logfmt event is a sequence of pairs in the form "key=value". Pairs are separated by a single space.

    event = pair *(WSP pair)
    pair  = key "=" value
    okchr = %x21 / %x23-3c / %x3e-5b / %x5d-7e ; VCHAR minus \ and " and =
    key   = 1*(okchr)
    value = key / quoted
    quoted = DQUOTE *( escaped / quoted-ok / okchr / eightbit ) DQUOTE
    escaped         = escaped-special / escaped-hex
    escaped-special = "\\" / "\n" / "\r" / "\t" / ("\" DQUOTE)
    escaped-hex     = "\x{" 2HEXDIG "}" ; lowercase forms okay also
    quoted-ok       = SP / "="
    eightbit        = %x80-ff

When formatting a value, if a value is already a valid "key" token, use it without further quoting.

Quoting a Unicode string

It is preferable to build quoted values from a Unicode string, because it's possible to know whether a given codepoint is a non-ASCII unsafe character, like "LINE SEPARATOR". Safe non-ASCII characters can be directly UTF-8 encoded, rather than quoted with "\x{...}". In that way, viewing logfmt events with a standard terminal can show something like:

    user.name="Jürgen"

To generate a "quoted" from a Unicode string, for each codepoint:

Finally, UTF-8 encode the entire string and wrap it in double qoutes.

This Perl implementation assumes that all string values to be encoded are character strings!

Quoting a bytestring

Encoding a Unicode string is preferable, but may not be practical. In those cases when you have only a byte sequence, apply these steps.

For each byte (using ASCII conventions):

Finally, wrap the string in double quotes.

Special values

There are no reserved or sentinel values for undefined values, null pointers, or the like. Convention is to use a single identifier surrounded by tildes to indicate such a value. For example, given a null pointer for the "user.name" value, a system might log:

    user.name=~null~

If "~null~" was a valid username, there would be ambiguity here. The choice of exact sentinels is left to the implementing system.

  my $octets = Log::Fmt->format_event_string([
    key1 => $value1,
    key2 => $value2,
  ]);

Note especially that if any value to encode is a reference to a reference, then String::Flogger is used to encode the referenced value. This means you can embed, in your logfmt, a JSON dump of a structure by passing a reference to the structure, instead of passing the structure itself.

String values are assumed to be character strings, and will be UTF-8 encoded as part of the formatting process.

  my $kv_pairs = Log::Fmt->parse_event_string($octets);

Given the kind of (byte) string emitted by "format_event_string", this method returns a reference to an array of key/value pairs. After being unquoted, value strings will be UTF-8 decoded into character strings.

This isn't exactly a round trip. First off, the formatting can change illegal keys by replacing characters with question marks, or replacing empty strings with tildes. Secondly, the formatter will expand some values like arrayrefs and hashrefs into multiple keys, but the parser will not recombined those keys into structures. Also, there might be other asymmetric conversions. That said, the string escaping done by the formatter should correctly reverse.

If the input string is badly formed, hunks that don't appear to be value key/value pairs will be presented as values for the key "junk".

    my $hashref = Log::Fmt->parse_event_string_as_hash($line);

This parses the given line as logfmt, then puts the key/value pairs into a hash and returns a reference to it.

Because nothing prevents a single key from appearing more than once, you should use this with the understanding that data could be lost. No guarantee is made of which value will be preserved.