Skip to content

Extending a parser's pipeline

We ship LogMan.io Library with standard parsers organized into pre-defined groups. However, sometimes you will want to extend the parsing process with custom parsers or enrichers.

Consider the following input event to be parsed with parsers from LogMan.io Library with group ID lmio_parser_default_syslog_rfc3164:

<163>Feb 22 14:12:56 vmhost01 2135: ERR042: Something went wrong.

Such event will be parsed into a structured event that looks like this:

{
   "@timestamp": 1614003176,
   "ecs.version": "1.6.0",
   "event.kind": "event",
   "event.dataset": "syslog.rfc3164",
   "message": "ERR042: Something went wrong.\n",
   "host.name": "vmhost01",
   "tenant": "default",
   "log.syslog.priority": 163,
   "log.syslog.facility.code": 20,
   "log.syslog.severity.code": 3,
   "event.ingested": 1614004510.4724128,
   "_s": "SzOe",
   "_id": "[ID]",
   "log.original": "<163>Feb 22 14:12:56 vmhost01 2135: ERR042: Something went wrong.\n"
}

The input event, however, contains another keyword of interest - an error code "ERR042", that is not part of the structured event. We can extract the value into a custom field of the structured event by adding an enricher (a type of a parser) that slices the "message" part of the event and picks up the error code.

Locate The Parsers Group To Extend

In the example above we use parsers with group ID lmio_parser_default_syslog_rfc3164. So let's navigate to this group's folder in the LogMan.io Library:

$ cd /opt/lmio-ecs # ... or your other location of lmio-ecs
$ cd syslog_rfc3164-parser

Create A New Declaration File

By default, with no extensions, there are these files in the parsers group's folder:

$ ls -l
p01-parser.YAML     p02-parser.YAML

These files contain parsers' declarations. For a declaration of the new enricher, create file e01-enricher.yaml.

  • The "e" stands for "enricher"
  • The "01" stands for the priority this enricher will be given
  • The "-enricher" can be replaced with anything meaningful to you
  • "yaml" is the mandatory extension

Add Contents To The Declaration File

Define

The Declaration is a YAML file with a YAML header (empty in our case) and a mandatory definition block. We are adding a standard enricher with the name "Error Code Enricher".

Append the following to the declaration file: 

---
define:
  name: Error Code Enricher
  type: enricher/standard

Predicate

We want our enricher to be applied to selected messages only, so we need to declare a Predicate using the declarative language.

Let's apply the enrichment to messages from host vmhost01.

Append the following to the declaration file:

predicate:
  !EQ
  - !ITEM EVENT host.name
  - "vmhost01"

Enrich

Looking at the "message" of the example event, we want to split the message by colons, take the value of the first item of results and store it as "error.code" (or another ECS field).

We can achieve that again with declarative language.

Append the following to the declaration file:

enrich:
  !DICT
  with: !EVENT
  set:
    error.code: !CUT
      what: !ITEM EVENT message
      delimiter: ':'
      field: 0

The result event passed to the parsers pipeline will consist of all fields from the original event and of one other field "error.code", the value of which is a result of !CUTting the "message" field from the original event (!ITEM EVENT message) using : as delimiter and picking up the item at index 0.

This is how the contents of e01-enricher.yaml look like as a result:

---
define:
  name: Error Code Enricher
  type: enricher/standard
predicate:
  !EQ
  - !ITEM EVENT host.name
  - "vmhost01"
enrich:
  !DICT
  with: !EVENT
  set:
    error.code: !CUT
      what: !ITEM EVENT message
      delimiter: ':'
      field: 0

Apply changes

The new declaration should be kept in version control. The lmio-parser instance that uses the parsers' group ID must be restarted.

Conclusion

We added a new enricher into the lmio_parser_default_syslog_rfc3164's parsers pipeline.

New events from the host vmhost01 will now be parsed and enriched resulting in this output event:

{
   "@timestamp": 1614003176,
   "ecs.version": "1.6.0",
   "event.kind": "event",
   "event.dataset": "syslog.rfc3164",
   "message": "ERR042: Something went wrong.\n",
   "host.name": "vmhost01",
   "tenant": "default",
   "log.syslog.priority": 163,
   "log.syslog.facility.code": 20,
   "log.syslog.severity.code": 3,
   "event.ingested": 1614004510.4724128,
   "_s": "SzOe",
   "_id": "[ID]",
   "log.original": "<163>Feb 22 14:12:56 vmhost01 2135: ERR042: Something went wrong.\n",
   "error.code": "ERR042"
}