Cascade Parser

Example

---
define:
  name: Syslog RFC5424
  type: parser/cascade
  field_alias: field_alias.default
  encoding: utf-8  # none, ascii, utf-8 ... (default: utf-8)
  target: parsed # optional, specify the target of the parsed event (default: parsed)

predicate:
  !AND
  - !CONTAINS
    what: !EVENT
    substring: 'ASA'
  - !INCLUDE predicate_filter

parse:
  !REGEX.PARSE
  what: !EVENT
  regex: '^(\w{1,3}\s+\d+\s\d+:\d+:\d+)\s(?:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|([^\s]+))\s%ASA-\d+-(.*)$'
  items:
    - rt:
        !DATETIME.PARSE
        value: !ARG
        format: '%b %d %H:%M:%S'
        flags: Y
    - dvchost

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be parser/cascade.

Item field_alias

Name of the field alias lookup to be loaded, so that alias names of event attributes can be used in the declaration alongside their canonical names.

Item encoding

Encoding of the incoming event.

Item target (optional)

Default target pipeline of the parsed event, unless specified differently in context. The options include: parsed, lookup, unparsed

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section predicate (optional)

The predicate filters incoming events using an expression. If the expression returns True, the event will enter parse section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

This section can be used to speed-up parsing by skipping lines with obviously non-relevant content.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in include or filters folder (this one is a global folder located at the top hierarchy of the LogMan.io library), use !INCLUDE statement:

!INCLUDE predicate_filter

where predicate_filter is the name of the file plus .yaml extension. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section parse

This section specifies the actual parsing mechanism. It expects a dictionary to be returned or None, which means that the parsing was not successful.

Typical statements in parse section

!FIRST statement allows to specify a list of parsing declarations, which will be evaluated in the order (top-down), the first declaration which returns non-None value stops the iteration and this value is returned.

!REGEX.PARSE statement allows to transform the log line into a dictionary structure. It also allows to attach sub-parsers to further decompose substrings.

Output routing

To indicate that the parser will not parse the event it received so far, an attribute target needs to be set to unparsed within the context. Then, other parsers in the pipeline may receive and parse the event.

In the same way, the target can be set to different destination groups, such as parsed.

To set the target in the context, the !CONTEXT.SET is used:

- !CONTEXT.SET
  what: <... expression ...>
  set:
    target: unparsed

Example of use in the parser. If no regex matches the incoming event, event is posted to unparsed target, so other parsers in the row may process it.

!FIRST
- !REGEX.PARSE
what: !EVENT
regex: '^(one)\s(two)\s(three)$'
items:
  - one
  - two
  - three
- !REGEX.PARSE
what: !EVENT
regex: '^(uno)\s(duo)\s(tres)$'
items:
  - one
  - two
  - three
# This is where the handling of partially parsed event starts
- !CONTEXT.SET
set:
  target: unparsed
- !DICT
set:
  unparsed: !EVENT