Parser declarations

A parser declaration takes an original event or a specific field of a partially parsed event as input, analyzes its individual parts, and stores them as key-value pairs to the event.

LogMan.io Parsec currently supports three types of parser declarations:

• Parser-combinator
• JSON parser
• XML parser
• Windows Events parser

Declaration structure

In order to determine the type of the declaration, you need to specify a define section.

define:
    type: parsec/<type>

For a parser declaration:

define:
    type: parsec/parser

Parser-combinator

Parser-combinator (parsec) is used for parsing events in plain string format. It is based on SP-Lang Parsec expressions.

For parsing original events, use the following declaration:

parser.yaml

define:
    name: My Parser
    type: parsec/parser

parse:
    !PARSE.KVLIST
    - ...
    - ...
    - ...

subparser.yaml

define:
    name: My Parser
    type: parsec/parser
    field: <custom_field>

parse:
    !PARSE.KVLIST
    - ...
    - ...
    - ...

When field is specified, parsing is applied on that field, otherwise it is applied on the original event. Therefore, it must be present in every sub-parser.

Examples of parser-combinator declarations

Example 1: Simple example

For the purpose of the example, let's say that we want to parse a collection of simple events:

Hello Miroslav from Prague!
Hi Kristýna from Pilsen.

Example of parser declaration:

define:
    type: parsec/parser

parse:
    !PARSE.KVLIST
    - !PARSE.UNTIL " " # 'Hello '
    - name: !PARSE.UNTIL " " # 'Miroslav '
    - !PARSE.EXACTLY "from " # 'from '
    - city: !PARSE.LETTERS # 'Prague'
    - !PARSE.CHARS # '!'

Outputs:

{
    "name": "Miroslav",
    "city": "Prague"
}

{
    "name": "Kristýna",
    "city": "Pilsen"
}

Example 2: More complex example

For the purpose of this example, let's say that we want to parse a collection of simple events:

Process cleaning[123] finished with code 0.
Process log-rotation finished with code 1.
Process cleaning[657] started.

And we want the output in the following format:

{
    "process.name": "cleaning",
    "process.pid": 123,
    "event.action": "process-finished",
    "return.code": 0
}

{
    "process.name": "log-rotation",
    "event.action": "process-finished",
    "return.code": 1
}

{
    "process.name": "cleaning",
    "process.pid": 657,
    "event.action": "process-started",
}

Declaration will be the following:

10_parser.yaml

define:
    type: parsec/parser

parse:
    !PARSE.KVLIST
    - !PARSE.UNTIL " "
    - !TRY
        - !PARSE.KVLIST
            - process.name: !PARSE.UNTIL "["
            - process.pid: !PARSE.UNTIL "]"
            - !PARSE.SPACE
        - !PARSE.KVLIST
            - process.name: !PARSE.UNTIL " "
    - !TRY
        - !PARSE.KVLIST
            - !PARSE.EXACTLY "started."
            - event.action: "process-started"
        - !PARSE.KVLIST
            - !PARSE.EXACTLY "finished with code "
            - event.action: "process-finished"
            - return.code: !PARSE.DIGITS

Example 3: Parsing syslog events

For the purpose of the example, let's say that we want to parse a simple event in syslog format:

<189>Sep 22 10:31:39 server-abc server-check[1234]: User "harry potter" logged in from 198.20.65.68

We would like the output in the following format:

{
    "log.syslog.priority": 189,
    "@timestamp": 1695421899,
    "host.hostname": "server-abc",
    "process.name": "server-check",
    "process.pid": 1234,
    "user.name": "harry potter",
    "source.ip": "198.20.65.68"
}

We will create two parsers. First parser will parse the syslog header and the second will parse the message.

10_parser.yaml

define:
    name: Syslog parser
    type: parsec/parser

parse:
    !PARSE.KVLIST
    - "<"
    - PRI: !PARSE.DIGITS # 189
    - ">"
    - "@timestamp": !PARSE.DATETIME RFC3164 # Sep 22 10:31:39
    - host.hostname: !PARSE.UNTIL " " # server-abc
    - process.name: !PARSE.UNTIL "[" # server-check
    - process.pid: !PARSE.UNTIL "]" # 1234
    - ":"
    - !PARSE.SPACES
    - message: !PARSE.CHARS # User "harry potter" logged in from 198.20.65.68

20_parser_message.yaml

    define:
        type: parsec/parser
        field: message

    parse:
        !PARSE.KVLIST
        - !PARSE.UNTIL " " # User
        - user.name: !PARSE.BETWEEN { what: '"' } # harry potter
        - " "
        - !PARSE.UNTIL " " # logged
        - !PARSE.UNTIL " " # in
        - !PARSE.UNTIL " " # from
        - source.ip: !PARSE.CHARS

JSON parser

JSON parser is used for parsing events with JSON structure.

parser_json.yaml

define:
    name: JSON parser
    type: parsec/parser/json

This is a complete JSON parser and will parse events into JSON object, which can be later referenced in sub-parsers and mapping.

XML parser

XML parser is used for parsing events with XML structure.

parser_xml.yaml

define:
    name: XML parser
    type: parsec/parser/xml

This is a complete XML parser and will parse events into XML object, which can be later referenced in sub-parsers and mapping.

Windows Event parser

Windows Events parser is used for parsing events that are produced from Microsoft Windows. These events are in XML format.

define:
    name: Windows Events Parser
    type: parsec/parser/windows-event

This is a complete Windows Event parser and will parse events from Microsoft Windows, separating the fields into key-value pairs.