Parsers

A parser declaration takes an original event or a specific field of a partially parsed event as input, analyzes its individual parts, and stores them as key-value pairs to the event.

LogMan.io Parsec currently supports three types of parser declarations:

• JSON parser
• Windows Event parser
• Parsec parser

Declaration structure

In order to determine the type of the declaration, you need to specify a define section.

define:
    type: <declaration_type>

For a parser declaration, specify the type as parser.

JSON parser

A JSON parser is used for parsing events with a JSON structure.

parser_json.yaml

define:
    name: JSON parser
    type: parser/json

This is a complete JSON parser and will parse events from a JSON structure, separating the fields into key-value pairs.

Warning

For now, LogMan.io Parsec does not support parsing of events with nested JSON format. For example, the event below cannot be parsed with JSON parser:

{
    "key": {
        "foo": 1,
        "bar": 2
    }
}

Windows Event parser

Windows Events parser is used for parsing events that are produced from Microsoft Windows. These events are in XML format.

define:
    name: Windows Events Parser
    type: parser/windows-event

This is a complete Windows Event parser and will parse events from Microsoft Windows, separating the fields into key-value pairs.

Parsec parser

A Parsec parser is used for parsing events in plain string format. It is based on SP-Lang Parsec expressions.

For parsing original events, use the following declaration:

parser.yaml

define:
    name: My Parser
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - ...
        - ...
        - ...

subparser.yaml

define:
    name: My Parser
    type: parser/parsec
    field: <custom_field>

parse:
    !PARSE.KVLIST
        - ...
        - ...
        - ...

When field is specified, parsing is applied on that field, otherwise it is applied on the original event. Therefore, it must be present in every sub-parser.

Examples of Parsec parser declarations

Example 1: Simple example

For the purpose of the example, let's say that we want to parse a collection of simple events:

Hello Miroslav from Prague!
Hi Kristýna from Pilsen.

{
    "name": "Miroslav",
    "city": "Prague"
}

{
    "name": "Kristýna",
    "city": "Pilsen"
}

define:
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.UNTIL " "
        - name: !PARSE.UNTIL " "
        - !PARSE.EXACTLY "from "
        - city: !PARSE.LETTERS

Example 2: More complex example

For the purpose of this example, let's say that we want to parse a collection of simple events:

Process cleaning[123] finished with code 0.
Process log-rotation finished with code 1.
Process cleaning[657] started.

And we want the output in the following format:

{
    "process.name": "cleaning",
    "process.pid": 123,
    "event.action": "process-finished",
    "return.code": 0
}

{
    "process.name": "log-rotation",
    "event.action": "process-finished",
    "return.code": 1
}

{
    "process.name": "cleaning",
    "process.pid": 657,
    "event.action": "process-started",
}

Declaration will be the following:

10_parser.yaml

define:
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.UNTIL " "
        - !TRY
            - !PARSE.KVLIST
                - process.name: !PARSE.UNTIL "["
                - process.pid: !PARSE.UNTIL "]"
                - !PARSE.SPACE
            - !PARSE.KVLIST
                - process.name: !PARSE.UNTIL " "
        - !TRY
            - !PARSE.KVLIST
                - !PARSE.EXACTLY "started."
                - event.action: "process-started"
            - !PARSE.KVLIST
                - !PARSE.EXACTLY "finished with code "
                - event.action: "process-finished"
                - return.code: !PARSE.DIGITS

Example 3: Parsing syslog events

For the purpose of the example, let's say that we want to parse a simple event in syslog format:

<189> Sep 22 10:31:39 server-abc server-check[1234]: User "harry potter" logged in from 198.20.65.68

We would like the output in the following format:

{
    "PRI": 189,
    "timestamp": 1695421899,
    "server": "server-abc",
    "process.name": "server-check",
    "process.pid": 1234,
    "user": "harry potter",
    "action": "log-in",
    "ip": "198.20.65.68"
}

We will create two parsers. First parser will parse the syslog header and the second will parse the message.

10_parser.yaml

define:
    name: Syslog parser
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.EXACTLY "<"
        - PRI: !PARSE.DIGITS
        - !PARSE.EXACTLY ">"

        - timestamp: ...
        - server: !PARSE.UNTIL " "
        - process.name: !PARSE.UNTIL "["
        - process.pid: !PARSE.UNTIL "]"
        - !PARSE.EXACTLY ":"
        - message: !PARSE.CHARS

This parser

20_parser_message.yaml

    define:
        type: parser/parsec
        field: message
        drop: yes

    parse:
        !PARSE.KVLIST
            - !PARSE.UNTIL " "
            - user: !PARSE.BETWEEN { what: '"' }
            - !PARSE.EXACTLY " "
            - !PARSE.UNTIL " "
            - !PARSE.UNTIL " "
            - !PARSE.UNTIL " "
            - ip: !PARSE.CHARS