Parsers¶

A parser declaration takes an original event or a specific field of a partially parsed event as input, analyzes its individual parts, and stores them as key-value pairs to the event.

LogMan.io Parsec currently supports three types of parser declarations:

JSON parser
Windows Event parser
Parsec parser

Declaration structure¶

In order to determine the type of the declaration, you need to specify a define section.

define:
    type: <declaration_type>

For a parser declaration, specify the type as parser.

JSON parser¶

JSON parser is used for parsing events (or events parts) with a JSON structure.

parser_json.yaml

define:
    name: JSON parser
    type: parser/json
    field: <custom_field>
    target: <custom_target>

When field is specified, parsing is applied to that field; by default, it is applied to the original event.

When target is specified, the parsed object is stored in the designated target field; by default, it is stored with json key. The custom target field must adhere to the regular expression json[0-9a-zA-Z] (beginning with "json" followed by any alphanumeric character).

Example

The following original event with a JSON structure is parsed by the JSON parser with the default settings:

{
    "key": {
        "foo": 1,
        "bar": 2
    }
}

10_parser.yaml

define:
    name: JSON parser
    type: parser/json

The result event will be:

{
    "json": <JSON object>,
}

The following event includes JSON part inside, so JSON parser can be applied on this preparsed field and the result will be stored in the custom jsonMessage field:

<14>1 2023-05-03 15:06:12 {"key": {"foo": 1, "bar": 2}}

20_parser_message.yaml

define:
    name: JSON parser
    type: parser/json
    field: message
    target: jsonMessage

The result event will be:

```json
{
    "log.syslog.priority": 14,
    "@timestamp": 140994182325993472,
    "message": "{"key": {"foo": 1, "bar": 2}}",
    "jsonMessage": <JSON object>
}
```

Windows Event parser¶

Windows Events parser is used for parsing events that are produced from Microsoft Windows. These events are in XML format.

define:
    name: Windows Events Parser
    type: parser/windows-event

This is a complete Windows Event parser and will parse events from Microsoft Windows, separating the fields into key-value pairs.

Parsec parser¶

A Parsec parser is used for parsing events in plain string format. It is based on SP-Lang Parsec expressions.

For parsing original events, use the following declaration:

parser.yaml

define:
    name: My Parser
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - ...
        - ...
        - ...

subparser.yaml

define:
    name: My Parser
    type: parser/parsec
    field: <custom_field>

parse:
    !PARSE.KVLIST
        - ...
        - ...
        - ...

When field is specified, parsing is applied on that field, otherwise it is applied on the original event. Therefore, it must be present in every sub-parser.

Types of field specification:

field: <custom_field> - regular field pre-parsed by the previous parser.
field: json /key/foo - JSON key /key/foo from pre-parsed JSON object json. Name of the JSON object and JSON key must be separated by space. JSON key is always starts with /, and every next level is separated by /.
JSON key with specified type, by default we assume string type.

field:
    json: /key/foo
        type: int

Examples of Parsec parser declarations¶

Example 1: Simple example

For the purpose of the example, let's say that we want to parse a collection of simple events:

Hello Miroslav from Prague!
Hi Kristýna from Pilsen.

{
    "name": "Miroslav",
    "city": "Prague"
}

{
    "name": "Kristýna",
    "city": "Pilsen"
}

define:
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.UNTIL " "
        - name: !PARSE.UNTIL " "
        - !PARSE.EXACTLY "from "
        - city: !PARSE.LETTERS

Example 2: More complex example

For the purpose of this example, let's say that we want to parse a collection of simple events:

Process cleaning[123] finished with code 0.
Process log-rotation finished with code 1.
Process cleaning[657] started.

And we want the output in the following format:

{
    "process.name": "cleaning",
    "process.pid": 123,
    "event.action": "process-finished",
    "return.code": 0
}

{
    "process.name": "log-rotation",
    "event.action": "process-finished",
    "return.code": 1
}

{
    "process.name": "cleaning",
    "process.pid": 657,
    "event.action": "process-started",
}

Declaration will be the following:

10_parser.yaml

define:
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.UNTIL " "
        - !TRY
            - !PARSE.KVLIST
                - process.name: !PARSE.UNTIL "["
                - process.pid: !PARSE.UNTIL "]"
                - !PARSE.SPACE
            - !PARSE.KVLIST
                - process.name: !PARSE.UNTIL " "
        - !TRY
            - !PARSE.KVLIST
                - !PARSE.EXACTLY "started."
                - event.action: "process-started"
            - !PARSE.KVLIST
                - !PARSE.EXACTLY "finished with code "
                - event.action: "process-finished"
                - return.code: !PARSE.DIGITS

Example 3: Parsing syslog events

For the purpose of the example, let's say that we want to parse a simple event in syslog format:

<189> Sep 22 10:31:39 server-abc server-check[1234]: User "harry potter" logged in from 198.20.65.68

We would like the output in the following format:

{
    "PRI": 189,
    "timestamp": 1695421899,
    "server": "server-abc",
    "process.name": "server-check",
    "process.pid": 1234,
    "user": "harry potter",
    "action": "log-in",
    "ip": "198.20.65.68"
}

We will create two parsers. First parser will parse the syslog header and the second will parse the message.

10_parser.yaml

define:
    name: Syslog parser
    type: parser/parsec

parse:
    !PARSE.KVLIST
        - !PARSE.EXACTLY "<"
        - PRI: !PARSE.DIGITS
        - !PARSE.EXACTLY ">"

        - timestamp: ...
        - server: !PARSE.UNTIL " "
        - process.name: !PARSE.UNTIL "["
        - process.pid: !PARSE.UNTIL "]"
        - !PARSE.EXACTLY ":"
        - message: !PARSE.CHARS

This parser

20_parser_message.yaml

    define:
        type: parser/parsec
        field: message
        drop: yes

    parse:
        !PARSE.KVLIST
            - !PARSE.UNTIL " "
            - user: !PARSE.BETWEEN { what: '"' }
            - !PARSE.EXACTLY " "
            - !PARSE.UNTIL " "
            - !PARSE.UNTIL " "
            - !PARSE.UNTIL " "
            - ip: !PARSE.CHARS

Example 4: Parsing JSON events

For the purpose of the example, let's say that we want to parse a JSON event:

    {
      "data": {
        "action": "allow",
        "backendStatusCode": "200",
        "clientAddr": "89.183.114.162",
        "countryCode": "cz",
        "host": "www.praha.cz",
        "response": {
          "backendTime": "0.043",
          "code": "200",
        },
      },
      "time": "2024-03-03T01:15:03.480Z",
    }

We will create two parsers. First parser will parse the original event and store it as JSON object in json field.

10_parser.yaml

define:
    type: parser/json

Result event will be:

{
    "json": <JSON object>
}

Next parser will parse /time field from the JSON object.

20_parser_timestamp.yaml

    define:
        type: parser/parsec
        field: json /time

    parse:
      !PARSE.KVLIST
      - "@timestamp": !PARSE.DATETIME
          - year: !PARSE.DIGITS
          - '-'
          - month: !PARSE.MONTH "number"
          - '-'
          - day: !PARSE.DIGITS
          - 'T'
          - hour: !PARSE.DIGITS
          - ':'
          - minute: !PARSE.DIGITS
          - ':'
          - second: !PARSE.DIGITS
          - microsecond: !PARSE.FRAC
                         base: "micro"

Result event will be:

{
    "json": <JSON object>,
    "@timestamp": 140994182325993472,
}