Skip to content

Declarations

Declarations describe how the event should be parsed. They are stored as YAML files in the Library. LogMan.io Parsec interprets these declarations and creates parsing processors.

There are three types of declarations:

  1. Parser declaration: A parser takes an original event or a specific field of a partially parsed event as input, analyzes its individual parts, and stores them as key-value pairs to the event.
  2. Mapping declaration: Mapping takes a partially parsed event as input, renames the field names, and eventually converts the data types. It works together with a schema (ECS, CEF).
  3. Enricher declaration: An enricher supplements a partially parsed event with extra data.

Data flow

A typical, recommended parsing sequence is a chain of declarations:

  1. The first main parser declaration begins the chain, and additional parsers (called sub-parsers) extract more detailed data from the fields created by the previous parser.
  2. Then, the (single) mapping declaration renames the keys of the parsed fields according to a schema and filters out fields that are not needed.
  3. Last, the enricher declaration supplements the event with additional data. While it's possible to use multiple enricher files, it's recommended to use just one.

Parsec pipeline flowchart

Important: Naming conventions

LogMan.io Parsec loads declarations alphabetically and creates the corresponding processors in the same order. Therefore, create the list of declaration files according to these rules:

  • Begin all declaration file names with a numbered prefix:

    10_parser.yaml, 20_parser_message.yaml, ..., 90_enricher.yaml.

    It is recommended to "leave some space" in your numbering for future declarations in case you want to add a new declaration between two existing ones (e.g., 25_new_parser.yaml).

  • Include the type of declaration in file names: 20_parser_message.yaml rather than 10_message.yaml.

  • Include the type of schema used in mapping file names: 40_mapping_ECS.yaml rather than 40_mapping.yaml.

Example:

/Parsers/MyParser/:
    - 10_parser.yaml
    - 20_parser_username.yaml
    - 30_parser_message.yaml
    - 40_mapping_ECS.yaml
    - 50_enricher_lookup.yaml
    - 60_enricher.yaml