TeskaLabs LogMan.io Library

A library of declaration is a folder on the filesystem that holds declarations for parsers, enrichers, correlators and other YAML elements such as !INCLUDE files.

A library has a prescribed structure:

library/
    <parser group 1>/
        p01_<parser>.yaml
        p02_<parser>.yaml
        e01_<enricher>.yaml
        e02_<enricher>.yaml
        include/
            head_parser.yaml
            spec_parser.yaml
            ...
        test/
            test01.yaml
            ...
    <parser group 2>/
    <parser group 2>/

    <correlator group 1>/
    <correlator group 2>/

    ...
    include/

The parser group is a set of parser and enricher declarations that is operated within the same parser type.

Naming pattern

The naming pattern eg. p01_<...>.yaml is recommended because it provides the control over the order of execution and a visual differentiator between parsers and enrichers. The order of files being loaded to the pipeline is alphabetical, thus parser with a name p01_<...>.yaml will be loaded into the pipeline before the p02_<...>.yaml parser.

Including declarations in the library

Declarations such as declarations of parsers can include other declarations from library include directories using the !INCLUDE expression.

The include directories are specified in include_search_path configuration option for LogMan.io Parser, Correlator etc.:

[declarations]
include_search_path=filters;filters/firewall;filters/common;filters/authentication

By specifying asterisk * after a slash, all subdirectories will be recursively included, so that user does not have to specify each of them in the include_search_path option:

[declarations]
include_search_path=filters/*

By default, the following include search path are always also implicitly included:

library/<group>/include is the implicit location of the !INCLUDE YAML files used within a parser group.

library/include is the location of the !INCLUDE YAML files used globally.

Declaration named predicate_filter.yaml located in one of the include search path directories can then included in the following way:

predicate:
  !AND
  - !EQ
    - !ITEM EVENT Type
    - UseIt
  - !INCLUDE predicate_filter

For more information, see Cascade Parser and Window Correlator sections.

Unit tests

library/<* group/test is location of the unit test for the given group, see lmio-parser and lmio-correlator for more details about how to approach unit tests of the library.

The library is designed to be easily manageable by a version control systems such as Git or Subversion.