Testing of parsers

It is important to test parsers to verify their functionality with various inputs. LogMan.io offers tools for manual and automated testing of parsers.

LogMan.io Parse Utility

This utility is meant for manual execution of parsers from command-line. It is useful for testing, since it applies selected parser groups on the input and unprocessed events are stored in a dedicated file so that parser can be improved till this "unparsed" output is empty. It is designed for parsing a very large inputs.

The parse utility is a command-line program. It is started by following command:

python3 ./parse.py -i input-syslog.txt -u unparsed-syslog.txt ./example/syslog_rfc5424-parser

-i, --input-file specifies the file with input lines for parsing

-u, --unparsed-file specifies the file to store the unparsed events from the input in

and then follow the parsers group(s) from a library, where to load the declarative parsers from.

The following application runs the parsing on a given input file with records divided by new lines, such as:

Feb 5 10:50:01 192.168.1.1 %ASA-1-105043 test1
Feb 5 10:55:10 192.168.1.1 %ASA-1-105043 test2
Feb 10 8:25:00 192.168.A1.1 X %ASA-1-105044 test3

and produces a file with only unparsed events, which has the same structure:

Feb 10 8:25:00 192.168.A1.1 X %ASA-1-105044 test3

Parser Unit test

The LogMan.io parses provides the tool for unit test execution over the library of parser and enricher declarations.

To start:

python3 ./test.py ./example [--config ./config.json]

The tool seeks for tests in the library, loads them and then execute them in the order.

Format of unit tests

Unit test file has to be placed in test directory and the name of the file has to comply with test*.yaml template. One YAML test file can contain one or more YAML documents with a test specification.

---
input: |
  line 1
  line 2
  ...

groups:

  # This means that everything from input will be parsed
  unparsed: []

  parsed:
    - msg: line
      num: 1
    - msg: line
      num: 2