Testing of parsers
It is important to test parsers to verify their functionality with various inputs. LogMan.io offers tools for manual and automated testing of parsers.
LogMan.io Parse Utility
This utility is meant for manual execution of parsers from command-line. It is useful for testing, since it applies selected parser groups on the input and unprocessed events are stored in a dedicated file so that parser can be improved till this "unparsed" output is empty. It is designed for parsing a very large inputs.
The parse utility is a command-line program. It is started by following command:
python3 ./parse.py -i input-syslog.txt -u unparsed-syslog.txt ./example/syslog_rfc5424-parser
-i, --input-file specifies the file with input lines for parsing
-u, --unparsed-file specifies the file to store the unparsed events from the input in
and then follow the parsers group(s) from a library, where to load the declarative parsers from.
The following application runs the parsing on a given input file with records divided by new lines, such as:
Feb 5 10:50:01 192.168.1.1 %ASA-1-105043 test1 Feb 5 10:55:10 192.168.1.1 %ASA-1-105043 test2 Feb 10 8:25:00 192.168.A1.1 X %ASA-1-105044 test3
and produces a file with only
unparsed events, which has the same structure:
Feb 10 8:25:00 192.168.A1.1 X %ASA-1-105044 test3
Parser Unit test
The LogMan.io parses provides the tool for unit test execution over the library of parser and enricher declarations.
python3 ./test.py ./example [--config ./config.json]
The tool seeks for tests in the library, loads them and then execute them in the order.
Format of unit tests
Unit test file has to be placed in
test directory and the name of the file has to comply with
test*.yaml template. One YAML test file can contain one or more YAML documents with a test specification.
--- input: | line 1 line 2 ... groups: # This means that everything from input will be parsed unparsed:  parsed: - msg: line num: 1 - msg: line num: 2