Parser preprocessor¶
The parser preprocessor allows to preprocess the input event by a imperative code, e.g. Python, Cython, C etc.
Example¶
---
define:
name: Demo of the build-in Syslog preprocessor
type: parser/preprocessor
tenant: Syslog_RFC5424.STRUCTURED_DATA.soc@0.tenant # (optional)
count: CEF.cnt # (optional)
function: lmiopar.preprocessor.Syslog_RFC5424
tenant specifies the tenant attribute to be read and passed to context['tenant']
for further distribution of parsed and unparsed events to tenant specific
indices/storages in LogMan.io Dispatcher
count specifies the count attribute
with count of events to be read and passed to context['count']
Built-in preprocessors¶
lmiopar.preprocessor module contains following commonly used preprocessors.
There preprocessors are optimized for high performace deployments.
Syslog RFC5425 built-in preprocessor¶
function: lmiopar.preprocessor.Syslog_RFC5424
This is a preprocessor for the Syslog protocol (new) according to RFC5425.
The input for this preprocessor is a valid Syslog entry, e.g.:
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog 10 ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry.
The output is, a message part of the log in the event and parsed elements in the context.syslog_rfc5424.
event: An application event log entry.
context:
Syslog_RFC5424:
PRI: 165
FACILITY: 20
PRIORITY: 5
VERSION: 1
TIMESTAMP: 2003-10-11T22:14:15.003Z
HOSTNAME: mymachine.example.com
APP_NAME: evntslog
PROCID: 10
MSGID: ID47
STRUCTURED_DATA:
exampleSDID@32473:
iut: 3
eventSource: Application
eventID: 1011
...
Syslog RFC3164 built-in preprocessor¶
function: lmiopar.preprocessor.Syslog_RFC3164
This is a preprocessor for the BSD syslog Protocol (old) according to RFC3164.
The Syslog RFC3164 preprocessor can be configured in the define section:
define:
type: parser/preprocessor
year: 1999
timezone: Europe/Prague
year specifies the numeric representation of the year that will be applied to the timestamp of the logs.
Also, you may specify smart (default) for the advanced selection of the year based on the month.
timezone specifies the timezone of the logs, the default is UTC.
The input for this preprocessor is a valid Syslog entry, e.g.:
<34>Oct 11 22:14:15 mymachine su[10]: 'su root' failed for lonvick on /dev/pts/8
The output is, a message part of the log in the event and parsed elements in the context.syslog_rfc3164.
event: "'su root' failed for lonvick on /dev/pts/8"
context:
Syslog_RFC3164:
PRI: 34
PRIORITY: 2
FACILITY: 4
TIMESTAMP: '2003-10-11T22:14:15.003Z'
HOSTNAME: mymachine
TAG: su
PID: 10
TAG and PID are optional parameters.
CEF built-in preprocessor¶
function: lmiopar.preprocessor.CEF
This is a preprocessor for the CEF or Common Event Format.
define:
type: parser/preprocessor
year: 1999
timezone: Europe/Prague
year specifies the numeric representation of the year that will be applied to the timestamp of the logs.
Also, you may specify smart (default) for the advanced selection of the year based on the month.
timezone specifies the timezone of the logs, the default is UTC.
The input for this preprocessor is a valid CEF entry, e.g.:
CEF:0|Vendor|Product|Version|foobar:1:2|Failed password|Medium| eventId=1234 app=ssh categorySignificance=/Informational/Warning categoryBehavior=/Authentication/Verify
The output is, a message part of the log in the event and parsed elements in the context.CEF:
context:
CEF:
Version: 0
DeviceVendor: Vendor
DeviceProduct: Product
DeviceVersion: Version
DeviceEventClassID: 'foobar:1:2'
Name: Failed password
Severity: Medium
eventId: '1234'
app: ssh
categorySignificance: /Informational/Warning
categoryBehavior: /Authentication/Verify
CEF can contain also a Syslog header. This is supported by chaining relevant Syslog preprocessor with a CEF preprocessor. Please refer to a preprocessor chaining chapter for details.
Apache HTTP Server log formats built-in preprocessor¶
There are high performance preprocessors for common Apache HTTP server access logs.
function: lmiopar.preprocessor.Apache_Common_Log_Format
This is a preprocessor for the Apache Common Log Format.
function: lmiopar.preprocessor.Apache_Combined_Log_Format
This is a preprocessor for the Apache Combined Log Format.
Apache Common Log example¶
Input:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
Output:
context:
Apache_Access_Log:
HOST: '127.0.0.1'
IDENT: '-'
USERID: 'frank'
TIMESTAMP: '2000-10-10T20:55:36.000Z'
METHOD: 'GET'
RESOURCE: '/apache_pb.gif'
PROTOCOL: 'HTTP/1.0'
STATUS_CODE: 200
DOWNLOAD_SIZE: 2326
Apache Combined Log example¶
Input:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"
Output:
context:
Apache_Access_Log:
HOST: '127.0.0.1'
IDENT: '-'
USERID: 'frank'
TIMESTAMP: '2000-10-10T20:55:36.000Z'
METHOD: 'GET'
RESOURCE: '/apache_pb.gif'
PROTOCOL: 'HTTP/1.0'
STATUS_CODE: 200
DOWNLOAD_SIZE: 2326
REFERE': http://www.example.com/start.html
USER_AGENT: Mozilla/4.08 [en] (Win98; I ;Nav)
Microsoft ULS built-in preprocessor¶
function: lmiopar.preprocessor.Microsoft_ULS
This is a preprocessor for the Microsoft_ULS according to Microsoft Docs.
For Microsoft SharePoint ULS logs, that do not contain server name nor correlation fields, a dedicated preprocessor is provided:
function: lmiopar.preprocessor.Microsoft_ULS_Sharepoint
The Microsoft SharePoint ULS preprocessor can be configured in the define section:
define:
type: parser/preprocessor
year: 1999
timezone: Europe/Prague
year specifies the numeric representation of the year that will be applied to the timestamp of the logs.
Also, you may specify smart (default) for the advanced selection of the year based on the month.
timezone specifies the timezone of the logs, the default is UTC.
The input for this preprocessor is a valid Microsoft ULS Sharepoint entry, e.g.:
04/28/2021 12:31:57.69 mssdmn.exe (0x38E0) 0x4D10 SharePoint Server Search Connectors:SharePoint dvt6 High SetSTSErrorInfo ErrorMessage = Error from SharePoint site: WebExceptionStatus: SendFailure The underlying connection was closed: An unexpected error occurred on a send. hr = 90141214 [sts3util.cxx:6994] search\native\gather\protocols\sts3\sts3util.cxx 3aeca97a-a9db-4010-970e-fe01483bfd4f
The output is, a message part of the log in the event and parsed elements in the context.Microsoft_ULS.
event: Message included in the log.
context:
Microsoft_ULS:
TIMESTAMP 1619613117.69
PROCESS: mssdmn.exe (0x38E0)
THREAD: 0x4D10
PRODUCT: SharePoint Server Search
CATEGORY: Connectors:SharePoint
EVENTID: dvt6
LEVEL: High
Query String preprocessor¶
function: lmiopar.preprocessor.Query_String
This is a preprocessor for the query string (key=value&key=value...) such as meta information from LogMan.io Collector
Example of input:
file_name=log.log&search=true
The output is, a message part of the log in the event and parsed elements in the context.QUERY_STRING.
event: Message included in the log.
context:
QUERY_STRING:
file_name: log.log
search: true
JSON built-in preprocessor¶
function: lmiopar.preprocessor.JSON
This is a preprocessor for the JSON format. It expects the input in a binary or textual format, the output dictionary is placed in the event.
Hence, the input for this preprocessor is a valid JSON entry.
XML built-in preprocessor¶
function: lmiopar.preprocessor.XML
This is a preprocessor for the XML format. It expects the input in a binary or textual format, the output dictionary is placed in the event.
Hence, the input for this preprocessor is a valid XML entry, e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Schannel" Guid="{1f678132-5938-4686-9fdc-c8ff68f15c85}" />
<EventID>36884</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2020-06-26T07:12:01.331577900Z" />
<EventRecordID>30286</EventRecordID>
<Correlation ActivityID="{8e20742a-4b06-0002-c274-208e064bd601}" />
<Execution ProcessID="788" ThreadID="948" />
<Channel>System</Channel>
<Computer>XX</Computer>
<Security UserID="S-1-5-21-1627182167-2524376360-74743131-1001" />
</System>
<UserData>
<EventXML xmlns="LSA_NS">
<Name>localhost</Name>
</EventXML>
</UserData>
<RenderingInfo Culture="en-US">
<Message>The certificate received from the remote server does not contain the expected name. It is therefore not possible to determine whether we are connecting to the correct server. The server name we were expecting is localhost. The TLS connection request has failed. The attached data contains the server certificate.</Message>
<Level>Error</Level>
<Task />
<Opcode>Info</Opcode>
<Channel>System</Channel>
<Provider />
<Keywords />
</RenderingInfo>
</Event>
The output of the preprocessor in the event:
{
"System.EventID": "36884",
"System.Version": "0",
"System.Level": "2",
"System.Task": "0",
"System.Opcode": "0",
"System.Keywords": "0x8000000000000000",
"System.EventRecordID": "30286",
"System.Channel": "System",
"System.Computer": "XX",
"UserData.EventXML.Name": "localhost",
"RenderingInfo.Message": "The certificate received from the remote server does not contain the expected name. It is therefore not possible to determine whether we are connecting to the correct server. The server name we were expecting is localhost. The TLS connection request has failed. The attached data contains the server certificate.",
"RenderingInfo.Level": "Error",
"RenderingInfo.Opcode": "Info",
"RenderingInfo.Channel": "System"
}
CSV built-in preprocessor¶
function: lmiopar.preprocessor.CSV
This is a preprocessor for the CSV format. It expects the input in a binary or textual format, the output dictionary is placed in the event.
Hence, the input for this preprocessor is a valid CSV entry, e.g.:
user,last_name\njack,black\njohn,doe
The output of the preprocessor in the context["CSV"]:
{
"lines": [
{"user": "jack", "last_name": "black"},
{"user": "john", "last_name": "doe"}
]
}
Parameters¶
In define section of the CSV preprocessor,
the following parameters may be set for CSV reading:
delimiter: (default: ",")
escapechar: escape character
doublequote: allow doublequote (default: true)
lineterminator: line terminator character, either \n or \r (default is the operation system line separator)
quotechar: default quote character (default: "\"")
quoting: type of quoting
skipinitialspace: skip initial space (default: false)
strict: strict mode (default: false)
Custom preprocessors¶
A custom preprocessors can be called from the parser, the respective code has to be accessible by a parser microservice thru a common Python import way.
---
define:
name: Demo of the custom Python preprocessor
type: parser/preprocessor
function: mypreprocessors.preprocessor
mypreprocessors is a module respective a folder with __init__.py that contains a function preprocessor().
The parser specifies a function to call.
It uses Python notation and it will automatically import the module.
The signature of the function:
def preprocessor(context, event):
...
return event
Preprocessor may (1) modify the event (!EVENT) and/or (2) modify the context (!CONTEXT).
The output of the preprocessor function will be passed to a subsequent parsers.
Preprocessor parser doesn't produce parsed events directly.
If the function returns None, the parsing of the eveny is silently terminated.
If the funtion raises the exception, the exception will be logged and the event will be forwarded into unparsed output.
Chaining of preprocessors¶
Preprocessors can be chained in order to parse more complex input formats. The output (aka event) of the first preprocessor is fed as an input of the second preprocessor (and so on).
For example, the input is a CEF format with Syslog RFC3164 header:
<14>Jan 28 05:51:33 connector-test CEF_PARSED_LOG: CEF:0|Vendor|Product|Version|foobar:1:2|Failed password|Medium| eventId=1234 app=ssh categorySignificance=/Informational/Warning categoryBehavior=/Authentication/Verify
The pipeline contains two preprocessors:
p01_parser.yaml:
---
define:
name: Preprocessor for Syslog RFC5424 part of the message
type: parser/preprocessor
tenant: Syslog_RFC5424.STRUCTURED_DATA.soc@0.tenant
function: lmiopar.preprocessor.Syslog_RFC5424
p02_parser.yaml:
---
define:
name: Preprocessor for CEF part of the message
type: parser/preprocessor
function: lmiopar.preprocessor.CEF
and final parser p03_parser.yaml:
---
define:
name: Finalize by parsing the event into a dictionary
type: parser/cascade
parse:
!DICT
set:
Syslog_RFC5424: !ITEM CONTEXT Syslog_RFC5424
CEF: !ITEM CONTEXT CEF
Message: !EVENT
Output example:
context:
CEF:
Version: 0
DeviceVendor: Vendor
DeviceProduct: Product
DeviceVersion: Version
DeviceEventClassID: 'foobar:1:2'
Name: Failed password
Severity: Medium
eventId: '1234'
app: ssh
categorySignificance: /Informational/Warning
categoryBehavior: /Authentication/Verify
Syslog_RFC3164:
PRI: 14
FACILITY: 1
PRIORITY: 6
HOSTNAME: connector-test'
TAG: CEF_PARSED_LOG
TIMESTAMP': '2020-01-28T05:51:33.000Z'
Message: ''
Cisco ASA built-in preprocessor¶
function: lmiopar.preprocessor.CiscoASA
Warning
This preprocessor will be replaced by SP-Lang based parser.