Output of parsing¶

When you use LogMan.io Parsec to analyze logs, the result of this process is what we refer to as the "parsed event." This output is an essential aspect of log management, as it transforms raw log data into a structured format that is easier to understand, analyze, and act upon.

A parsed event is not just any collection of data; it is a meticulously structured output that presents the information in a flat list format. This means that each piece of information from the original log is extracted and presented as key-value pairs. These pairs are straightforward, making it easy to identify what each piece of data represents.

Key-Value Pairs¶

Key: This is a unique identifier that describes the type of information contained in the value. Keys are predefined labels that represent specific aspects of the log data, such as time stamps, error codes, user IDs, and so on. Keys are defined by the schema.
Value: This is the actual data or information associated with the key. Values can vary widely, from numerical codes and timestamps to textual descriptions or user inputs. The type of the value is defined in the schema.

The output event is typically serialized as JSON object.

Example of parsed event

{
    "@timestamp": 12345678901,
    "event.created": 12345678902,
    "event.ingested": 12345678903,
    "event.original": "<1> 2023-03-01 myhost myapp: Hello world!",
    "key1": "value1",
    "key2": "value2",
}

Common fields of parsed events¶

Warning

This chapter uses ECS schema!

From the parsec (implicit):

@timestamp: If timestamp is not parsed, this field is automatically created with the time of parsing.

From the collector:

event.original: The original event in its raw format.
event.created: The time when the event was collected by a LogMan.io Collector.
lmio.source: The name of the log source (created by LogMan.io Collector).

From the receiver:

event.ingested: The time when the event was ingested to LogMan.io Receiver.
tenant: The name of the LogMan.io tenant in which the Parsec processing that event, as specified in configuration.
_id: Unique identifier of the event.

Tags¶

Roadmap

There will be an option to add arbitrary tags to the event which will enable custom filtering.

At this time, the only tag that is automatically added to the tags field is the version of the LogMan.io Parsec.

Error events¶

When parsing fails or an unexpected error occurs, the event is sent to others event lane (ErrorPipeline), where it is enriched with the information about when and why it happened.

Every error event contains:

@timestamp: The time when the event was processed with failure in UNIX timestamp (number of seconds from epoch).
event.original: The original event in its raw format.
error.message: The error message.
error.stack_trace: The data about where in the code the exception happened.
event.dataset: The name of the dataset specified in mapping or the path for the parser in the Library.
event.created: The time when the event was created in LogMan.io Collector.
event.ingested: The time when the event was ingested to LogMan.io Receiver.
tenant: The name of the tenant this event was aimed for.