Event Lane¶

When you connect a new log source in LogMan.io Collector provisioned to one tenant, events will start sending to LogMan.io Archive. LogMan.io Receiver will create a new stream for these events to store them in logical order. Therefore, every tenant owns multiple streams. Events from a particular stream can be (immediately or afterward) pulled from the Archive for parsing and storing in Elasticsearch database for further analysis.

To create a logical data stream in LogMan.io (from Archive through Kafka to Elasticsearch) and connect the adjacent content in Library(dashboards, reports, correlations, etc), the concept of event lane is used. The event lane is represented by a declaration in Library which determines:

what parsing rules will be applied for the data stream
what Library content is assigned to the data stream
categorization of the data stream (e.g. categorization of the technology that is producing such events)
what schema is used for the final structured data
what additional enrichment will be applied to the stream
configuration entries of the technologies operating with the data (Kafka, Elasticsearch)

The following image illustrates the process in the example, where the new log source (Fortinet FortiGate) is connected (under operating tenant mytenant):

LogMan.io Receiver collects the incoming events and stores the data in Archive in the stream fortinet-fortigate-10110 for tenant mytenant.
These events can be pulled back from the Archive for parsing. There are sent to Kafka topic received.mytenant.fortinet-fortigate-10110.
LogMan.io Elman detects this topic, assigns a new event lane and creates single or multiple instances of LogMan.io Parsec.
LogMan.io Parsec consumes raw events from that topic and sends the successfully parsed events and unparsed events to Kafka topics events.mytenant.fortinet-fortigate-10110 and others.mytenant, respectively.
LogMan.io Depositor consumes events from those topics and stores them in Elasticsearch indexes lmio-mytenant-events-fortinet-fortigate-10110 and lmio-mytenant-others.

Event Lane declaration¶

Event Lane declaration is specified in a YAML file in Library. The following example is event lane declaration for log source Microsoft 365 and tenant mytenant:

/EventLanes/mytenant/microsoft-365-v1.yaml

define:
  name: Microsoft 365
  type: lmio/event-lane
  timezone: UTC

parsec:
  name: /Parsers/Microsoft/365

content:
  reports: /Reports/Microsoft/365
  dashboards: /Dashboards/Microsoft/365

kafka:
  received:
    topic: received.mytenant.microsoft-365-v1
  events:
    topic: events.mytenant.microsoft-365-v1
  others:
    topic: others.mytenant

elasticsearch:
  events:
    index: lmio-mytenant-events-microsoft-365-v1
  others:
    index: lmio-mytenant-others

Define¶

The define part specifies the type of declaration and the properties of event lane used for parsing and analyzing the data, such as used schema, timezone and charset.

define:
  name: Microsoft 365
  type: lmio/event-lane
  schema: /Schemas/ECS.yaml  # (optional, default: /Schemas/ECS.yaml)
  timezone: Europe/Prague  # (optional, default is obtained from the tenant configuration)
  charset: utf-8  # (optional, default: utf-8)

Parsec¶

Section parsec refers to the microservice LogMan.io Parsec. In particular, parsec/name is the directory for parsing rules in Library. It has to always start with /Parsers/:

parsec:
  name: /Parsers/Microsoft/365

Content¶

Section content refers to the event lane content in the Library, such as dashboards, reports, correlations, etc.

When a new event lane is created, LogMan.io Elman automatically enables the content described in this section.

content:
  # Entire directory, described as a single string
  dashboards: /Dashboards/Microsoft/365

  # Multiple items described as a list
  reports:
  - /Reports/Microsoft/365/Daily Report.json
  - /Reports/Microsoft/365/Weekly Report.json
  - /Reports/Microsoft/365/Monthly Report.json

Kafka, Elasticsearch¶

Sections kafka and elasticsearch specify properties of Kafka topics and Elasticsearch indexes which belong to that eventlane. These are important for LogMan.io Parsec and LogMan.io Depositor.

The most important property is the name of received, events, and others topics and lmio-events and lmio-others indexes.

Kafka topics follow the naming convention:

<type>.<tenant>.<stream>

Elasticsearch indexes follow the naming convention:

lmio-<tenant>-<type>-<stream>

where:

type can be received, events or others
tenant is the name of the tenant
stream is the name of the log stream

kafka:
  received:
    topic: received.mytenant.microsoft-365-v1
  events:
    topic: events.mytenant.microsoft-365-v1
  others:
    topic: others.mytenant

elasticsearch:
  events:
    index: lmio-mytenant-events-microsoft-365-v1
  others:
    index: lmio-mytenant-others

Note

Every tenant has only one others topic, therefore, there is no specification of the stream in others topic and index.

Furthermore, additional properties of Elasticsearch (such as number of shards, index lifecycle etc) are configured in elasticsearch section. Read more in LogMan.io Depositor documentation.

Event Lane template declaration¶

For automatic assignment of parsing rules and Library content, event lane templates are used. When a new stream is found, LogMan.io Elman seeks for a suitable event lane template. When it is found, new event lane is automatically fulfilled with the properties of that template.

The following example illustrates the event lane template for log source Microsoft 365:

/Templates/EventLanes/Microsoft/microsoft-365-v1.yaml

---
define:
  type: lmio/event-lane-template
  name: Microsoft 365
  stream: microsoft-365-v1
  timezone: UTC

logsource:
  vendor: 
    - microsoft
  product:
    - m365
  service:
    - audit
    - activitylogs

parsec:
  name: /Parsers/Microsoft/365

content:
  dashboards: /Dashboards/Microsoft/365
  reports: /Reports/Microsoft/365

Define¶

define:
  type: lmio/event-lane-template
  name: Microsoft 365
  stream: microsoft-365-v1
  timezone: UTC

name: Human readable name for Event Lane, derived from the technology of the log source. It is used e.g. for configuration of Discover.
stream: The name that will be matched with the actual stream. There can be an exact match (such as in microsoft-365-v1), but wildcards (such as *) are allowed to match a wide range of streams (e.g. fortinet-fortigate-*).
timezone (optional): Various log sources send the events in a firmly established timezone (e.g. Microsoft 365 uses always UTC). To reflect that, timezone can be prescribed here. Otherwise, each event lane is handled manually.

Categorization¶

Section logsource is used for categorization of the log source connected to the event lane. It is derived from Sigma rules.

logsource:
  vendor: 
    - microsoft
  product:
    - m365
  service:
    - audit
    - activitylogs

Parsec¶

Option parsec/name is the directory for parsing rules in Library. It has to always start with /Parsers/:

parsec:
  name: /Parsers/Microsoft/365

Content¶

Section content refers to the event lane content in the Library, such as dashboards, reports, correlations, etc.

LogMan.io Elman automatically disables every content from all event lane templates. When a new event lane is created, its content is enabled.

content:
  # Entire directory, described as a single string
  dashboards: /Dashboards/Microsoft/365

  # Multiple items described as a list
  reports:
  - /Reports/Microsoft/365/Daily Report.json
  - /Reports/Microsoft/365/Weekly Report.json
  - /Reports/Microsoft/365/Monthly Report.json