Event Lanes¶

Relation to LogMan.io Depositor¶

TeskaLabs LogMan.io Depositor reads all event lanes from the library and creates Kafka-to-Elasticsearch pipelines based on kafka and elasticsearch sections.

Note

All deployed instances of TeskaLabs LogMan.io Depositor share the same Group ID within Kafka. This means that all depositors reading all event lanes will distribute the Kafka partitions among themselves and thus provide scalability natively.

Declaration¶

This example is the most basic event lane definition possible, located in the /EventLanes folder in the library:

---
define:
    type: lmio/event-lane

kafka:
    events:
        topic: events-default

    others:
        topic: others-default

elasticsearch:
    events:
        index: lmio-default-events

    others:
        index: lmio-default-others

When Depositor is started and the event lane is loaded, Depositor creates two pipelines, one for events and the other for others. The input is specified in the kafka section, while the output index alias is specified in elasticsearch section. Elasticsearch then automatically maps the alias name to the proper index name ending with -0000 number.

Warning

Complex events lane need custom declarations. Unlike Depositor's predecessor Dispatcher, Depositor does not natively read from the events-complex Kafka topic.

Note

Depositor considers ALL event lane files regardless of if they are disabled for the given tenant in the UI or not. Depositor is not a tenant-specific service.

Index template¶

When Depositor is started, and then periodically every ten minutes, it creates index template in Elasticsearch for the given event lane. The mappings in the index template are based on the default schema, which is /Schemas/ECS.yaml or another schema specified in the Depositor's configuration.

The default schema path can be overriden in the event lane by specifying the schema attribute in the define section:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

It is also possible to specify number_of_shards and number_of_replicas in the settings section in elasticsearch:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        settings:
            number_of_shards: 6
            number_of_replicas: 1

The default number_of_shards is 6 and number_of_replicas is 1.

Note

Please consider carefully before changing the default settings and schema. Changing the defaults usually causes issues such as non-matching detection rules for the given event lane that uses a different schema.

Warning

Changes to the index template will only take effect after the next index rollover if an index already exists in Elasticsearch.

Lifecycle Policy¶

When Depositor is started, and then periodically every ten minutes, it refreshes the Index Lifecycle Policy in Elasticsearch for the given event lane.

Default¶

The default lifecycle policy contains four phases: hot, warm, cold, and delete.

The default hot phase for the given index ends when primary shard size exceedes 16 GB or is older than 7 days.

The default warm phase for the given index starts either when hot ends, or after 7 days, and turns on shrinking.

The default cold phase for the given index starts after 14 days.

The delete phase deletes the index after 180 days.

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        lifecycle:
            hot:
                min_age: "0ms"
                actions:
                    rollover:
                        max_primary_shard_size: "16gb"
                        max_age: "7d"
                    set_priority:
                        priority: 100

            warm:
                min_age: "3d"
                actions:
                    shrink:
                        number_of_shards: 1
                    set_priority:
                        priority: 50

            cold:
                min_age: "14d"
                actions:
                    set_priority:
                        priority: 0

            delete:
                min_age: "180d"
                actions:
                    delete:
                        delete_searchable_snapshot: true

Custom¶

The default ILM can be changed, even though it is not recommended for most cases. You can do so by specifying the lifecycle section within the event lane's elasticsearch section:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        lifecycle:
            hot:
                min_age: "0ms"
                actions:
                    rollover:
                        max_primary_shard_size: "25gb"  # We want bigger primary shards than default
                        max_age: "7d"
                    set_priority:
                        priority: 100

            warm:
                min_age: "7d"
                actions:
                    shrink:
                        number_of_shards: 1
                    set_priority:
                        priority: 50

            cold:
                min_age: "14d"
                actions:
                    set_priority:
                        priority: 0

            # There is no delete phase

Index¶

When Depositor is started, and periodically every ten minutes, Depositor checks if the indices for the given aliases from events and others sections within elasticsearch exist.

If these indices are absent, Depositor creates the new index ending with -000001, enabling it to write and assign the alias.

If the indices already exist, Depositor takes no action.