Event Lanes

Relation to Depositor

TeskaLabs LogMan.io Depositor reads all event lanes from the library and creates Kafka to Elasticsearch pipelines based on kafka and elasticsearch sections.

Note

All deployed instances of TeskaLabs LogMan.io Depositor share the same Group ID within Kafka. This means that all depositors reading all event lanes will distribute the Kafka partitions among themselves and thus provide scalability natively.

Declaration

This is the minimalistic example of the event lane definition, located in /EventLanes folder in the library:

---
define:
    type: lmio/event-lane

kafka:
    events:
        topic: events-default

    others:
        topic: others-default

elasticsearch:
    events:
        index: lmio-default-events

    others:
        index: lmio-default-others

When the Depositor is started and the event lane is loaded, Depositor creates two pipelines, one for events and the other for others. The input is specified in the kafka section, while the output index alias is specified in elasticsearch section. ElasticSearch then automatically map the alias name to the proper index name ending with -0000 number.

Warning

Complex event lane needs a custom declaration. Unlike Dispatcher, Depositor does not natively read from events-complex Kafka topic.

Note

Depositor considers ALL event lane files regardless if they are disabled for the given tenant in the UI or not. Depositor is not tenant specific service.

Index Template

When depositor is started, and periodically every ten minutes, it creates index template in ElasticSearch for the given event lane. The mappings in the index template are based on default schema, which is /Schemas/ECS.yaml or other specified in the Depositor's configuration.

The default schema path can be overriden in the event lane by specifying schema attribute in the define section:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

It is also possible to specify number_of_shards and number_of_replicas in the settings section in elasticsearch:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        settings:
            number_of_shards: 6
            number_of_replicas: 1

The default number_of_shards is 6 and number_of_replicas is 1.

Note

Please consider changing the default settings and schema. It usually brings about issues such as non-matching detection rules for the given event lane that uses a different schema.

Warning

If the index template changed and there was already an index in Elasticsearch, the changes are taken into effect only after next index rollover.

Lifecycle Policy

When depositor is started, and periodically every ten minutes, it refreshes the Index Lifecycle Policy in Elasticsearch for the given event lane.

Default

The default lifecycle policy contains four phases: hot, warm, cold and delete

The default hot phase for the given index ends when primary shard size exceedes 16 GB or is older than 7 days.

The default warm phase for the given index starts either when hot ends, or after 7 days, and turns on shrinking.

The default cold phase for the given index starts after 14 days.

The delete phase deletes the index after 180 days.

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        lifecycle:
            hot:
                min_age: "0ms"
                actions:
                    rollover:
                        max_primary_shard_size: "16gb"
                        max_age: "7d"
                    set_priority:
                        priority: 100

            warm:
                min_age: "3d"
                actions:
                    shrink:
                        number_of_shards: 1
                    set_priority:
                        priority: 50

            cold:
                min_age: "14d"
                actions:
                    set_priority:
                        priority: 0

            delete:
                min_age: "180d"
                actions:
                    delete:
                        delete_searchable_snapshot: true

Custom

The default ILM can be changed, even though it is not recommended for most cases. You can do so by specifying the lifecycle section within the event lane's elasticsearch section:

---
define:
    type: lmio/event-lane
    schema: /Schemas/CEF.yaml

kafka:
    ...

elasticsearch:
    ...

    events:
        ...

        lifecycle:
            hot:
                min_age: "0ms"
                actions:
                    rollover:
                        max_primary_shard_size: "25gb"  # We want bigger primary shards than default
                        max_age: "7d"
                    set_priority:
                        priority: 100

            warm:
                min_age: "7d"
                actions:
                    shrink:
                        number_of_shards: 1
                    set_priority:
                        priority: 50

            cold:
                min_age: "14d"
                actions:
                    set_priority:
                        priority: 0

            # There is no delete phase

Index

When depositor is started, and periodically every ten minutes, it checks if the indices for the given aliases from events and others section within elasticsearch exists.

If not, it creates the index ending with -000001, while enabling it for writing and assigning the alias.

If yes, nothing is done.