Event Lane Profile¶
Event lane profiles define the data lifecycle for event lanes. A profile specifies how long data is stored in Archive, Kafka, and Elasticsearch in each phase, and when it is deleted.
Event lane profile assignment¶
When creating or modifying an event lane, you can assign a profile to it in the event lane declaration file located in the Library at /EventLanes/tenant/eventlane.yaml
.
---
define:
type: lmio/event-lane
profile: /Profiles/EventLanes/Profile.yaml # Assign the profile here
By default, if no profile is specified, the /Profiles/EventLanes/Default.yaml
profile is used.
Profile definition¶
Profiles are defined in the Library under /Profiles/EventLanes/
. Each profile is a YAML file that specifies the lifecycle policies for Archive streams, Kafka topics, and Elasticsearch indices associated with event lanes.
Every section can be overridden in the event lane declaration, allowing for customization of lifecycle policies on a per-event-lane basis.
The following sections can be defined in a profile:
archive
: Defines the lifecycle policies for Archive streams, including phases like hot, warm, and cold, and actions such as copy, move, compress, and delete. Read more about Archive lifecycle management.kafka
: Specifies the retention settings for Kafka topics, includingreceived
andevents
.elasticsearch
: Outlines the lifecycle policies for Elasticsearch indices, including phases like hot, warm, cold, and delete, along with actions such as rollover, shrink, set_priority, and delete. Read more about Elasticsearch lifecycle management.
Warning
Each section must be fully defined in the profile. Partial definitions are not supported. For example, if you define the elasticsearch
section in a profile, you must include all necessary phases and actions within that section.
---
define:
type: lmio/event-lane-profile
name: Default
archive:
lifecycle:
hot:
- copy:
phase: cold
- move:
age: 1d
phase: warm
warm:
- delete:
age: 3M
cold:
- compress:
preset: 6
threads: 4
type: xz
- delete:
age: 18M
kafka:
received:
settings:
retention: 1d
events:
settings:
retention: 1d
elasticsearch:
events:
lifecycle:
hot:
min_age: 0ms
actions:
rollover:
max_primary_shard_size: 16gb
max_age: 14d
set_priority:
priority: 100
warm:
min_age: 3d
actions:
shrink:
number_of_shards: 1
set_priority:
priority: 50
cold:
min_age: 14d
actions:
set_priority:
priority: 0
delete:
min_age: 180d
actions:
delete:
delete_searchable_snapshot: true