Skip to content

Kafka

The Apache Kafka serves as a queue to temporarily store events among the LogMan.io microservices. For more information, see Architecture

Kafka within LogMan.io

Topic naming in event lanes

Each event lane has received, events and others topics specified.

Each topic name contains the name of the tenant and the event lane's stream in the following manner:

received.tenant.stream
events.tenant.stream
others.tenant

received.tenant.stream

The received topic stores the incoming logs for the incoming tenant and event lane's stream.

events.tenant.stream

The events topic stores the parsed events for the given event lane defined by tenant and stream.

others.tenant

The others topic stores the unparsed events for the given tenant.

Internal topics

There are the following internal topics for LogMan.io:

lmio-alerts

This topic stores the triggered alerts and is read by LogMan.io Alerts microservice.

lmio-notifications

This topic stores the triggered notifications and is read by ASAB IRIS microservice.

lmio-lookups

This topic stores the requested changes in lookups and is read by LogMan.io Watcher microservice.

There are three instances of Apache Kafka, one on each node.

The number of partitions for each topic must be at least the same as the number of consumers (3) and divisible by 2, hence the recommended number for partitions is always 6.

The recommended replica count is 1.

Each topic must have a reasonable retention set based on the available size of SSD drives.

In the LogMan.io cluster environment, where average EPS is above 1000 events per second and SSD disk space is below 2 TB, the retention is usually 1 day (86400000 milliseconds). See the Commands section.

Hint

When the EPS is lower or there is more SSD space, it is recommended to set the retention for Kafka topics to higher values like 2 or more days in order to give to the administrators more time to solve potential issues.

To create the partitions, replicas and retention properly, see the Commands section.