Kafka retention policy settings

TeskaLabs LogMan.io ensures the settings of Kafka topics are held in certain boundaries.

• Event lane topics are automatically managed by LogMan.io Event Lane Manager. Their retention can be set globally.
• Retention of any Kafka topic can be changed manually in Kafka.

Event lane topics

LogMan.io Event Lane Manager automatically sets the retention policy for event lane topics received.<tenant>.<stream>, events.<tenant>.<stream> and others.<tenant>. It is possible to set the retention globally (for all topics) in configuration:

Configuration in modelConfiguration file

/Site/model.yaml

services:
  lmio-elman:
    asab:
      config:
        kafka:
          retention.ms_optimal: ...  # default: 3 days
          retention.ms_max: ...  # default: 7 days
          retention.ms_min: ...  # default: 1 day (24 hours)

lmio-elman.conf

[kafka]
retention.ms_optimal=...  ; default: 3 days
retention.ms_max=...  ; default: 7 days
retention.ms_min=...  ; default: 1 day (24 hours)

When a new event lane topic is created, it's retention is set to retention.ms_optimal. After that, retention of each event lane topic can then be changed manually, but it must higher than retention.ms_min and lower than retention.ms_max, otherwise it is automatically set back to retention.ms_optimal.

Sanity checks

Warning

The following lines are relevant for non-automated deployments of TeskaLabs LogMan.io without LMIO Event Lane Manager.

For installations where there is no LMIO Event Lane Manager running, there is a supporting mechanism that sets the retention for event lane topics.

LMIO Receiver checks all received.<tenant>.<stream> Kafka topics and ensures their retention is between configured minimum and maximum. The default minimum retention is 12 hours and default maximum is 7 days. If the retention is smaller, it is set to the configured minimum value. If the retention is bigger, it is set to configured maximum value.

You can change the configuration in the LMIO Receiver config. Use time in milliseconds:

[kafka:sanity]
min.retention.ms=...  # default: 12 hours
max.retention.ms=...  # default: 7 days

LMIO Depositor implements the same logic, using same configuration, affecting the topics events.<tenant>.<stream> and others.<tenant> and providing the same defaults.

Mind that the retention can be controlled based on both the age of the logs in the queue and size of the topic (in bytes). LogMan.io components do not control retention configuration based on index size.

Setting the retention in Kafka

Retention of each Kafka topic can be changed in Kafka.

Each Kafka Docker container is provided with Kafka Command-Line Interface Tools which can be accessed.

1. Enter the shell of Kafka Docker container:

   docker exec -it kafka-1 bash
   

2. Ensure that the topic exists and can be reached:

   /usr/bin/kafka-topics --bootstrap-server localhost:9092 --list | grep "<topic>"
   

   (Replace <topic> with your topic name.)

3. Set the retention for a single topic using the command:

   /usr/bin/kafka-configs --bootstrap-server localhost:9092 \
   --entity-type topics --entity-name "<topic>" \
   --alter --add-config retention.ms=86400000
   

   (Replace <topic> with your topic name and retention.ms with your value.)

Why milliseconds?

You may wonder why are we using milliseconds everywhere we configure Kafka retention. Why just not use hours or days?

In fact, there are configuration options for setting the retention using hours or days. But retention.ms option always has the highest priority over all other options. To prevent possible conflicts with other services, we decided to use milliseconds, which is also recommended by Confluent.