Dispatcher migration to Depositor

The migration from LogMan.io Dispatcher to LogMan.io Depositor needs to be done event lane by event lane following the steps mentioned below.

Warning

Before starting the migration, it is necessary to follow the Prerequisites, mainly the proper configuration of node roles for Elasticsearch nodes in the cluster.

Migration steps

Select one event lane to be migrated and follow this guide:

1.) In Kibana, go to Stack Management, then Index Management. Click on Index Template and find the index template associated with the event lane being migrated, usually the name is in the format of lmio-tenant-events-eventlane-template. In Actions (three dots) on the right, click on Clone.

2.) In the Clone, set the priority to 0 and change the name to backup-lmio-tenant-events-eventlane-template

3.) Go to Review template and click on Create template button

4.) Check that the backup-lmio-tenant-events-eventlane-template template exists in Index Template

5.) Delete the original lmio-tenant-events-eventlane-template and keep only the backup from the previous point

6.) Go to LogMan.io UI, to Library section and /EventLanes folder

7.) If not existent, create the new event lane file with the name fortigate.yaml (replace fortigate with your event lane name) in the /EventLanes/tenant folder (replace tenant with the actual value). If the /EventLanes/tenant folder does not exist, create it in ZooKeeper UI.

8.) Create the kafka and elasticsearch sections for the given event lane with both events and others sections specified, see Event Lane. The default schema for field mapping is /Schemas/ECS.yaml, unless specified in the event lane.

9.) If not deployed, deploy LogMan.io Depositor with kafka, elasticsearch, zookeeper and library sections specified, see Configuration

10.) Check LogMan.io Depositor logs for warnings. Please check both Docker logs and file logs (if file logs are configured). The Docker logs can be accessed via the following command:

docker logs -f -n 1000 <lmio-depositor>

Replace <lmio-depositor> with LogMan.io Depositor Docker container name in your deployment.

11.) In Kibana, go to Stack Management, then Index Management, check that the new lmio-tenant-events-eventlane-template and lmio-tenant-others-template index templates were created by depositor. Click on the index template and check its settings and mappings. The default settings include 6 shards and 1 replica, see Event Lane

12.) In Kibana, go to Stack Management, then Index Lifecycle Policies and check if the lmio-tenant-events-eventlane-ilm and lmio-tenant-others-ilm were created. Click on their name to check the hot, warm, cold and delete phase settings.

13.) If not deployed or configured, deploy or configure LogMan.io Parsec to send data to the Kafka event topic specified in the event lane declaration (here: fortigate.yaml ). Please see Parsec Configuration section.

14.) In Kibana, go to Dev Tools and run index rollover:

POST /lmio-tenant-eventlane/_rollover

Of course, replace tenant and eventlane with your setting.

15.) Check that the new index written in the response in the right box on the screen was created. So go to Stack Management, then Index Management, Indices and find the index lmio-tenant-events-eventlane-0000x

16.) Click on the lmio-tenant-events-eventlane-0000x, check that it is connected to the proper lifecycle policy, which should be lmio-tenant-events-eventlane-ilm, also check that Current phase is hot. Then click on Settings and Mappings to check the number of shards (default is 6) and fields mapping that is loaded from the schema. The default schema is /Schemas/ECS.yaml, unless specified in the event lane.

17.) In Kibana, go to Discover and check that the data are coming to the given event lane.

18.) In LogMan.io UI, go to Discover and check that the data are coming to the given event lane.

19.) Repeat the steps 1.) - 18.) for all remaining event lanes (their events index). Only then you can finish the migration by doing the same procesure for others indices.

Hint

In the following days, periodically check that all indices are connected to the lifecycle policy (point 16.). Also, make sure the indices in hot phase are allocated to the hot Elasticsearch nodes, which can be seen in Kibana in Stack Monitoring -> Indices.

Note

When everything is fine after a week, the original backup index template backup-lmio-tenant-events-eventlane-template can be deleted.