Skip to content

Dispatcher migration to Depositor

The migration from LogMan.io Dispatcher to LogMan.io Depositor needs to be done one event lane at a time following the steps mentioned below.

Warning

Before starting the migration, you must follow the Prerequisites, making sure to properly configure node roles for Elasticsearch nodes in the cluster.

Migration steps

Select one event lane to be migrated and follow this guide:

1. In Kibana, go to Management > Stack Management, then Index Management. Click on Index Templates and find the index template associated with the event lane being migrated. Usually, the name is in the format of lmio-tenant-events-eventlane-template. In the Actions column (three dots) on the right, click on Clone.

2. In Clone, Change the Name to backup-lmio-tenant-events-eventlane-template, and set the Priority to 0.

3. Go to Review template and click Create template.

4. Check that the backup-lmio-tenant-events-eventlane-template template exists in the Index Template tab.

5. Delete the original lmio-tenant-events-eventlane-template, and keep only the backup you just created.

6. Go to the LogMan.io UI, to the Library section and the /EventLanes folder

7. If the event lane file does not exist already, create the new event lane file with the name fortigate.yaml (replace "fortigate" with your event lane name) in the /EventLanes/tenant folder (replace "tenant" with the name of your tenant). If the /EventLanes/tenant folder does not exist already, you need to create it in ZooKeeper UI.

8. Create the kafka and elasticsearch sections for the given event lane with both events and others sections specified (see Event Lane). The default schema for field mapping is /Schemas/ECS.yaml, unless specified in the event lane.

9. If not deployed, deploy LogMan.io Depositor with kafka, elasticsearch, zookeeper, and library sections specified (see Configuration).

10. Check LogMan.io Depositor logs for warnings. Please check both Docker logs and file logs (if file logs are configured). The Docker logs can be accessed via the following command:

docker logs -f -n 1000 <lmio-depositor>

Replace <lmio-depositor> with the LogMan.io Depositor Docker container name in your deployment.

11. In Kibana, go to Management > Stack Management, then Index Management, and check that the new lmio-tenant-events-eventlane-template and lmio-tenant-others-template index templates were created by Depositor. Click on the index template and check its settings and mappings. The default settings include 6 shards and 1 replica (see Event Lane).

12. In Kibana, go to Management > Stack Management, then Index Lifecycle Policies and check if lmio-tenant-events-eventlane-ilm and lmio-tenant-others-ilm were created. Click on their name to check the hot, warm, cold, and delete phase settings.

13. If LogMan.io is not deployed or configured for this purpose already, deploy or configure Parsec to send data to the Kafka event topic specified in the event lane declaration (here: fortigate.yaml ). Please see the Parsec Configuration section.

14. In Kibana, go to Management > Dev Tools and run index rollover, replacing tenant and eventlane with the name of your tenant and your event lane:

POST /lmio-tenant-eventlane/_rollover

15. Check that the new index written in the response in the box on the right side of the screen was created. Go to Management > Stack Management, then Index Management, to the tab Indices and find the index lmio-tenant-events-eventlane-0000x.

16. Click on lmio-tenant-events-eventlane-0000x, check that it is connected to the proper lifecycle policy, which should be lmio-tenant-events-eventlane-ilm, and also check that Current phase is hot. Then, click on Settings and Mappings to check the number of shards (default is 6) and fields mapping that is loaded from the schema. The default schema is /Schemas/ECS.yaml, unless specified in the event lane.

17. In Kibana, go to Analysis > Discover and check that the data is coming to the given event lane.

18. In LogMan.io UI, go to Discover and check that the data is coming to the given event lane.

19. Repeat steps 1 to 18 for all remaining event lanes (their events index). Only then you can finish the migration by doing the same procedure for the others indices.

Hint

In the following days, periodically check that all indices are connected to the lifecycle policy (step 16). Also, make sure the indices in hot phase are allocated to the hot Elasticsearch nodes, which can be seen in Kibana in Management > Stack Monitoring > Indices.

Note

When you can confirm that everything is working properly after a week, you can delete the original backup index template backup-lmio-tenant-events-eventlane-template.