Migration to Parsec with event lanes¶

For migrating from LogMan.io Parser or LogMan.io Parsec version less than v24.14, which does not use event lanes, use the following migration guide.

Prerequisites¶

LogMan.io Depositor >v24.11-beta
LogMan.io Baseliner >v24.11-beta
LogMan.io Correlator >v24.11-beta

Migration steps¶

Create new event lane YAML file /EventLanes/<tenant>/<eventlane>.yaml in the Library.

Add the following properties to the eventlane:

/EventLanes/tenant/eventlane.yaml

---
define:
    type: lmio/event-lane

parsec:
    name: /Parsers/path/to/parser

kafka:
    received:
        topic: received.<tenant>.<stream>
    events:
        topic: events.<tenant>.<stream>
    others:
        topic: others.<tenant>

elasticsearch:
    events:
        index: lmio-<tenant>-events-<eventlane>
    others:
        index: lmio-<tenant>-others

(Replace <tenant>, <stream>, <eventlane> and /path/to/parser with the specific values.)

Create configuration for LogMan.io Parsec:

lmio-parsec.conf

[tenant]
name=<tenant>

[eventlane]
name=/EventLanes/<tenant>/<eventlane>.yaml

[library]
providers=
    zk:///library
    ...

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Use either the old Consumer group or create a new one. First, open kafdrop, search for the corresponding received topic, see Consumers and find Group ID of the old LMIO Parser/LMIO Parsec. Decide, whether to keep the old group.id or create a new one.

Warning

By creating new group.id, a new consumer group will be created and begin to read events from the start. (This depends on auto.offset.reset parameter of Kafka cluster, which is by default earliest.)

In case you want to keep the old group.id, add the following section to the configuration:
lmio-parsec.conf
```
[pipeline:ParsecPipeline:KafkaSource]
group_id=<your group id>
```
Otherwise, group.id will be automatically created based on the event lane name: lmio-parsec-<tenant>-<eventlane>.

Start the service. Ensure it is running by looking at its logs.

You should see no error logs. If so, see troubleshooting. You should also see notice logs similar to these:

NOTICE lmioparsec.app Event Lane /EventLanes/default/linux-syslog-rfc3164-10001.yaml loaded successfully.
NOTICE lmioparsec.app [sd timezone="Europe/Prague" charset="utf-8" schema="/Schemas/ECS.yaml" parser="/Parsers/Linux/Common"] Configuration loaded.
NOTICE lmioparsec.declaration_loader [sd parsers="3" mappings="1" enrichers="1"] Declarations loaded.
NOTICE lmioparsec.parser.pipeline [sd source_topic="received.default.linux-syslog-rfc3164-10001" events_topic="events.default.linux-syslog-rfc3164-10001" others_topic="others.default" group.id="custom-group-id"] ParsecPipeline is ready.

There you should see the correct Kafka topics, group.id, charset, schema and timezone.

Ensure the service is consuming from the right topic with the group id. Open once again kafdrop and find the received topic. Check whether either the new consumer group was created or whether the Combined Lag of the old group starts decreasing.
Check if new messages are incoming into Kafka events topic.

events topic is not created

Check for others topic. If new messages are coming there, the parsing rule is not correct. Check once again if you are using the proper parser name in event lane:

yaml "eventlane.yaml" parsec: name: /Parsers/path/to/parser

If so, then parsing rules are incorrect and should be changed.

Check if LogMan.io Depositor is running. Open events topic and check if Depositor is consuming from it (in Combined Lag).
Check if messages are visible in Discover screen on LogMan.io UI.

Messages are not visible at all

If you cannot find data on Discover screen, wait for some time (cca 1-2 minutes), the process might take some time. Then, check if the proper events topic exists and whether the event lane is properly configured. If so, check if messages are not incoming with incorrect timezone by setting time range to (-1 day, +1 day).

Messages are visible in incorrect timezone

If timezone or used schema is incorrect, you can overwrite it inside event lane:

```yaml title="/EventLanes/<tenant>/<eventlane>.yaml"
define:
    type: lmio/event-lane
    timezone: UTC
    schema: /Schemas/ECS.yaml
```