Skip to content

Configuring new LogMan.io Parser instance

To create a new parser instance for a new data source (files, lookups, SysLog etc.) the following three steps must be taken:

  • Creation of a new Kafka topic to load the collected data to
  • Configuration of associated LogMan.io Parser instances in site-repository
  • Deployment

Creation of a new Kafka topic to load the collected data to

First, it is needed to create a new collected events topic.

Collected events topics are specific for every data source type and tenant (customer). The standard for naming such Kafka topics is as follows:

collected-<tenant>-<type>

where tenant is the lowercase tenant name and type is the data source type. Examples include:

collected-railway-syslog
collected-ministry-files
collected-johnandson-databases
collected-marksandmax-lookups

Collected topic for all tenants can have the following format:

collected-default-lookups

To create a new Kafka topic:

1.) Enter any Kafka container via docker exec -it o2czsec-central_kafka_1 bash

2.) Use the following command to create the topic using /usr/bin/kafka-topics:

/usr/bin/kafka-topics --zookeeper lm1:12181,lm2:22181,lm3:32181 --create --topic collected-company-type -partitions 6 --replication-factor 1

The number of partitions are dependant on the expected amount of data and number of instances of LogMan.io Parser. Since in most deployment there are three running servers in the cluster, it is recommended to use at least three partitions.

Configuration of associated LogMan.io Parser instances in site-repository

Enter the site repository with configurations for LogMan.io cluster. To learn more about the site repository, please refer to the Naming Standards in Reference section.

Then in every server folder (such as lm1, lm2, lm3) create the following entry in docker-compose.yml file:

  <tenant>-<type>-lmio-parser:
    restart: on-failure:3
    image: docker.teskalabs.com/lmio/lmio-parser
    network_mode: host
    depends_on:
      - kafka
      - elasticsearch-master
    volumes:
      - ./<tenant>-<type>/lmio-parser:/data
      - ../lookups:/lookups
      - /data/hdd/log/<tenant>-<type>/lmio-parser:/log
      - /var/run/docker.sock:/var/run/docker.sock

replace <tenant> with tenant/customer name (such as railway) and <type> with data type (such as lookups), examples include:

railway-lookups-lmio-parser
default-lookups-lmio-parser
hbbank-syslog-lmio-parser

When the Docker Compose entry is included in docker-compose.yml, follow these steps:

1.) In every server folder (lm1, lm2, lm3), create <tenant>-<type> folder

2.) In <tenant>-<type> folders, create lmio-parser folder

3.) In the created lmio-parser folders, create lmio-parser.conf file

4.) Modify the lmio-parser.conf and enter the following configuration:

[asab:docker]
name_prefix=<server_name>-
socket=/var/run/docker.sock

# Declarations

[declarations]
library=zk://lm1:12181,lm2:22181,lm3:32181/lmio/library.lib ./data/declarations
groups=<group>
raw_event=raw_event
count=count
tenant=tenant
timestamp=@timestamp

# API

[asab:web]
listen=0.0.0.0 0

[lmioparser:web]
listen=0.0.0.0 0

# Logging

[logging:file]
path=/log/log.log
backup_count=3
rotate_every=1d

# Kafka connection

[connection:KafkaConnection]
bootstrap_servers=lm1:19092,lm2:29092,lm3:39092

[pipeline:ParsersPipeline:KafkaSource]
topic=collected-<tenant>-<type>
group_id=lmio_parser_<tenant>_<type>

# Kafka sinks

[pipeline:EnrichersPipeline:KafkaSink]
topic=lmio-events

[pipeline:ParsersPipeline:KafkaSink]
topic=lmio-others

[pipeline:ErrorPipeline:KafkaSink]
topic=lmio-others

[asab:zookeeper]
servers=lm1:12181,lm2:22181,lm3:32181
path=/lmio/library.lib

[zookeeper]
urls=lm1:12181,lm2:22181,lm3:32181
servers=lm1:12181,lm2:22181,lm3:32181
path=/lmio/library.lib

# Lookup persistent storage

[asab:storage]  # this section is used by lookups
type=elasticsearch

[elasticsearch]
url=http://<server_name>:9200/
username=<secret_username>
password=<secret_password>

# Update lookups pipelines

[pipeline:LookupChangeStreamPipeline:KafkaSource]
topic=lmio-lookups
group_id=lmio_parser_<tenant>_<type>_<server_name>

[pipeline:LookupModificationPipeline:KafkaSink]
topic=lmio-lookups

# Metrics

[asab:metrics]
target=influxdb

[asab:metrics:influxdb]
url=http://lm4:8086/
db=db0
username=<secret_username>
password=<secret_password>

where replace every occurrence of:

<group> with parser declaration group loaded in ZooKeeper; for more information refer to Library in Reference section of this documentation

<server_name> with root server folder name such as lm1, lm2, lm3

<tenant> with your tenant name such as hbbank, default, railway etc.

<type> with your data source type such as lookups, syslog, files, databases etc.

<secret_username> and <secret_password> with ElasticSearch and InfluxDB technical account credentials, which can be seen in other configurations in the site repository

For more information about what each of the configuration section means, please refer to Configuration section in the side menu.

Deployment

To deploy the new parser, please:

1.) Go to each of the LogMan.io servers (lm1, lm2, lm3)

2.) Do git pull in the site repository folder, which should be located in /opt directory

3.) Run docker-compose up -d <tenant>-<type>-lmio-parser to start the LogMan.io Parser instance

4.) Deploy and configure SyslogNG, LogMan.io Ingestor etc. to send the collected data to collected-<tenant>-<type> Kafka topic

4.) See logs in /data/hdd/log/<tenant>-<type>/lmio-parser folder for any errors to debug

(replace <tenant> and <type> accordingly)

Notes

To create data stream for lookups, please use lookups as type and refer to Lookups section in the side menu to properly create the parsing declaration group.