LogMan.io Correlator Configuration

First it is needed to specify which library to load the declarations from, which can be either ZooKeeper or File.

[library]
providers=zk://library

ZooKeeper Library layer requires zookeeper configuration section.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Also, every running instance of the parser must know which groups to load from the libraries and which tenant it belongs to, see below:

# Tenant
[tenant]
ids=mytenant

# Declarations

[declarations]
groups=Firewall Common Authentication

# Complex event lane (optional)
[eventlane]
path=/EventLanes/mytenant/complex.yaml

groups - names of groups to be used from the library separated by spaces; if the group is located within a folder's subfolder, use slash as a separator, f. e. /Correlators/Firewall

Next, it is needed to know which Kafka topics to use at the default fallback input and output (unless specified in the correlations in logsources section and complex event lane).

Kafka connection needs to be also configured to know which Kafka servers to connect to.

# Kafka connection

[kafka]
bootstrap_servers=lm1:19092;lm2:29092;lm3:39092

# The default Kafka topic to read from when no logsource is specified in the correlation rule (optional)

[pipeline:CorrelatorsPipeline:KafkaSource]
topic=lmio-events
group_id=lmio_correlator_firewall

# The default kafka topic for event trigger unless there is a complex event specified (optional)

[pipeline:OutputPipeline:KafkaSink]
topic=lmio-output

The last mandatory section specifies which Elasticsearch setting that allow to work with Lookups. For more information, see Lookups section.

# Lookup persistent storage

[elasticsearch]
url=http://elasticsearch:9200

Installation

Docker Compose

    lmio-correlator:
        image: docker.teskalabs.com/lmio/lmio-correlator:VERSION
        volumes:
            - ./lmio-correlator:/conf
            - /data/ssd/lookups:/lookups
            - /data/hdd/log/lmio-correlator:/log
            - /data/ssd/correlators/lmio-correlator:/data

Replace lmio-correlator with the name of the correlator's instance.

The correlator needs to know its configuration path, path to lookups (the folder can be empty, depens if the lookups are used), logging path and the path to store its data.

Warning

The data path is mandatory and must be located on the fast drive, that is SSD.