LogMan.io Correlator Configuration

First it is needed to specify which library to load the declarations from, which can be either ZooKeeper or File.

Also, every running instance of the parser must know which groups to load from the libraries, see below:

# Declarations

[declarations]
library=zk://zookeeper:12181/lmio/library.lib ./data/declarations
groups=Firewall Common Authentication
include_search_path=filters;filters/firewall;filters/common;filters/authentication
timestamp=@timestamp

groups - names of groups to be used from the library separated by spaces; if the group is located within a folder's subfolder, use slash as a separator, f. e. correlators/Firewall

include_search_path - specifies folders to search for YAML files to be later used in !INCLUDE expression statement (such as !INCLUDE myFilterYAMLfromFiltersCommonSubfolder) in declarations, separated by ;. By specifying asterisk * after a slash in the path, all subdirectories will be recursively included. !INCLUDE expression expects file name without path and without extension as input. The behavior is similar to -I include attribute when building C/C++ code.

timestamp - field name of timestamp attribute for normalization check after an event trigger is processed

Next, it is needed to know which Kafka topics to use at the input and output (for event triggers). Kafka connection needs to be also configured to know which Kafka servers to connect to.

# Kafka connection

[connection:KafkaConnection]
bootstrap_servers=lm1:19092;lm2:29092;lm3:39092

[pipeline:CorrelatorsPipeline:KafkaSource]
topic=lmio-events
group_id=lmio_correlator_firewall

# Kafka sink for event trigger

[pipeline:OutputPipeline:KafkaSink]
topic=lmio-output

The last mandatory section specifies which Kafka topic to use for the information about changes in lookups (i. e. reference lists) and which ElasticSearch instance to load them from.

# Lookup persistent storage

[asab:storage]
type=elasticsearch
elasticsearch_url=http://elasticsearch:9200

# Update lookups pipelines

[pipeline:LookupChangeStreamPipeline:KafkaSource]
topic=lookups
group_id=lmio_correlator_firewall

[pipeline:LookupModificationPipeline:KafkaSink]
topic=lookups

Installation

Docker Compose

  lmio-correlator:
    image: docker.teskalabs.com/lmio/lmio-correlator
    volumes:
      - ./lmio-correlator:/data