Skip to content

LogMan.io Baseliner configuration

The LogMan.io Baseliner requires following dependencies:

  • Apache ZooKeeper
  • NGINX (for production deployments)
  • Apache Kafka
  • MongoDB
  • ElasticSearch
  • SeaCat Auth
  • LogMan.io Library with /Baselines folder and a schema in /Schemas folder

Example

This is the minimalistic example of the LogMan.io Baseliner configuration:

[declarations]
# The /Baselines is a default path
groups=/Baselines

[tenants]
ids=default

[pipeline:BaselinerPipeline:KafkaSource]
topic=^events.tenant.*

[pipeline:OutputPipeline:KafkaSink]
topic=complex.tenant

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

[library]
providers=zk://zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/lmio/library.lib

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

[elasticsearch]
url=http://es01:9200/

[mongodb.storage]
mongodb_uri=mongodb://mongodb1,mongodb2,mongodb3/?replicaSet=rs0
mongodb_database=baseliners

[auth]
multitenancy=yes
public_keys_url=http://localhost:8081/openidconnect/public_keys

Zookeeper

Specify locations of the Zookeeper server in the cluster.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Hint

For non-production deployments, the use of a single Zookeeper server is possible.

Library

Specify the path(s) to the library to load declarations from.

[library]
providers=zk://zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/lmio/library.lib

Hint

Since ECS.yaml schema in /Schemas is utilized by default, consider using LogMan.io Common Library.

Kafka

Specify bootstrap servers of the Kafka cluster.

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

Hint

For non-production deployments, the use of a single Kafka server is possible.

ElasticSearch

Specify URLs of ElasticSearch master nodes.

ElasticSearch is necessary for using lookups, f. e. as !LOOKUP expression or a lookup trigger.

[elasticsearch]
url=http://es01:9200
username=MYUSERNAME
password=MYPASSWORD

MongoDB

Specify URL of the MongoDB cluster with replica set.

MongoDB is used to store the baselines and counters of incoming events to.

[mongodb.storage]
mongodb_uri=mongodb://mongodb1,mongodb2,mongodb3/?replicaSet=rs0
mongodb_database=baseliners

Auth

The Auth section enables only users with access to the given tenant to get the baselines.

[auth]
multitenancy=yes
public_keys_url=http://localhost:8081/openidconnect/public_keys

Input

The events for the baselines are read from the Kafka topics:

[pipeline:BaselinerPipeline:KafkaSource]
topic=^events.tenant.*

Declarations (optional)

It is necessary to specify path for the declarations of baselines to be used. By default, /Baselines path and default fallback schema /Schemas/ECS.yaml is used.

Optionally, the default ECS schema path can be also changed.

[declarations]
groups=/Baselines
schema=/Schemas/ECS.yaml

Tenants

Specify the tenants to create the baselines for, separated by comma in ids.

The URL to load tenants from SeaCat Auth can be also specified.

[tenants]
ids=tenant1
tenant_url=http://localhost:8080/tenant

It is recommended to run a baseliner per tenant.

Output

If triggers are utilized, the default topic for output pipeline can be changed:

[pipeline:OutputPipeline:KafkaSink]
topic=complex.tenant

Web APIs

The Baseliner provides one web API.

The Web API is designed for the communication with UI.

[web]
listen=0.0.0.0 8999

The default port of the public web API is tcp/8999.

This port is designed to serve as the NGINX upstream for connections from collectors.