LogMan.io Baseliner configuration
The LogMan.io Baseliner requires following dependencies:
- Apache ZooKeeper
- NGINX (for production deployments)
- Apache Kafka
- MongoDB
- ElasticSearch
- SeaCat Auth
- LogMan.io Library with
/Baselines
folder and a schema in/Schemas
folder
Example
This is the minimalistic example of the LogMan.io Baseliner configuration:
[declarations]
# The /Baselines is a default path
groups=/Baselines
[tenants]
ids=default
[pipeline:BaselinerPipeline:KafkaSource]
topic=^events.tenant.*
[pipeline:OutputPipeline:KafkaSink]
topic=complex.tenant
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
[library]
providers=zk://zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/lmio/library.lib
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
[elasticsearch]
url=http://es01:9200/
[mongodb.storage]
mongodb_uri=mongodb://mongodb1,mongodb2,mongodb3/?replicaSet=rs0
mongodb_database=baseliners
[auth]
multitenancy=yes
public_keys_url=http://localhost:8081/openidconnect/public_keys
Zookeeper
Specify locations of the Zookeeper server in the cluster.
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
Hint
For non-production deployments, the use of a single Zookeeper server is possible.
Library
Specify the path(s) to the library to load declarations from.
[library]
providers=zk://zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181/lmio/library.lib
Hint
Since ECS.yaml schema in /Schemas is utilized by default, consider using LogMan.io Common Library.
Kafka
Specify bootstrap servers of the Kafka cluster.
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
Hint
For non-production deployments, the use of a single Kafka server is possible.
ElasticSearch
Specify URLs of ElasticSearch master nodes.
ElasticSearch is necessary for using lookups, f. e. as !LOOKUP
expression or a lookup trigger.
[elasticsearch]
url=http://es01:9200
username=MYUSERNAME
password=MYPASSWORD
MongoDB
Specify URL of the MongoDB cluster with replica set.
MongoDB is used to store the baselines and counters of incoming events to.
[mongodb.storage]
mongodb_uri=mongodb://mongodb1,mongodb2,mongodb3/?replicaSet=rs0
mongodb_database=baseliners
Auth
The Auth section enables only users with access to the given tenant to get the baselines.
[auth]
multitenancy=yes
public_keys_url=http://localhost:8081/openidconnect/public_keys
Input
The events for the baselines are read from the Kafka topics:
[pipeline:BaselinerPipeline:KafkaSource]
topic=^events.tenant.*
Declarations (optional)
It is necessary to specify path for the declarations of baselines to be used. By default, /Baselines
path and default fallback schema /Schemas/ECS.yaml
is used.
Optionally, the default ECS schema path can be also changed.
[declarations]
groups=/Baselines
schema=/Schemas/ECS.yaml
Tenants
Specify the tenants to create the baselines for, separated by comma in ids.
The URL to load tenants from SeaCat Auth can be also specified.
[tenants]
ids=tenant1
tenant_url=http://localhost:8080/tenant
It is recommended to run a baseliner per tenant.
Output
If triggers are utilized, the default topic for output pipeline can be changed:
[pipeline:OutputPipeline:KafkaSink]
topic=complex.tenant
Web APIs
The Baseliner provides one web API.
The Web API is designed for the communication with UI.
[web]
listen=0.0.0.0 8999
The default port of the public web API is tcp/8999.
This port is designed to serve as the NGINX upstream for connections from collectors.