Receiver configuration

The receiver requires following dependencies:

• Apache ZooKeeper
• NGINX (for production deployments)
• Apache Kafka

Example

This is the minimalistic example of the LogMan.io receiver configuration:

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

[lifecycle]
hot=/data/ssd/receiver
warm=/data/hdd/receiver
cold=/data/nas/receiver

Zookeeper

Specify locations of the Zookeeper server in the cluster.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Hint

For non-production deployments, the use of a single Zookeeper server is possible.

Lifecycle

Each lifecycle phases needs to be specified, typically by specifying the filesystem paths.

[lifecycle]
hot=/data/ssd/receiver
warm=/data/hdd/receiver
cold=/data/nas/receiver

task_limit=20

A lifecycle phase name (i.e. 'hot', 'warm', 'cold') must not contain "_" character.

See the Lifecycle chapter for more details.

task_limit option specifies a maximum number of concurrently running lifecycle tasks on this node. 20 is a default value.

Warning

Do not change lifecycle paths after the receiver has already stored data. Changing the paths will not apply retrospectively, and the receiver will be unable to locate data for the lifecycle phase.

Web APIs

The receiver provides two Web APIs: public and private.

Public Web API

Public Web API is designed for the communication with collectors.

[web:public]
listen=3080

The default port of the public web API is tcp/3080.

This port is designed to serve as the NGINX upstream for connections from collectors. It is a recommended production setup.

Standalone Public Web API

You can operate lmio-receiver without NGINX in stand-alone non-production setup.

Warning

Don't use this mode for production deployments.
This is ideal for development and testing environments, because you don't need a NGINX.

This is the configuration example:

[web:public]
listen=3443 ssl:web:public

[ssl:web:public]
key=${THIS_DIR}/server-key.pem
cert=${THIS_DIR}/server-cert.pem
verify_mode=CERT_OPTIONAL

This is how to generate self-signed server certificate for above config:

$ openssl ecparam -genkey -name secp384r1 -out server-key.pem
$ openssl req -new -x509 -subj "/OU=LogMan.io Receiver" -key server-key.pem -out server-cert.pem -days 365

Private Web API

[web]
listen=0.0.0.0 8950

The default port of the private web API is tcp/8950.

Certificate Authority

The receiver automatically creates Certificate Authority used for a collector provisioning.

The CA artefacts are stored in the Zookeeper at /lmio/receiver/ca folder.

 ./lmio-receiver.py -c ./etc/lmio-receiver.conf 
29-Jun-2023 19:43:50.651408 NOTICE asab.application is ready.
29-Jun-2023 19:43:51.716978 NOTICE lmioreceiver.ca.service Certificate Authority created
...

The default CA configration:

[ca]
curve=secp384r1
auto_approve=no

auto_approve option automates the collector enrollment process, every received CSR is automatically approved when set to yes.

Websocket

The default configuration of the websocket (to collectors)

[websocket]
timeout=30
compress=yes
max_msg_size=4M

Apache Kafka

The connection to Apache Kafka can be configured:

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

If the configuration is not present, then events are not forwarded to Apache Kafka.

Archive

The archive is enabled by default. Set the archive option to no to disable archive functionality.

[general]
archive=yes

Metrics

The receiver produces own telemetry and also forwards the telemetry from collectors to the configured telemetry data storage, such as InfluxDB.

[asab:metrics]
...

Signing keys

The signing key is used to digitally sign raw log archives.

You can specify the EC curve to be used for signing keys. The default is prime256v1, also known as secp256r1.

[signing]
curve=prime256v1