Skip to content

Depositor configuration

Configuration sample

This is the minimalistic example of the LogMan.io Depositor configuration:

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

[library]
providers=zk:///library

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

[elasticsearch]
url=http://es01:9200

Zookeeper

Specify locations of the Zookeeper server in the cluster.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Hint

For non-production deployments, the use of a single Zookeeper server is possible.

Library

Specify the path(s) to the library to load declarations from.

[library]
providers=zk:///library

Hint

Since ECS.yaml schema in /Schemas is utilized by default, consider using LogMan.io Common Library.

Kafka

Specify bootstrap servers of the Kafka cluster.

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

Hint

For non-production deployments, the use of a single Kafka server is possible.

ElasticSearch

Specify URLs of ElasticSearch master nodes.

The ESConnection section is used for setting advanced parameters of the connection (see below).

The elasticsearch section is used for storing urls and authorization items.

The asab:storage section is used to explicitely allow the Storage initialization.

[asab:storage]
type=elasticsearch

[connection:ESConnection]
precise_error_handling=true
bulk_out_max_size=1582912
output_queue_max_size=5
loader_per_url=1
cluster_status_throttle=red
cluster_status_unthrottle=green
active_shards_percent_throttle=50
retry_errors=unavailable_shards_exception
throttle_errors=circuit_breaking_exception

[elasticsearch]
url=http://es01:9201
username=MYUSERNAME
password=MYPASSWORD

Hint

The url should point to the Elasticsearch hot node on the same server the Depositor is deployed to.

ElasticSearch Connection Advanced Settings

precise_error_handling

Specifies ElasticSearch should return information about which events caused the issue together with the error.

bulk_out_max_size

Size of one bulk being sent to ElasticSearch in bytes.

Events are grouped into bulks to lower the number of requests sent to ElasticSearch.

output_queue_max_size

Maximum queue size for ElasticSearch bulks.

If the number is exceeded, the given pipeline is throttled.

loader_per_url

Number of tasks/loaders per URL. It basically specifies the number of requests which can be sent simultaneously to every URL specified in the url attribute.

cluster_status_throttle

The state the cluster must enter in order to stop/throttle the Depositor. Can be set to none.

Default: red

Options: red, yellow, none

cluster_status_unthrottle

The state the cluster must enter in order to start again/unthrottle the Depositor, if throttled.

Default: green

Options: red, yellow, green

active_shards_percent_throttle

The number of active shards in percent which must be present in order to send events through Depositor.

The number should be set to 100 / (number of replicas + 1).

Default: 50

retry_errors

List of retryable errors separated by comma, which cause the retry of the event to be send again to the events index specified in the event lane.

Hint

Using this configuration option is discouraged for most cases.

throttle_errors

List of errors separated by comma that cause the throttle of Dispatcher until resolved.

Hint

Using this configuration option is discouraged for most cases.

Declarations

Optional section to specify where to load the declarations of event lanes from and which schema is going to be used by default (if it is not specified in the given event lane declaration).

[declarations]
path=/EventLanes/
schema=/Schemas/ECS.yaml

Hint

Make sure to change the schema, if you are using other than ECS as default in your deployment. Changing the path for event lanes is discouraged.