Depositor configuration¶

Configuration sample¶

This is the most basic configuration required for LogMan.io Depositor:

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

[library]
providers=zk:///library

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

[elasticsearch]
url=http://es01:9200

Zookeeper¶

Specify locations of the Zookeeper server in the cluster.

[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181

Hint

For non-production deployments, the use of a single Zookeeper server is possible.

Library¶

Specify the path(s) to the library to load declarations from.

[library]
providers=zk:///library

Hint

Since ECS.yaml schema in /Schemas is utilized by default, consider using the LogMan.io Common Library.

Kafka¶

Specify bootstrap servers of the Kafka cluster.

[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

Hint

For non-production deployments, the use of a single Kafka server is possible.

Elasticsearch¶

Specify URLs of Elasticsearch master nodes.

The ESConnection section is used for setting advanced parameters of the connection (see below).

The elasticsearch section is used for storing URLs and authorization items.

The asab:storage section is used to explicitly allow the storage initialization.

[asab:storage]
type=elasticsearch

[connection:ESConnection]
precise_error_handling=true
bulk_out_max_size=1582912
output_queue_max_size=5
loader_per_url=1
cluster_status_throttle=red
cluster_status_unthrottle=green
active_shards_percent_throttle=50
retry_errors=unavailable_shards_exception
throttle_errors=circuit_breaking_exception

[elasticsearch]
url=http://es01:9201
username=MYUSERNAME
password=MYPASSWORD

Hint

The URL should point to the Elasticsearch hot node on the same server the Depositor is deployed to.

Elasticsearch Connection Advanced Settings¶

precise_error_handling¶

Specifies Elasticsearch should return information about which events caused the issue together with the error.

bulk_out_max_size¶

Size of a single bulk being sent to Elasticsearch in bytes.

Events are grouped into bulks to lower the number of requests sent to Elasticsearch.

output_queue_max_size¶

Maximum queue size for Elasticsearch bulks.

If the number is exceeded, the given pipeline is throttled.

loader_per_url¶

Number of tasks/loaders per URL. It specifies the number of requests which can be sent simultaneously to every URL specified in the URL attribute.

cluster_status_throttle¶

The state the cluster must enter in order to stop/throttle the Depositor. Can be set to none.

Default: red

Options: red, yellow, none

cluster_status_unthrottle¶

The state the cluster must enter in order to resume/unthrottle the Depositor, if throttled.

Default: green

Options: red, yellow, green

active_shards_percent_throttle¶

The minimum percentage of total shards that must be active/available in order for Depositor to send events.

The value, which is a percentage, should be set to 100 / (number of replicas + 1).

Default: 50

retry_errors¶

List of retryable errors, separated by comma, which cause the retry of the event to be sent again to the events index specified in the event lane.

Note

This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.

throttle_errors¶

List of errors, separated by comma, that cause the throttle of Dispatcher until resolved.

Note

This configuration option is unnecessary in most cases, so it is recommended to exclude it from your configuration file.

Declarations¶

Optional section to specify where to load the declarations of event lanes from and which schema is going to be used by default (if it is not specified in the given event lane declaration).

[declarations]
path=/EventLanes/
schema=/Schemas/ECS.yaml

Hint

Make sure to change the schema if you are using a schema other than ECS in your deployment as your default. Changing the path for event lanes is discouraged.