Depositor configuration
Configuration sample
This is the minimalistic example of the LogMan.io Depositor configuration:
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
[library]
providers=zk:///library
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
[elasticsearch]
url=http://es01:9200
Zookeeper
Specify locations of the Zookeeper server in the cluster.
[zookeeper]
servers=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
Hint
For non-production deployments, the use of a single Zookeeper server is possible.
Library
Specify the path(s) to the library to load declarations from.
[library]
providers=zk:///library
Hint
Since ECS.yaml schema in /Schemas is utilized by default, consider using LogMan.io Common Library.
Kafka
Specify bootstrap servers of the Kafka cluster.
[kafka]
bootstrap_servers=kafka-1:9092,kafka-2:9092,kafka-3:9092
Hint
For non-production deployments, the use of a single Kafka server is possible.
ElasticSearch
Specify URLs of ElasticSearch master nodes.
The ESConnection
section is used for setting advanced parameters of the connection (see below).
The elasticsearch
section is used for storing urls and authorization items.
The asab:storage
section is used to explicitely allow the Storage initialization.
[asab:storage]
type=elasticsearch
[connection:ESConnection]
precise_error_handling=true
bulk_out_max_size=1582912
output_queue_max_size=5
loader_per_url=1
cluster_status_throttle=red
cluster_status_unthrottle=green
active_shards_percent_throttle=50
retry_errors=unavailable_shards_exception
throttle_errors=circuit_breaking_exception
[elasticsearch]
url=http://es01:9201
username=MYUSERNAME
password=MYPASSWORD
Hint
The url should point to the Elasticsearch hot node on the same server the Depositor is deployed to.
ElasticSearch Connection Advanced Settings
precise_error_handling
Specifies ElasticSearch should return information about which events caused the issue together with the error.
bulk_out_max_size
Size of one bulk being sent to ElasticSearch in bytes.
Events are grouped into bulks to lower the number of requests sent to ElasticSearch.
output_queue_max_size
Maximum queue size for ElasticSearch bulks.
If the number is exceeded, the given pipeline is throttled.
loader_per_url
Number of tasks/loaders per URL. It basically specifies the number of requests which can be sent simultaneously to every URL specified in the url attribute.
cluster_status_throttle
The state the cluster must enter in order to stop/throttle the Depositor. Can be set to none
.
Default: red
Options: red
, yellow
, none
cluster_status_unthrottle
The state the cluster must enter in order to start again/unthrottle the Depositor, if throttled.
Default: green
Options: red
, yellow
, green
active_shards_percent_throttle
The number of active shards in percent which must be present in order to send events through Depositor.
The number should be set to 100 / (number of replicas + 1)
.
Default: 50
retry_errors
List of retryable errors separated by comma, which cause the retry of the event to be send again to the events index specified in the event lane.
Hint
Using this configuration option is discouraged for most cases.
throttle_errors
List of errors separated by comma that cause the throttle of Dispatcher until resolved.
Hint
Using this configuration option is discouraged for most cases.
Declarations
Optional section to specify where to load the declarations of event lanes from and which schema is going to be used by default (if it is not specified in the given event lane declaration).
[declarations]
path=/EventLanes/
schema=/Schemas/ECS.yaml
Hint
Make sure to change the schema, if you are using other than ECS as default in your deployment. Changing the path for event lanes is discouraged.