Depositor prerequisites¶
LogMan.io Depositor has the following dependencies:
- Elasticsearch
- Apache ZooKeeper
- Apache Kafka
- LogMan.io Library with an
/EventLanes
folder and a schema in the/Schemas
folder
Elasticsearch configuration¶
The Elasticsearch cluster needs to be configured in the following way in order for LogMan.io Depositor to work properly.
The following is a Docker Compose entry of Elasticsearch nodes, when using a 3-node cluster architecture with lm1
, lm2
, and lm3
server nodes.
Note
Please note that, in the Docker Compose file, the proper node roles are assigned to Elasticsearch nodes based on the ILM. For example, hot nodes for the ILM hot phase must contain node roles data_hot
and data_content
.
When creating Docker Compose records for Elasticsearch nodes, the following attributes must be changed:
- NODE_ID: The name of the server where the Elasticsearch instance is running
- INSTANCE_ID: The name of the Elasticsearch instance. Make sure its postfix
-1
is changed to-2
at the second instance of this service etc. INSTANCE_ID is thus a unique identifier for each of the instances. - network.host: The name of the server where the Elasticsearch instance is running
- node.attr.rack_id: The name of the server rack (for large deployments) or the name of the server where the Elasticsearch instance is running
- discovery.seed_hosts: The server host names and ports of all Elasticseach master nodes
- xpack.security.transport.ssl.certificate: The path to the certificate specific for the given Elasticsearch instance
- xpack.security.transport.ssl.key: The path to the certificate key specific for the given Elasticsearch instance
- volumes: The path to the given Elasticsearch instance's data
elasticsearch-master-1:
network_mode: host
user: "1000:1000"
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
environment:
- NODE_ID=lm1
- SERVICE_ID=elasticsearch
- INSTANCE_ID=elasticsearch-master-1
- network.host=lm1 #
- node.attr.rack_id=lm1 #
- node.name=elasticsearch-master-1
- node.roles=master,ingest
- cluster.name=lmio-es #
- cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3 #
- discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
- http.port=9200
- transport.port=9300 #
- "ES_JAVA_OPTS=-Xms4g -Xmx4g" #
- ELASTIC_PASSWORD=$ELASTIC_PASSWORD
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.certificate=certs/elasticsearch-master-1/elasticsearch-master-1.crt
- xpack.security.transport.ssl.key=certs/elasticsearch-master-1/elasticsearch-master-1.key
volumes:
- /data/ssd/elasticsearch/elasticsearch-master-1/data:/usr/share/elasticsearch/data
- ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
restart: always
elasticsearch-hot-1:
network_mode: host
user: "1000:1000"
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
depends_on:
- es-master
environment:
- NODE_ID=lm1
- SERVICE_ID=elasticsearch
- INSTANCE_ID=elasticsearch-hot-1
- network.host=lm1 #
- node.attr.rack_id=lm1 #
- node.attr.data=hot #
- node.name=elasticsearch-hot-1
- node.roles=data_hot,data_content #
- cluster.name=lmio-es #
- cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3 #
- discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
- http.port=9201
- transport.port=9301 #
- "ES_JAVA_OPTS=-Xms31g -Xmx31g" #
- ELASTIC_PASSWORD=$ELASTIC_PASSWORD
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.certificate=certs/elasticsearch-hot-1/elasticsearch-hot-1.crt
- xpack.security.transport.ssl.key=certs/elasticsearch-hot-1/elasticsearch-hot-1.key
volumes:
- /data/ssd/elasticsearch/elasticsearch-hot-1/data:/usr/share/elasticsearch/data
- ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
elasticsearch-warm-1:
network_mode: host
user: "1000:1000"
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
depends_on:
- es-master
environment:
- NODE_ID=lm1
- SERVICE_ID=elasticsearch
- INSTANCE_ID=elasticsearch-warm-1
- network.host=lm1 #
- node.attr.rack_id=lm1 #
- node.attr.data=warm #
- node.name=elasticsearch-warm-1
- node.roles=data_warm #
- cluster.name=lmio-es #
- cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3 #
- discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
- http.port=9202
- transport.port=9302 #
- "ES_JAVA_OPTS=-Xms31g -Xmx31g" #
- ELASTIC_PASSWORD=$ELASTIC_PASSWORD
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.certificate=certs/elasticsearch-warm-1/elasticsearch-warm-1.crt
- xpack.security.transport.ssl.key=certs/elasticsearch-warm-1/elasticsearch-warm-1.key
volumes:
- /data/hdd/elasticsearch/elasticsearch-warm-1/data:/usr/share/elasticsearch/data
- ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
elasticsearch-cold-1:
network_mode: host
user: "1000:1000"
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
depends_on:
- es-master
environment:
- NODE_ID=lm1
- SERVICE_ID=elasticsearch
- INSTANCE_ID=elasticsearch-cold-1
- network.host=lm1
- node.attr.rack_id=lm1 #
- node.attr.data=cold #
- node.name=elasticsearch-cold-1
- node.roles=data_cold #
- cluster.name=lmio-es #
- cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3 #
- discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
- http.port=9203
- transport.port=9303 #
- "ES_JAVA_OPTS=-Xms31g -Xmx31g" #
- ELASTIC_PASSWORD=$ELASTIC_PASSWORD
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.certificate=certs/elasticsearch-cold-1/elasticsearch-cold-1.crt
- xpack.security.transport.ssl.key=certs/elasticsearch-cold-1/elasticsearch-cold-1.key
volumes:
- /data/hdd/elasticsearch/elasticsearch-cold-1/data:/usr/share/elasticsearch/data
- ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
-
The node will bind to the public address and will also use it as its publish address.
Index templates¶
LogMan.io Depositor creates its own index templates with the events
index from the event lane's elasticsearch
configuration, adding the postfix -template
. All previous index templates, if present, must have a different name and their priority set to 0.