Depositor prerequisites

The LogMan.io Depositor requires following dependencies:

Elasticsearch
Apache ZooKeeper
Apache Kafka
LogMan.io Library with /EventLanes folder and a schema in /Schemas folder

Elasticsearch

The Elasticsearch cluster needs to be configured in the following in order for LogMan.io Depositor to work properly.

The following is a Docker Compose entry of Elasticsearch nodes, when using 3 nodes cluster architecture with lm1, lm2 and lm3 server nodes.

Note

Please note that, in Docker Compose file, the proper node roles are assigned to Elasticsearch nodes based on the ILM. For example, hot nodes for ILM hot phase must contain node roles data_hot,data_content.

When creating Docker Compose records for Elasticsearch nodes, the following attributes must be changed:

NODE_ID: The name of the server where the Elasticsearch instance is running at
INSTANCE_ID: The name of the Elasticsearch instance, make sure its postfix -1 is changed to -2 at the second instance of this service etc. INSTANCE_ID is thus a unique identifier for each of the instances.
network.host: The name of the server where the Elasticsearch instance is running at
node.attr.rack_id: The name of the server rack (for large deployments) or the name of the server where the Elasticsearch instance is running at
discovery.seed_hosts: The server host names and ports of all Elasticseach master nodes
xpack.security.transport.ssl.certificate: The path to the certificate specific for the given Elasticsearch instance
xpack.security.transport.ssl.key: The path to the certificate key specific for the given Elasticsearch instance
volumes: The path to the given Elasticsearch instance's data

docker-compose.yaml: elasticsearch-master-1

  elasticsearch-master-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-master-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.name=elasticsearch-master-1
      - node.roles=master,ingest
      - cluster.name=lmio-es  # (3)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (6)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9200
      - transport.port=9300  # (4)
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"  # (5)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-master-1/elasticsearch-master-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-master-1/elasticsearch-master-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-master-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs   
    restart: always

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in the LogMan.io.
Ports for internal communication among nodes.
Memory allocated by this Elasticsearch instance. 31 GB is maximum recommended value and the server node must have the overall memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-hot-1

  elasticsearch-hot-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-hot-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.attr.data=hot  # (3)
      - node.name=elasticsearch-hot-1
      - node.roles=data_hot,data_content  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9201
      - transport.port=9301  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-hot-1/elasticsearch-hot-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-hot-1/elasticsearch-hot-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-hot-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of back compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in the LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is maximum recommended value and the server node must have the overall memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-warm-1

  elasticsearch-warm-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-warm-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.attr.data=warm  # (3)
      - node.name=elasticsearch-warm-1
      - node.roles=data_warm  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9202
      - transport.port=9302  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-warm-1/elasticsearch-warm-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-warm-1/elasticsearch-warm-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-warm-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of back compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in the LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is maximum recommended value and the server node must have the overall memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-cold-1

  elasticsearch-cold-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-cold-1
      - network.host=lm1
      - node.attr.rack_id=lm1  # (2) 
      - node.attr.data=cold  # (3)
      - node.name=elasticsearch-cold-1
      - node.roles=data_cold  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9203
      - transport.port=9303  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-cold-1/elasticsearch-cold-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-cold-1/elasticsearch-cold-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-cold-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of back compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in the LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is maximum recommended value and the server node must have the overall memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

Index templates

LogMan.io Depositor creates its own index templates with the events index from the event lane's elasticsearch configuration, adding the postfix -template. All previous index templates, if present, must have a different name and their priority set to 0.