Depositor prerequisites¶

LogMan.io Depositor has the following dependencies:

Elasticsearch
Apache ZooKeeper
Apache Kafka
LogMan.io Library with an /EventLanes folder and a schema in the /Schemas folder

Elasticsearch configuration¶

The Elasticsearch cluster needs to be configured in the following way in order for LogMan.io Depositor to work properly.

The following is a Docker Compose entry of Elasticsearch nodes, when using a 3-node cluster architecture with lm1, lm2, and lm3 server nodes.

Note

Please note that, in the Docker Compose file, the proper node roles are assigned to Elasticsearch nodes based on the ILM. For example, hot nodes for the ILM hot phase must contain node roles data_hot and data_content.

When creating Docker Compose records for Elasticsearch nodes, the following attributes must be changed:

NODE_ID: The name of the server where the Elasticsearch instance is running
INSTANCE_ID: The name of the Elasticsearch instance. Make sure its postfix -1 is changed to -2 at the second instance of this service etc. INSTANCE_ID is thus a unique identifier for each of the instances.
network.host: The name of the server where the Elasticsearch instance is running
node.attr.rack_id: The name of the server rack (for large deployments) or the name of the server where the Elasticsearch instance is running
discovery.seed_hosts: The server host names and ports of all Elasticseach master nodes
xpack.security.transport.ssl.certificate: The path to the certificate specific for the given Elasticsearch instance
xpack.security.transport.ssl.key: The path to the certificate key specific for the given Elasticsearch instance
volumes: The path to the given Elasticsearch instance's data

docker-compose.yaml: elasticsearch-master-1

  elasticsearch-master-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-master-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.name=elasticsearch-master-1
      - node.roles=master,ingest
      - cluster.name=lmio-es  # (3)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (6)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9200
      - transport.port=9300  # (4)
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"  # (5)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-master-1/elasticsearch-master-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-master-1/elasticsearch-master-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-master-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs   
    restart: always

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations, a hostname is fine.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in LogMan.io.
Ports for internal communication among nodes.
Memory allocated by this Elasticsearch instance. 31 GB is the maximum recommended value, and the server node must have adequate memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes' names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-hot-1

  elasticsearch-hot-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-hot-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.attr.data=hot  # (3)
      - node.name=elasticsearch-hot-1
      - node.roles=data_hot,data_content  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9201
      - transport.port=9301  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-hot-1/elasticsearch-hot-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-hot-1/elasticsearch-hot-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-hot-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of backward compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is the maximum recommended value and the server node must have adequate memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-warm-1

  elasticsearch-warm-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-warm-1
      - network.host=lm1  # (1)
      - node.attr.rack_id=lm1  # (2)
      - node.attr.data=warm  # (3)
      - node.name=elasticsearch-warm-1
      - node.roles=data_warm  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9202
      - transport.port=9302  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-warm-1/elasticsearch-warm-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-warm-1/elasticsearch-warm-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-warm-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of backward compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is the maximum recommended value and the server node must have adequate memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

docker-compose.yaml: elasticsearch-cold-1

  elasticsearch-cold-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-cold-1
      - network.host=lm1
      - node.attr.rack_id=lm1  # (2) 
      - node.attr.data=cold  # (3)
      - node.name=elasticsearch-cold-1
      - node.roles=data_cold  # (6)
      - cluster.name=lmio-es  # (4)
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # (8)
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9203
      - transport.port=9303  # (5)
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # (7)
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-cold-1/elasticsearch-cold-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-cold-1/elasticsearch-cold-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-cold-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

The node will bind to the public address and will also use it as its publish address.
Rack ID or datacenter name. This is meant for ES to effectively and safely manage replicas. For smaller installations a hostname is fine.
Attributes node.attr.data are in the configuration because of backward compatibility for legacy ILM, where custom allocation by node.attr.data is used. This applies for installations of LogMan.io before 01/2024.
The name of the Elasticsearch cluster. There is only one Elasticsearch cluster in LogMan.io.
Ports for internal communication among nodes.
Node roles are here for ILM default allocation to work properly.
Memory allocated by this Elasticsearch instance. 31 GB is the maximum recommended value and the server must have adequate memory available (if there are three Elasticsearch nodes with 31 GB and one master with 4 GB, there must be at least 128 GB available).
Intial master nodes are the instance IDs of all Elasticsearch master nodes available in the LogMan.io cluster. The master nodes names must be aligned with node.name. In LogMan.io (as defined by Maestro), it is the same as INSTANCE_ID.

Index templates¶

LogMan.io Depositor creates its own index templates with the events index from the event lane's elasticsearch configuration, adding the postfix -template. All previous index templates, if present, must have a different name and their priority set to 0.