Skip to content

Depositor prerequisites

LogMan.io Depositor has the following dependencies:

  • Elasticsearch
  • Apache ZooKeeper
  • Apache Kafka
  • LogMan.io Library with an /EventLanes folder and a schema in the /Schemas folder

Elasticsearch configuration

The Elasticsearch cluster needs to be configured in the following way in order for LogMan.io Depositor to work properly.

The following is a Docker Compose entry of Elasticsearch nodes, when using a 3-node cluster architecture with lm1, lm2, and lm3 server nodes.

Note

Please note that, in the Docker Compose file, the proper node roles are assigned to Elasticsearch nodes based on the ILM. For example, hot nodes for the ILM hot phase must contain node roles data_hot and data_content.

When creating Docker Compose records for Elasticsearch nodes, the following attributes must be changed:

  • NODE_ID: The name of the server where the Elasticsearch instance is running
  • INSTANCE_ID: The name of the Elasticsearch instance. Make sure its postfix -1 is changed to -2 at the second instance of this service etc. INSTANCE_ID is thus a unique identifier for each of the instances.
  • network.host: The name of the server where the Elasticsearch instance is running
  • node.attr.rack_id: The name of the server rack (for large deployments) or the name of the server where the Elasticsearch instance is running
  • discovery.seed_hosts: The server host names and ports of all Elasticseach master nodes
  • xpack.security.transport.ssl.certificate: The path to the certificate specific for the given Elasticsearch instance
  • xpack.security.transport.ssl.key: The path to the certificate key specific for the given Elasticsearch instance
  • volumes: The path to the given Elasticsearch instance's data
docker-compose.yaml: elasticsearch-master-1
  elasticsearch-master-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-master-1
      - network.host=lm1  # 
      - node.attr.rack_id=lm1  # 
      - node.name=elasticsearch-master-1
      - node.roles=master,ingest
      - cluster.name=lmio-es  # 
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # 
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9200
      - transport.port=9300  # 
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"  # 
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-master-1/elasticsearch-master-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-master-1/elasticsearch-master-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-master-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs   
    restart: always
docker-compose.yaml: elasticsearch-hot-1
  elasticsearch-hot-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-hot-1
      - network.host=lm1  # 
      - node.attr.rack_id=lm1  # 
      - node.attr.data=hot  # 
      - node.name=elasticsearch-hot-1
      - node.roles=data_hot,data_content  # 
      - cluster.name=lmio-es  # 
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # 
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9201
      - transport.port=9301  # 
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # 
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-hot-1/elasticsearch-hot-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-hot-1/elasticsearch-hot-1.key
    volumes:
      - /data/ssd/elasticsearch/elasticsearch-hot-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
docker-compose.yaml: elasticsearch-warm-1
  elasticsearch-warm-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-warm-1
      - network.host=lm1  # 
      - node.attr.rack_id=lm1  # 
      - node.attr.data=warm  # 
      - node.name=elasticsearch-warm-1
      - node.roles=data_warm  # 
      - cluster.name=lmio-es  # 
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # 
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9202
      - transport.port=9302  # 
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # 
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-warm-1/elasticsearch-warm-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-warm-1/elasticsearch-warm-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-warm-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs
docker-compose.yaml: elasticsearch-cold-1
  elasticsearch-cold-1:
    network_mode: host
    user: "1000:1000"
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9
    depends_on:
      - es-master
    environment:
      - NODE_ID=lm1
      - SERVICE_ID=elasticsearch
      - INSTANCE_ID=elasticsearch-cold-1
      - network.host=lm1
      - node.attr.rack_id=lm1  #  
      - node.attr.data=cold  # 
      - node.name=elasticsearch-cold-1
      - node.roles=data_cold  # 
      - cluster.name=lmio-es  # 
      - cluster.initial_master_nodes=elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3  # 
      - discovery.seed_hosts=lm1:9300,lm2:9300,lm3:9300
      - http.port=9203
      - transport.port=9303  # 
      - "ES_JAVA_OPTS=-Xms31g -Xmx31g"  # 
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/elasticsearch-cold-1/elasticsearch-cold-1.crt
      - xpack.security.transport.ssl.key=certs/elasticsearch-cold-1/elasticsearch-cold-1.key
    volumes:
      - /data/hdd/elasticsearch/elasticsearch-cold-1/data:/usr/share/elasticsearch/data
      - ./elasticsearch/certs:/usr/share/elasticsearch/config/certs

Index templates

LogMan.io Depositor creates its own index templates with the events index from the event lane's elasticsearch configuration, adding the postfix -template. All previous index templates, if present, must have a different name and their priority set to 0.