Storage structure

Each server within the cluster should have a dedicated fast storage on SSD storage and a slow storage on traditional hard drives (HDD).

Optinally, the system can be connected to an archive storage.

Fast storage

Fast storage contains data, that are needed to be quickly accessed, such as persisted Kafka logs/messages and data for ElasticSearch master and hot data nodes.

The fast storage has to be mounted on /data/ssd.

Inside, each folder is named according to the Docker container in the LogMan.io Deployment:

/data/ssd/kafka
/data/ssd/zookeeper
/data/ssd/influxdb
/data/ssd/es-master
/data/ssd/es-hot1
...

Size and redundancy

The fast storage must be located at mirrored dist space (aka RAID 1). That means that you need twice the physical capacity for a given logical capacity.

We recommend to have 1TB of the logical fast storage capacity.

Partitioning

  • 1x Swap partition
  • 1x LVM partition
    • LVM provides one logical volume (LV) that is mirrored over two physical devices
    • This logical volume uses Ext4 filesystem and it is mounted at /data/ssd.
    • noatime flag is recommended to be set

Important: The total swap size should always be the same size of RAM memory, stored on fast SSD disk storage. When two SSD drives are used, half of the swap size should be allocated on the each drive for a swap.

Slow storage

The slow storage contains data, that does not have to be quickly accessed, and usually contain older logs and events, such as warm and cold indices for ElasticSearch.

The slow storage has to be mounted to /data/hdd.

Inside, each folder is named according to the Docker container:

/data/ssd/es-warm1
/data/ssd/es-cold1
/data/hdd/docker
...

Note: /data/hdd/docker is sym-linked so to /var/lib/docker.

Size and redundancy

We recommend to apply disk storage redundancy such as RAID6 on the slow storage space. This means that the physical storage size will be bigger than a logical size.

Partitioning

A single big partition with Ext4 or XFS. noatime flag is recommended to be set.

Archive storage

The very old data could be moved into the archive for a long-term storage. The archive could be network attached storage (NAS), cloud-base storage such as AWS S3 etc.

The offline archive is optional component.

The data in the archive are not readilly available in the system and has to be restored back prior their use.