Install using Docker

Prerequisites

Docker

Docker is necessary for deployment of all subsequent applications in containers, namely Kafka, ElasticSearch, NGINX and individual pumps.

The following guide explains how to install Docker on individual operating systems.

CentOS

1.) Make sure you can download files from the Internet using wget.

2.) Uninstall the pre-installed versions of Docker, if there are some:

sudo yum remove docker \
  docker-common \
  docker-selinux \
  docker-engine

3.) Install dependencies for Docker:

sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

4.) Add repository to download Docker from to the installation manager:

sudo yum-config-manager \
   --add-repo \
   https://download.docker.com/linux/centos/docker-ce.repo

5.) Install Docker:

sudo yum install docker-ce

6.) Configure Docker, if necessary.

For instance, if you need to use your custom subnet for the default Docker bridge network interface, create the /etc/docker/daemon.json file with the following content:

{
  "bip": "172.26.0.1/24"
}

Due to reported Docker Swarm issues, please use /24 as mask.

7.) Start the Docker daemon:

sudo systemctl start docker

8.) Test if the Docker daemon runs:

docker ps

You should see an empty list of running containers and no error message.

Login to docker.teskalabs.com

To pull custom Docker images from TeskaLabs’ Docker registry, it is necessary to first login with the username and password provided to you by TeskaLabs developers.

docker login docker.teskalabs.com

After successful login, the user is free to pull all Docker containers from subfolder named <PARTNER_CODE>.

Docker Compose

Docker Compose serves to pull and deploy multiple images at once using a configuration file named docker-compose.yml.

The following guide explains how to install Docker Compose on individual operating systems.

CentOS

1.) Make sure the Docker is installed (see the guide above)

2.) Download and install Docker compose with the following commands:

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/bin/docker-compose

sudo chmod +x /usr/bin/docker-compose

3.) Make sure /usr/bin is inside the PATH variable:

PATH=$PATH:/usr/bin

4.) Check that Docker Compose is installed using the following command:

docker-compose --version

5.) In order to work with Docker Compose, you need to have a docker-compose.yml with all images you want to pull and run specified inside. This file is recommended to be located inside the /opt/docker-compose folder:

mkdir -p /opt/docker-compose

vi /opt/docker-compose/docker-compose.yml

Docker Compose Setting

Environment variables

Generic environment variables should be set in .env file located next to individual docker-compose.yml files in the same folder.

Every container created by Docker Compose is named in the following manner:

<project_name>_<image_name>_<instance_number>

Where project name is defined in the COMPOSE_PROJECT_NAME environment variable in the .env file, i. e.:

COMPOSE_PROJECT_NAME=my-location

Where location is the deployment location (name of the company that manages the server etc.).

Multiple instances

Multiple instances of the same image are created using Docker Compose Scale option, which will thus ensure horizontal scaling of the container.

To scale a specific container, use the following command:

docker-compose scale syslog-lmio-parser=3

There are going to be three containers with the same name, only differing in the instance number (see above).

Docker Swarm

Docker Swarm is a cluster of Docker servers, that makes sure the containers on such servers all share the so-called overlay network. Hence they can ping, see and communicate with one another.

List of ports that need to be open

The following ports need to be open on every server, that is going to be included in the Docker Swarm cluster. Docker Swarm uses these ports for communication:

  • 2376/tcp
  • 2377/tcp
  • 7946/tcp
  • 50/tcp
  • 7946/udp
  • 4789/udp
  • 50/udp

Leader node

1.) On the leader node / server, just run the following command and copy its output:

docker swarm init

The output is a command (docker swarm join ...) with a specific token, which will enable other servers to join the swarm.

Worker node

1.) On other nodes / server, just run the docker swarm join ... command from the previous steps. You should see a message that the worker node successfully joined the Docker Swarm.

Deployment

Make sure you are given access to lmio-deploy repository, where templates for both the build and production (including collector) environments are located in.

Based on the repository, you should construct site configuration for all components (parser(s), ingestor(s) etc.) and Docker Compose files for every server in the LogMan.io cluster.

When the repository is ready, clone it to /opt folder at every server:

cd /opt
git clone https://gitlab.com/TeskaLabs/<PARTNER_GROUP>/<MY_CONFIG_REPO_PATH>

Enter the repository and deploy the server specific Docker Compose file:

docker-compose -f docker-compose-<SERVER_ID>.yml pull
docker-compose -f docker-compose-<SERVER_ID>.yml build
docker-compose -f docker-compose-<SERVER_ID>.yml up -d

Check that all containers are running:

docker ps

Utilities

Status script

To check the status of a Docker container, the following sample shell script can be used. The script also implements start, stop and restart actions upon the container.

#!/bin/bash
. /etc/rc.d/init.d/functions

CONTAINER=<YOUR_CONTAINER_NAME>

RUNNING=$(docker inspect --format="{ .State.Running }" $CONTAINER 2> /dev/null)

start(){
	if [ $? -eq 1 ]; then
	  echo "$CONTAINER does not exist."
	  exit 3
	fi

	echo "STARTING ..."
	docker start $CONTAINER
}

stop(){
	if [ $? -eq 1 ]; then
	  echo "$CONTAINER does not exist."
	  exit 3
	fi

	if [ "$RUNNING" == "false" ]; then
	  echo "$CONTAINER is not running."
	  exit 2
	fi

	echo "STOPPING ..."
	docker stop $CONTAINER
}

reload(){
	if [ $? -eq 1 ]; then
	  echo "UNKNOWN - $CONTAINER does not exist."
	  exit 3
	fi

	echo "RELOADING ..."
	docker restart $CONTAINER
}

status(){
	if [ $? -eq 1 ]; then
	  echo "$CONTAINER does not exist."
	  exit 3
	fi

	if [ "$RUNNING" == "false" ]; then
	  echo "$CONTAINER is not running."
	  exit 2
	fi

	STARTED=$(docker inspect --format="{ .State.StartedAt }" $CONTAINER)
	NAME=$(docker inspect --format="{ .Name }" $CONTAINER)
	NETWORKMODE=$(docker inspect --format="{ .HostConfig.NetworkMode }" $CONTAINER)
	NETWORK=$(docker inspect --format="{ .NetworkSettings.Networks."$NETWORKMODE".IPAddress }" $CONTAINER)

	echo "Ok - $CONTAINER is running. IP: $NETWORK, StartedAt: $STARTED, Named: $NAME"
}

case "$1" in
		start)
			start
			;;
		stop)
			stop
			;;
		restart)
			reload
			;;
		reload)
			reload
		;;
		status)
			status
			;;
		*)
		echo $"Usage: $0 {start|stop|restart|reload|status}"
esac