Deployment

Design

lmio-watcher

LogMan.io Watcher manages the content of lookups in Elasticsearch. Watcher reads the lookup events from HTTP(S) API and Kafka.

lmio-lookupbuilder

LogMan.io Lookup Builder takes generic lookup contents from Elasticsearch and lookup declarations from the Library and builds lookup binary files. The lookup binary files are then used by other microservices such as LogMan.io Parsec, LogMan.io Correlator, etc.

lmio-ipaddrproc

LogMan.io IP Address Processor takes IP address lookup contents from Elasticsearch and lookup declarations from thè Library and builds IP lookup binary files. The IP lookup binary files are then used by other microservices such as LogMan.io Parsec, LogMan.io Correlator, etc. It also downloads built-in lookups from Azure storage from the internet.

Step-by-step guide

In order to work with lookups, follow these deployment steps. For more information about what lookups are and what they are used for, go to Lookups.

1. At every machine within the LogMan.io cluster, deploy one instance of LogMan.io Watcher.

2. At every machine within the LogMan.io cluster, deploy one instance of lmio-lookupbuilder.

   The information about configuration and records in Docker Compose is located in the Configuration section.

3. At every machine within the LogMan.io cluster, deploy one instance of lmio-ipaddrproc

   The information about configuration and records in Docker Compose is located in the Configuration section.

4. Add the path to the /lookups folder to the Docker Compose volumes section of every instance of LogMan.io Parsec, LogMan.io Correlator, LogMan.io Alerts, and LogMan.io Baseliner. The path is by default:

   volumes:
   - /data/ssd/lookups:/lookups
   

5. Include LogMan.io Watcher in the configuration file of every NGINX instance as a location record to /api/lmio-lookups:

   location /api/lmio-lookup {
       auth_request /_oauth2_introspect;
       rewrite ^/api/lmio-lookup/(.*) /$1 break;
       proxy_pass http://lmio-watcher;
   }
   

   Notice the proxy_pass that points to lmio-watcher upstream, which should be defined at the top of each NGINX configuration file:

   upstream lmio-watcher {
       server HOSTNAME_OF_FIRST_SERVER_IN_THE_CLUSTER:8952 max_fails=0 fail_timeout=30s;
       server HOSTNAME_OF_SECOND_SERVER_IN_THE_CLUSTER:8952 max_fails=0 fail_timeout=30s;
       server HOSTNAME_OF_THIRD_SERVER_IN_THE_CLUSTER:8952 max_fails=0 fail_timeout=30s;
   }
   

   Replace HOSTNAME_OF_FIRST_SERVER_IN_THE_CLUSTER, HOSTNAME_OF_SECOND_SERVER_IN_THE_CLUSTER, HOSTNAME_OF_THIRD_SERVER_IN_THE_CLUSTER with the hostnames of the servers that LogMan.io Watcher is deployed to in the LogMan.io cluster environment.

   That's it! Now you are ready to create lookup declarations and lookup content. Go back to Lookups for next steps.