LogMan.io Collector

Configuration

By default, the collector expects lmio-collector.yaml configuration file in /data/ folder. The file path can be overridden in site.conf:

[config]
path=./etc/lmio-collector.yaml

The configuration itself is a YAML file based on BSPump declarative language.

Here, every section starts with either input:, transform:, or output:. Output creates a BSPump pipeline with specified sink/output and conversion processors. Transform performs a preprocessing of the event before is passed to the output. Input adds a source/input to a given pipeline, using output: <OUTPUT_ID> command.

input|transform|output:<TYPE>:<ID>

Every input and output is based on a type, specified after :. The last section is an ID, which is used for reference (such as input-output mentioned above).

Usage

Docker Compose

version: '3'
services:

  lmio-collector:
    image: docker.teskalabs.com/<PARTNER_CODE>/lmio-collector
    container_name: lmio-collector
    volumes:
      - ./lmio-collector:/data
    network_mode: host
    ports:
      - "8888:8888"

Command-line

python3 ./lmio-collector.py -c etc/site.conf

Configuration example

The configuration may look like this. See explanations in the comments:

---
input:ODBC:ODBCInput:
  host: 192.168.99.100
  port: 3306
  user: root
  password: root
  db: sampledb
  driver: MySQL ODBC 8.0 Unicode Driver
  query: SELECT first_name, last_name, side FROM characters;
  # true/false
  last_value_enabled: true
  # Specify table for SELECT max({}) from {};
  last_value_table: characters
  # To specify the last value query entirely, use
  # last_value_query:
  # The column in query to be used for obtainment of last value
  last_value_column: id
  # The first value to start from
  last_value_start: 0
  # Persistent storage for the current last value
  # Please make sure you use different file for every case
  last_value_storage: "./etc/mysql_character_last_value.json"
  # How often the query should be executed (seconds)
  chilldown_period: 5
  output: UnixSocketOutput

input:WEC:WECInput:
  listen: "0.0.0.0 8081"  # Where to expose the server to
  cert: "/data/cert.pem"  # Specify path to the certificate
  key: "/data/key.pem"  # Specify path to the private key
  # Specify issuer (CA) certificate thumbprints separated by space
  issuer_thumbprints: "d6982fff2104f21ab0c7ccb279217abe29c9808c"

  queries: | # Specify WEC query in format <QUERY_PATH> <QUERY_TEXT>\n
    System *[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]
    Security *[Security[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]
  read_existing_events: true

  connection_retries: 60  # How many retries is acceptable
  connection_retries_wait: 10.0  # Wait for connection retry
  heartbeat: 60  # How often the heart beat should be called upon subscriptions (in seconds)

  # Last value storage
  # Please make sure you use different file for every case
  # Also, when you change "queries", consider deleting the file
  last_value_storage: "/data/wec_last_value_storage.json"

  # Output
  output: WebSocketOutputForWindows

input:WinRM:WinRMInput:
  endpoint: http://127.0.0.1:5985/wsman
  # Authentication
  transport: ntlm
  server_cert_validation: ignore
  # cert_pem:, cert_key_pem:
  username: <DOMAIN>\<USER>  # The user must be in "Event Log Readers group"
  password: pwd
  # Read 1000 system logs once per 2 seconds
  command: wevtutil qe system /c:1000 /rd:true
  chilldown_period: 2
  # Check for duplicities:
  duplicity_check: true
  # Works with "wevtutil qe system /c:500 /rd:true"
  duplicity_reverse_order: true
  # Persistent storage for the current last value
  # Please make sure you use different file for every case
  last_value_storage: "/data/winrm_last_value.json"
  # Specify custom output
  output: WebSocketOutputForWindows

# Works only on Windows machine, i. e. not working in Linux-based Docker container
input:WinEvent:WinEventInput:
  server: localhost
  event_type: System
  # Please make sure you use different file for every case
  last_value_storage: "/data/winevent_last_value_storage.json"
  # Optimization
  # Events to be read at once
  buffer_size: 1024
  # Specify custom output
  output: FileTestOutput

input:Stream:TCPInput:
  address: 127.0.0.1:8888
  output: UnixSocketOutput

input:SubProcess:SubProcessInput:
  command: tail -f /data/tail.log
  output: UnixSocketOutput

input:File:FileInput:
  path: /data/lines/*
  chilldown_period: 10
  output: UnixSocketOutput

input:XML:XMLInput:
  path: /data/xml/*
  chilldown_period: 10
  # Every input can specify its output explicitly
  output: XMLUnixSocketOutput


# Declarative pre-processing
transform:Declarative:DeclarativeProcessor1:
  # Specify declaration file
  declaration: /data/output_declaration.yaml
  # Specify output
  output: UnixSocketOutput

# Transform XML to Dictionary
transform:XMLToDict:MyXMLParser:
  output: XMLUnixSocketOutput


output:UnixSocket:UnixSocketOutput:
  address: /myunixsocket
  # Set to true to enable logging of the events
  debug: false

output:WebSocket:WebSocketOutputForWindows:
  url: http://127.0.0.1:9999/ws

output:UnixSocket:XMLUnixSocketOutput:
  address: /myunixsocket
  # For WebSocket use "url" instead of "address"

output:File:FileTestOutput:
  path: /data/my_output_file.txt

The following is an example of declarative pre-processing, that in the case of the example above takes place in transform:Declarative:DeclarativeProcessor1 just before the event is passed to the output defined by the address.

--- !DICT
with: !EVENT
set:
  customer: MyBrilliantCustomer