LogMan.io Collector

Inputs

Available inputs and configuration options

input:ODBC:, input:TCP:, input:Stream:, input:Datagram:, input:SubProcess:, input:SmartFile:, input:FileBlock:, input:File:, input:XML:

For Windows inputs, see Collecting from Windows chapter.

input:ODBC

Provides input via ODBC driver connection to a specified database.

Configuration options related to the connection establishment:

host:  # Hostname of the database server
port:  # Port where the database server is running
user:  # Username to loging to the databse server (usually a technical/access account)
password:  # Password for the user specified above
driver:  # Pre-installed ODBC driver (see list below)
db:  # Name of the databse to access
connect_timeout:  # (optional) Connection timeout in seconds for the ODBC pool (default: 1)
reconnect_delay:  # (optional) Reconnection delay in seconds after timeout for the ODBC pool (default: 5.0)
output_queue_max_size:  # (optional) Maximum size of the output queue, i. e. in-memory storage (default: 10)
max_bulk_size:  # (optional) Maximum size of one bulk composed of the incoming records (default 2)
output:  # Which output to send the incoming events to

Configuration options related to querying the database:

query:  # Query to periodically call the database
chilldown_period:  # Specify in seconds how often the query above will be called (default: 5)
last_value_enabled:  # Enable last value duplicity check (true/false)
last_value_table:  # Specify table for SELECT max({}) from {};
last_value_column:  # The column in query to be used for obtainment of last value
last_value_storage:  # Persistent storage for the current last value (please make sure you use different file for every case)
last_value_query:  # (optional) To specify the last value query entirely (in case this option is set, last_value_table will not be considered)
last_value_start:  # (optional) The first value to start from (default: 0)
Available ODBC drivers for input:ODBC:

ODBC Driver 17 for SQL Server, MySQL ODBC 8.0 Unicode Driver, MySQL ODBC 8.0 ANSI Driver, Oracle 19 ODBC driver, MariaDB

These drivers are pre-installed in the lmio-collector Docker container.

input:TCP, input:Stream, input:Datagram

These inputs listen on a given address or a given file using TCP/UDP or Unix Socket. Make sure, the ports or files are also propagated outside of the Docker container, if using Docker.

The configuration options for listening on a given path:

address:  # Specify IPv4, IPv6 or UNIX file path to listen from (f. e. 127.0.0.1:8888 or /data/mysocket)
output:  # Which output to send the incoming events to

The following configuration options are available only for input:Datagram:

max_packet_size:  # (optional) Specify the maximum size of packets in bytes (default: 65536)
receiver_buffer_size:  # (optional) Limit the receiver size of the buffer in bytes (default: 0)

input:SubProcess

The SubProcess input runs a command as a subprocess of the LogMan.io collector, while periodically checking for its output at stdout (lines) and stderr.

The configuration options include:

command:  # Specify the command to be run as subprocess (f. e. tail -f /data/tail.log)
output:  # Which output to send the incoming events to
line_len_limit:  # (optional) The length limit of one read line (default: 1048576)
ok_return_codes:  # (optional) Which return codes signify the running status of the command (default: 0)

input:SmartFile

Simulates tail -f behavior on multiple files, whose content may be dynamically modified or the files may be deleted altogether by another process.

The available configuration options include:

path:  # File paths separated by ';'
scan_period:  # (optional) File scan period in seconds (default: 5)
read_size: 4096,  # (optional) One read cycle in bytes (default: 4096)
newline:  # (optional) File line separator, f. e. \n (default is OS line separator)
lines_per_event:  # (optional) The number of lines after which the read method enters the idle state to allow other operations to perform their tasks (default: 10000)
event_idle_time:  # (optional) The time in seconds for which the read method enters the idle state (default: 0.01)

input:File, input:FileBlock, input:XML

These inputs read specified files by lines (input:File) or as a whole block (input:FileBlock, input:XML) and pass their content further to the pipeline.

Depending on the mode, the files may be then renamed to <FILE_NAME>-processed and if more of them are specified using a wildcard, another file will be open, read and processed in the same way.

The available configuration options for opening, reading and processing the files include:

path:  # Specify the file path(s), wildcards can be used as well (f. e. /data/lines/*)
chilldown_period:  # If more files or wildcard is used in the path, specify how often in seconds to check for new files (default: 5)
output:  # Which output to send the incoming events to
mode:  # (optional) The mode by which the file is going to be read (default: 'rb')
newline:  # (optional) File line separator (default is value of os.linesep)
post:  # (optional) Specifies what should happen with the file after reading - delete (delete the file), noop (no renaming), move (rename to `<FILE_NAME>-processed`, default)
exclude:  # (optional) Path of filenames that should be excluded (has precedence over 'include')
include:  # (optional) Path of filenames that should be included
encoding:  # (optional) Charset encoding of the file's content
move_destination:  # (optional) Destination folder for post 'move', make sure it is outside of the path specified above
lines_per_event:  # (optional) The number of lines after which the read method enters the idle state to allow other operations to perform their tasks (default: 10000)
event_idle_time:  # (optional) The time in seconds for which the read method enters the idle state, see above (default: 0.01)