Kafka topics

LogMan.io default topics

The following topics are default LogMan.io topics used to pass processed events (log lines) among individual components such as LogMan.io Ingestor, LogMan.io Parser etc. See the following table:

Kafka topic Description Kafka consumer groups Producer pipelines Consumer pipelines
lmio-lookups Stores lookup events such as information about update of a lookup item etc. lmio-parser-x, lmio-correlator-x, lmio-watcher parser/LookupModificationPipeline, correlator/LookupModificationPipeline, watcher/LookupChangeStreamPipeline, parser/ParsersPipeline parser/LookupChangeStreamPipeline correlator/LookupChangeStreamPipeline watcher/LookupModificationPipeline parser/ParsersPipeline
lmio-output Stores parsed output correlations/alerts as well as lookup events. lmio-integ, lmio-watcher-output correlator/OutputPipeline, parser/ParsersPipeline integ/ArcSightPipeline watcher/OutputToInputPipeline
lmio-events Stores parsed log events. lmio-dispatcher, lmio_correlator-x parser/EnrichersPipeline, watcher/OutputToInputPipeline, watcher/LookupLogPipeline, dispatcher/ToEventsPipeline dispatcher/EventsPipeline
lmio-others Stores unparsed log events and parsing errors. lmio-dispatcher parser/ErrorPipeline, dispatcher/ToOthersPipeline dispatcher/OthersPipeline

The default topics are present at every LogMan.io deployment and can be set up using Kafka topics initializer in LogMan.io Parser.

LogMan.io topics for collected events

Collected events topics are specific for every data source type and tenant (customer). The standard for naming such Kafka topics is as follows:

collected-<tenant>-<type>

where tenant is the lowercase tenant name and type is the data source type. Examples include:

collected-railway-syslog
collected-ministry-files
collected-johnandson-databases

The summary is included in the following table:

Kafka topic Description Kafka consumer groups Producer pipelines Consumer pipelines
collected-tenant-type Stores raw collected logs for tenant in type data type. lmio-parser-* ingestor/WebSocketPipeline parser/ParsersPipeline