Kafka Topics

LogMan.io default topics

The following topics are default LogMan.io topics used to pass processed events (log lines) among individual components such as LogMan.io Ingestor, LogMan.io Parser etc. See the following table:

Kafka topic	Description	Kafka consumer groups	Producer pipelines	Consumer pipelines
lmio-lookups	Stores lookup events such as information about update of a lookup item etc.	lmio-parser-x, lmio-correlator-x, lmio-watcher	parser/LookupModificationPipeline, correlator/LookupModificationPipeline, watcher/LookupChangeStreamPipeline, parser/ParsersPipeline	parser/LookupChangeStreamPipeline correlator/LookupChangeStreamPipeline watcher/LookupModificationPipeline parser/ParsersPipeline
lmio-output	Stores parsed output correlations/alerts as well as lookup events.	lmio-integ, lmio-watcher-output	correlator/OutputPipeline, parser/ParsersPipeline	integ/ArcSightPipeline watcher/OutputToInputPipeline
lmio-events	Stores parsed log events.	lmio-dispatcher, lmio_correlator-x	parser/EnrichersPipeline, watcher/OutputToInputPipeline, watcher/LookupLogPipeline, dispatcher/ToEventsPipeline	dispatcher/EventsPipeline
lmio-others	Stores unparsed log events and parsing errors.	lmio-dispatcher	parser/ErrorPipeline, dispatcher/ToOthersPipeline	dispatcher/OthersPipeline

The default topics are present at every LogMan.io deployment and can be set up using Kafka topics initializer in LogMan.io Parser.

LogMan.io topics for collected events

Collected events topics are specific for every data source type and tenant (customer). The standard for naming such Kafka topics is as follows:

collected-<tenant>-<type>

where tenant is the lowercase tenant name and type is the data source type. Examples include:

collected-railway-syslog
collected-ministry-files
collected-johnandson-databases

The summary is included in the following table:

Kafka topic	Description	Kafka consumer groups	Producer pipelines	Consumer pipelines
collected-tenant-type	Stores raw collected logs for tenant in type data type.	lmio-parser-*	ingestor/WebSocketPipeline	parser/ParsersPipeline