Kafka Topics¶
LogMan.io default topics¶
The following topics are default LogMan.io topics used to pass processed events among individual components.
| Kafka topic | Producer | Consumer(s) | Type of stored messages |
|---|---|---|---|
received.<tenant>.<stream> |
LogMan.io Receiver | LogMan.io Parsec | raw logs / events |
events.<tenant>.<stream> |
LogMan.io Parsec | LogMan.io Depositor LogMan.io Baseliner | successfully parsed events. |
others.<tenant> |
LogMan.io Parsec | LogMan.io Depositor | unsuccessfully parsed events. |
complex.<tenant> |
LogMan.io Correlator | LogMan.io Watcher | |
lmio-alerts |
LogMan.io Alerts | ||
lmio-lookups |
LogMan.io Lookups | lookup events (information about update of a lookup item) | |
lmio-notifications |
|||
lmio-stores |
LogMan.io topics within Event Lane¶
After the provisioning of LogMan.io Collector, LogMan.io Receiver automatically creates RECEIVED Kafka topic based on the tenant and stream.
This name is of the form received.<tenant>.<stream>. This topic is unique for every Event Lane.
LogMan.io Parsec consumes from RECEIVED topic and parses the messages. When the parsing succeeds, message is sent to EVENTS topic, when it fails, message is sent to OTHERS topic.
EVENTS topic has the form events.<tenant>.<stream> and is unique for every Event Lane.
OTHERS topic has the form others.<tenant> and stores all incorrectly parsed messages, regardless of the Event Lane, unique for every tenant.
LogMan.io Depositor consumes from both EVENTS and OTHERS topics and sends the messages to the corresponding Elasticsearch indexes.
Example
Suppose the name of the tenant is hogwarts and the name of the stream is microsoft-365. Then the following topics are created:
received.hogwarts.microsoft-365: The topic where raw (unparsed) events are stored.events.hogwarts.microsoft-365: The topic where successfully parsed events are stored.others.hogwarts: If some events are parsed unsuccessfully, they are stored in this topic.