Skip to content

TeskaLabs Architecture

Diagram: High-level Architecture

lmio-collector Collector serves to receive log lines from various sources such as SysLog NG, files, Windows Event Forwarding, databases via ODBC connectors and so on. The log lines may be further processed by a declarative processor and put into Ingestor via WebSocket.

lmio-ingestor Ingestor receives events via WebSocket, transforms them to Kafka-readable format and put them to Kafka collected- topic. There are multiple ingestors for different event formats, such as SysLog, databases, XML and so on.


Diagram: Parser Architecture Parser runs in multiple instances to receive different formats of incoming events (different Kafka topics) and/or the same events (the instances then run in the same Kafka group to distribute events among them). Parser loads the Library via ZooKeeper or from files to load declarative parsers and enrichers from configured parsing groups.

If the events were parsed by the loaded parser, they are put to lmio-events Kafka topic, otherwise they enter the lmio-others Kafka topic.

lmio-dispatcher Dispatcher loads events from lmio-events Kafka topic and sends them both to all subscribed (via ZooKeeper) Correlator instances and ElasticSearch in the appropriate index, where all events can be queried and visualized using Kibana. Dispatcher runs in multiple instances as well.


Diagram: Correlator Architecture Correlator uses ZooKeeper to subscribe to all Dispatcher instances to receive parsed events (log lines etc.). Then Correlator loads the Library from ZooKeeper or from files to create correlators based on the declarative configuration. Events produced by correlators (Window Correlator, Match Correlator) are then handed down to Dispatcher and Watcher via Kafka.

lmio-watcher Watcher observes changes in lookups used in Parsers and Correlators instances. When a change occurs, all running components that use Library are notified via Kafka topic lmio-lookups about the change and the lookup is updated in the ElasticSearch, which serves as a persistent storage for all lookups.

lmio-integ Integ allows to be integrated with supported external systems via expected message format and output/input protocol.