LogMan.io Integ allows LogMan.io to be integrated with supported external systems via expected message format and output/input protocol.
LogMan.io Integ utilizes declarative Transformers specified in YAML files and stored in a group (like integ etc.) in the LogMan.io Library.
To connect LogMan.io with an external system like ArcSight,
configure an instance of LogMan.io Integ,
while specifying the transformer group,
Kafka input and Kafka or TCP output(s) in custom sections like
# Set transformers [declarations] library=/library groups=integ include_search_path=filters/* # Set input [connection:KafkaConnection] bootstrap_servers=lm1:19092,lm2:29092,lm3:39092 [pipeline:TransformersPipeline:KafkaSource] topic=lmio-events # Set output(s) - they are selected by individual transformers [MyTCPOutput] address=lm1:9999 [MyKafkaOutput] topic=lmio-output
Transformers are declarative processors, that create their own dedicated pipeline with KafkaSource and output sink.
--- define: name: ArcSight Transformer for Correlations type: transformer/default format: string output: - type: kafka/tcp/unix-stream/null config: MyTCPOutput format: string/json latch: 0 predicate: !EQ - !ITEM EVENT type - Correlation transform: !JOIN delimiter: "" items: - !JOIN delimiter: "|" items: - CEF:0 - TeskaLabs - LogMan.io - 1.2.3 - !ITEM EVENT name - !DICT.FORMAT what: !EVENT type: cef
This section contains the common definition and meta data.
Shorter human-readable name of this declaration.
The type of this declaration, must be
Longed, possibly multiline, human-readable description of the declaration.
Name of the field alias lookup to be loaded, so that alias names of event attributes can be used in the declaration alongside their canonical names.
output specifies the lists of outputs.
Type of the output, f. e. kafka, unix-stream, tcp, null.
Format of the event produced by the transformator, either
Config section for the given output stored in
MyTCPOutput (see configuration section above).
topic for Kafka output and
address for TCP/Unix Stream.
If set, output events are stored in latch queue to be accessible via periodic API call to
/latch. The number of stored events is passed through value, f. e. latch: 30 will keep last 30 events for each transformer. Default: !!null.
predicate filters incoming events using an expression.
If the expression returns
True, the event will enter
If the expression returns
False, then the event is skipped.
Other returned values are undefined.
This section can be used to speed-up the integration by skipping lines with obviously non-relevant content.
Include of nested predicate filters
Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.
If you want to include an external predicate filter, located either in
(this one is a global folder located at the top hierarchy of the LogMan.io library),
predicate_filter is the name of the file plus
The content of
predicate_filter.yaml is an expression to be included, like:
--- !EQ - !ITEM EVENT category - "MyEventCategory"
This section specifies the actual transforming mechanism.
It expects a dictionary to be returned or
None, which means that the transforming was not successful.