Skip to content

TeskaLabs LogMan.io Integration Service

LogMan.io Integ allows TeskaLabs LogMan.io to be integrated with supported external systems via expected message format and output/input protocol.

LogMan.io Integ utilizes declarative Transformers specified in YAML files and stored in a group (like integ etc.) in the LogMan.io Library.

Configuration

To connect LogMan.io with an external system like ArcSight, configure an instance of LogMan.io Integ, while specifying the transformer group, Kafka input and Kafka or TCP output(s) in custom sections like MyTCPOutput:

# Set transformers

[declarations]
library=/library
groups=integ
include_search_path=filters/*

# Set input

[connection:KafkaConnection]
bootstrap_servers=lm1:19092,lm2:29092,lm3:39092

[pipeline:TransformersPipeline:KafkaSource]
topic=lmio-events

# Set output(s) - they are selected by individual transformers

[MyTCPOutput]
address=lm1:9999

[MyKafkaOutput]
topic=lmio-output

Transformers

Transformers are declarative processors, that create their own dedicated pipeline with KafkaSource and output sink.

Example

---
define:
  name: ArcSight Transformer for Correlations
  type: transformer/default
   format: string

output:
  - type: kafka/tcp/unix-stream/null
    config: MyTCPOutput
    format: string/json
    latch: 0

predicate:
  !EQ
  - !ITEM EVENT type
  - Correlation

transform:
  !JOIN
  delimiter: ""
  items:
  - !JOIN
    delimiter: "|"
    items:
    - CEF:0
    - TeskaLabs
    - LogMan.io
    - 1.2.3
    - !ITEM EVENT name
  - !DICT.FORMAT
    what: !EVENT
    type: cef

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be transformer/default.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Item field_alias (optional)

Name of the field alias lookup to be loaded, so that alias names of event attributes can be used in the declaration alongside their canonical names.

Section output

Section output specifies the lists of outputs.

Item type

Type of the output, f. e. kafka, unix-stream, tcp, null.

Item format

Format of the event produced by the transformator, either string or json (default: string).

Item config

Config section for the given output stored in .conf file, such as MyKafkaOutput or MyTCPOutput (see configuration section above).

Specify topic for Kafka output and address for TCP/Unix Stream.

Item latch

If set, output events are stored in latch queue to be accessible via periodic API call to /latch. The number of stored events is passed through value, f. e. latch: 30 will keep last 30 events for each transformer. Default: !!null.

Section predicate (optional)

The predicate filters incoming events using an expression. If the expression returns True, the event will enter transform section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

This section can be used to speed-up the integration by skipping lines with obviously non-relevant content.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in include or filters folder (this one is a global folder located at the top hierarchy of the LogMan.io library), use !INCLUDE statement:

!INCLUDE predicate_filter

where predicate_filter is the name of the file plus .yaml extension. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section transform

This section specifies the actual transforming mechanism. It expects a dictionary to be returned or None, which means that the transforming was not successful.