Parsing lookups

When a lookup is received from LogMan.io Collector via LogMan.io Ingestor, it can either be a whole lookup content (full frame), or just one record (delta frame).

Preprocessing

Based on the input lookup file format, a preprocessor should be used in order to simplify following declarations and optimize the speed of lookup loading. Usually, either JSON, XML or CSV preprocessor will be used:

---
define:
  name: Preprocessor for CSV
  type: parser/preprocessor

function: lmiopar.preprocessor.CSV

Thus, the parsed file content is stored in CONTEXT, where it can be accessed from.

Full frame

In order to store the entire lookup in ElasticSearch through LogMan.io Watcher, and notify other instances of LogMan.io Parser and LogMan.io Correlator about the change in the entire lookup, a Cascade Parser declaration should be used with target: lookup configuration.

Thus, the lookup will not enter the input topic, but the lookups topic, from where it is going to be processed by LogMan.io Watcher to update data in ElasticSearch.

The LogMan.io Watcher expects the following event format:

{
    'action': 'full',
    'data': {
        'items': [{
                '_id': 'myId',
                ...
            }
        ]
    },
    'lookup_id': 'myLookup'
}

where action equals full signifies, that the existing lookup content should be replaced with the items in data.

To create this structure, use the following declarative example of Cascade Parser.

Sample declaration

---
define:
  name: Demo of lookup loading parser
  type: parser/cascade
  target: lookup

parse:
    !DICT
    set:
      action: full
      lookup_id: myLookup
      data:
        !DICT
        set:
          items:
            !FOR
            each: !ITEM CONTEXT CSV
            do:
              !DICT
              set:
                _id: !ITEM ARG myId
                ...

When the lookup content enters the LogMan.io Parser, the parsed lookup is being sent to LogMan.io Watcher to store it in ElasticSearch.

Delta frame

In order to update ONE item in an existing lookup in ElasticSearch through LogMan.io Watcher, and notify other instances of LogMan.io Parser and LogMan.io Correlator about the change in the lookup, a Cascade Parser declaration should be used with target: lookup configuration.

Thus, the lookup item will not enter the input topic, but the lookups topic, from where it is going to be processed by LogMan.io Watcher to update data in ElasticSearch.

The LogMan.io Watcher expects the following event format:

{
    'action': 'update_item',
    'data': {
        '_id': 'existingOrNewItemId',
        ...
    },
    'lookup_id': 'myLookup'
}

where action equals update_item signifies, that the existing lookup item content should be replaced items in data, or a new lookup item should be created.

To create this structure, use the following declarative example of Cascade Parser.

Sample declaration

---
define:
  name: Demo of lookup item loading parser
  type: parser/cascade
  target: lookup

parse:
    !DICT
    set:
      action: update_item
      lookup_id: myLookup
      data:
        !DICT
        set:
          _id: !ITEM CONTEXT CSV.0.myID
          ...

When the lookup content enters the LogMan.io Parser, the parsed lookup is being sent to LogMan.io Watcher to store it in ElasticSearch.