Lookup events

When a change in one of the lookups or its items happens, usually due to a triggered lookup event or modification via UI, all LogMan.io Correlator instances are notified using an incoming event.

The lookup change may be either the creation or deletion of the lookup or creation, update, deletion as well as deletion caused by expiration of individual lookup items.

The lookup events are sent by each component, that creates the lookup events, to lmio-lookups topics. If the ElasticSearch update was successful, LogMan.io Watcher sends the event also to lmio-stores Kafka topic to notify about the lookup status update.

Lookup event structure

Lookup event is a JSON structure with three mandatory attributes: action, lookup_id and data. The @timestamp and tenant attributes are added automatically as well as other configured meta attributes.

action

Specifies the action the lookup event caused. The action is taken on the entire lookup or just one of its items. Refer to the list below to see all available actions and their associated events.

lookup_id

ID/name of the lookup. The lookup ID in lookup events also contain the tenant name after the dot . character, so every component knows, which tenant the lookup is specific for.

data

Specification od lookup data (i. e. lookup item) to be created/updated, as well as meta information (in case of item deletion).

Lookup items contain their ID in _id attribute of the data structure. The _id is a string based on:

Single key

If the lookup has only one key (f. e. userName), the _id is the value itself in case of string.

'data': {
    '_id': 'JohnDoe'
}
...

If the value is bytes, _id is UTF-8 decoded string representation of the value. If the value is neither string nor bytes, it is handled the same way as ID, when using compound keys.

Compound key

If the lookup consists of more keys (f. e. [userName, location]), the _id is a hash representation of the value.

The original values are then stored in _keys attribute inside the data structure:

'data': {
    '_id': '<INTERNAL_HASH>',
    '_keys': ['JohnDoe', 'Company 1']
}
...

Create lookup

When a lookup is created, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'create_lookup',
    'data': {},
    'metadata': {
        '_keys': ['key1name', 'key2name' ...]
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Meta data contain information about the lookup creation, such as names of the individual keys (f. e. [userName, location]) in the case of compound keys.

Delete lookup

When a lookup is deleted, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_lookup',
    'data': {},
    'lookup_id': 'myLookup.tenant'
}

Create item

When an item is created, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'create_item',
    'data': {
        '_id': 'newItemId',
        '_keys': [],
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Update item

When an item is updated, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'update_item',
    'data': {
        '_id': 'existingOrNewItemId',
        '_keys': [],
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Delete item

When an item is deleted, the following action is produced.

Expiration

In case of deletion due to an expiration:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_item',
    'data': {
        '_id': 'existingItemId',
        'reason': 'expiration'
    },
    'lookup_id': 'myLookup.tenant'
}

Please note: Unless the option use_default_expiration_when_update is disabled (set to false) in the Lookup Meta information, the expiration is refreshed with each lookup item update (current time + default expiration). Thus the deletion due to expiration will happen only if there was no update of the item in the meantime for the duration of the expiration period.

Delete

In other reasons:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_item',
    'data': {
        '_id': 'existingItemId',
        'reason': 'delete'
    },
    'lookup_id': 'myLookup.tenant'
}