Lookups API

Lookup changes can include creating or deleting the entire lookup structure, as well as adding, updating, or removing specific items within a lookup. Additionally, items can be automatically deleted when they expire. These changes can be made through the system's UI (HTTPS API) or through Apache Kafka.

The lookup events are sent by each component that creates the lookup events to lmio-lookups topics.

Lookup event structure

A lookup event has a JSON structure with three mandatory attributes: action, lookup_id, and data. The @timestamp and tenant attributes are added automatically as well as other configured meta attributes.

action

Specifies the action the lookup event caused. The action can be taken on the entirety lookup or just one of its items. Refer to the list below to see all available actions and their associated events.

lookup_id

ID or name of the lookup. The lookup ID in lookup events also contains the tenant name after the period . character, so every component knows which tenant the lookup is specific for.

data

Specification od lookup data (i.e. lookup item) to be created or updated, as well as meta information (in case of item deletion).

Lookup items contain their ID in the _id attribute of the data structure. The _id is a string based on:

Single key

If the lookup has only one key (e.g. userName), the _id is the value itself for a string value.

'data': {
    '_id': 'JohnDoe'
}
...

If the value is in bytes, _id is UTF-8 decoded string representation of the value. If the value is neither a string nor bytes, it is handled the same way as ID when using compound keys.

Compound key

If the lookup consists of more keys (e.g. [userName, location]), the _id is a hash representation of the value.

The original values are then stored in _keys attribute inside the data structure:

'data': {
    '_id': '<INTERNAL_HASH>',
    '_keys': ['JohnDoe', 'Company 1']
}
...

Create lookup

When a lookup is created, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'create_lookup',
    'data': {},
    'metadata': {
        '_keys': ['key1name', 'key2name' ...]
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Meta data contains information about the lookup creation, such as names of the individual keys (e.g. [userName, location]) in the case of compound keys.

Delete lookup

When a lookup is deleted, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_lookup',
    'data': {},
    'lookup_id': 'myLookup.tenant'
}

Create item

When an item is created, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'create_item',
    'data': {
        '_id': 'newItemId',
        '_keys': [],
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Update item

When an item is updated, the following action is produced:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'update_item',
    'data': {
        '_id': 'existingOrNewItemId',
        '_keys': [],
        ...
    },
    'lookup_id': 'myLookup.tenant'
}

Delete item

When an item is deleted, the following action is produced.

Expiration

In case of deletion due to an expiration:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_item',
    'data': {
        '_id': 'existingItemId',
        'reason': 'expiration'
    },
    'lookup_id': 'myLookup.tenant'
}

Please note: Unless the option use_default_expiration_when_update is disabled (set to false) in the Lookup meta information, the expiration is refreshed with each lookup item update (current time + default expiration). Thus the deletion due to expiration will happen only if there was no update of the item in the meantime for the duration of the expiration period.

Delete

For other reasons:

{
    '@timestamp': <UNIX_TIMESTAMP>,
    'tenant': <TENANT>,
    'action': 'delete_item',
    'data': {
        '_id': 'existingItemId',
        'reason': 'delete'
    },
    'lookup_id': 'myLookup.tenant'
}