Skip to content

Lookups

Lookups are list of entities with attributes that are relevant either for parsing or for detection of cybersecurity incidents.

Lookups can be simple list of suspicious IP addresses, active VPN connections and so on.

Lookups can be list of user names with user attributes like user.id, user.email and so on.

Lookups can be lists of compound keys like IP address and user name combinations for monitoring user activity.

Declaration

All lookups are defined by their declarations stored in the /Lookups folder.

The convention for naming declaration is lookupname.yaml, for instance myuserlookup.yaml:

---
define:
  type: lookup
  name: myuserlookup

keys:
  - name: user.id
    type: str

fields:
  username:
    type: str

In define, specify the lookup type, lookup name (tenant information will be added automatically), keys with their names (optional) and types and fields in the output record structure. The record structure is NOT based on schema and should NOT contain dots.

IP lookups

There are two types of IP lookups: lookup/ipaddressrange for single IP addresses and lookup/ipaddress for IP address ranges

IP lookups are file lookups created from CSV files etc. that contain ranges of IP addresses (like 34603520,34603775).

The creation of synchronous lookups from CSV files is handled by a IP Address Processor service (see configuration), whose output is stored in /lookups/ip folder.

The sample input for IP lookups looks as follows:

34603520,34603775,CZ,Prague,,Prague,,50.0755,14.4378,
34706176,34706431,CZ,Prague,,Prague,,50.0755,14.4378,
34949632,34949887,CZ,Prague,,Prague,,50.0755,14.4378,
35063808,35064831,CZ,Prague,,Prague,,50.0755,14.4378,
37289216,37289471,CZ,Prague,,Prague,,50.0755,14.4378,
83965952,83966719,CZ,Karlovarsky kraj,,Sokolov,,50.1807,12.6396,
83966720,83966975,CZ,Karlovarsky kraj,,Svatava,,50.1922,12.6252,
83966976,83967999,CZ,Karlovarsky kraj,,Sokolov,,50.1807,12.6396,
85366784,85367039,CZ,Kralovehradecky kraj,,Trutnov,,50.5692,15.9133,
86157760,86159103,CZ,Prague,,Prague,,50.052,14.4411,
86451456,86451711,CZ,Hlavni mesto Praha,,New Town,,50.0794,14.4295,
86493184,86493439,CZ,Ustecky kraj,,Chomutov,,50.4583,13.3974,
86493440,86493695,CZ,Vysocina,,Cernovice,,49.3727,14.9609,
86493696,86494207,CZ,Prague,,Prague,,50.0755,14.4378,
86761472,86761983,CZ,Hlavni mesto Praha,,Prague,,50.0755,14.4378,
86761984,86762239,CZ,Jihocesky kraj,,Ceske Budejovice,,48.9926,14.4741,
86762240,86762751,CZ,Jihocesky kraj,,Ceske Budejovice,,48.9925,14.474,
86762752,86763007,CZ,Jihocesky kraj,,Ceske Budejovice,,48.9926,14.4741,
86763008,86763519,CZ,Hlavni mesto Praha,,Prague,,50.0755,14.4378,
86849536,86849791,CZ,Hlavni mesto Praha,,Prague,,50.052,14.4411,
86849792,86850047,CZ,Prague,,Prague,,50.052,14.4411,
86850048,86850303,CZ,Hlavni mesto Praha,,Prague,,50.052,14.4411,
86850304,86850815,CZ,Prague,,Prague,,50.052,14.4411,

All provided CSV files need to be accompanied by a YAM file with description of the individual attributes like (except the first IP ranges fields):

- name: city
  type: str
- name: country
  type: str

IP lookups are by definition synchronous.

Synchronous lookups

Synchronous lookups are lookups loaded from files, which include:

  • IP lookups
  • ElasticSearch/MongoDB lookups serialized to file

The creation of synchronous lookups is handled by a LogMan.io Lookup Builder (see configuration), whose output stored in /lookups folder.

Asynchronous lookups

Lookups directly loaded from ElasticSearch/MongoDB.

If synchronous lookup file is missing or is corrupted, the processing is automatically handled by asynchronous lookups.

Asynchronous lookups require less setting, but are less optimal than synchronous lookups.