Skip to content

IP Enricher

IP Enricher extends events with geographic and other data associated with the given IPv6 or IPv4 address. This module replaces the IP2Location Enricher and the GeoIP Enricher, which are now obsolete.

Declaration

Section define

This section defines the name and the type of the enricher.

Item name

Short human-readable name of this declaration.

Item type

There are four types of IP Enricher available, differing in IP version (IPv4 or IPv6) and IP address representation (integer or string). Using the integer input is the faster, preferred option.

  • enricher/ipv6 processes IPv6 addresses in 128-bit decimal integer format (such as 281473902969579).
  • enricher/ipv4 processes IPv4 addresses in 32-bit decimal integer format (such as 3221226219).
  • enricher/ipv6str processes IPv6 addresses in colon-separated hexadecimal string format (such as 2001:db8:0:0:1:0:0:1) as defined in RFC 5952. It can also convert and process IPv4 string addresses (like enricher/ipv4str).
  • enricher/ipv4str processes IPv4 addresses in dotted decimal string format (such as 192.168.16.0) as defined in RFC 4001.

Item base_path

Specifies the base URL path which contains lookup zone files. It can point to: * local filesystem directory, e.g. /path/to/files/ * a location in zookeeper, e.g. zk://zookeeper-server:2181/path/to/files/ * an HTTP location, e.g. http://localhost:3000/path/to/files/

Section tenants

IP Enricher can be configured for multitenant settings. This section lists tenants which should be considered by the enricher in creating tenant-specific lookups. Events annotated with a tenant that is not listed in this section will only get global enrichment (see below).

Section zones

Specifies a list of lookup zones to be used by the enricher, plus the information whether they are global or tenant-specific.

Global lookups can enrich any event, regardless of its tenant. Tenant-specific lookups can only enrich events with matching tenant context.

Lookup zones should be ordered by their priority, highest to lowest, as the lookup iterates over the zones sequentially and stops as soon as the first match is found.

zones:
- tenant: lookup-zone-1.pkl.gz
- tenant: lookup-zone-2.pkl.gz
- global: lookup-zone-glob.pkl.gz

The zone names must match the corresponding file names including the extension. The global lookup files need to exist directly in the base_path directory. The tenant lookup files need to be organized under base_path into folders by their respective tenant. For example, assuming we have declared first_tenant and second_tenant, the zones declaration above expects the following file structure:

/base_path/
- first_tenant/
  - lookup-zone-1.pkl.gz
  - lookup-zone-2.pkl.gz
- second_tenant/
  - lookup-zone-1.pkl.gz
  - lookup-zone-2.pkl.gz
- lookup-zone-glob.pkl.gz

An incoming event whose contxt is second_tenant will first try to match in lookup second_tenant/lookup-zone-1.pkl.gz, then in second_tenant/lookup-zone-2.pkl.gz and finally in lookup-zone-glob.pkl.gz

Section attributes

This section specifies which event attributes contain the IP address and what attributes shall be added to the event if a match is found. It has the following dictionary structure:

ip_address: 
  lookup_attribute_name: event_attribute_name

The IP address is extracted from the event attribute ip_address. If it matches any lookup zone, the value of lookup_attribute_name is saved into event_attribute_name. For example:

source_ip:
  country_code: source_country

This will try to match the event based on its source_ip attribute and store the corresponding country_code value into source_country event field.

Zone name enrichment

It may be useful to record the name of the lookup zone where the match happened. To add the zone name into the event, use zone as the lookup attribute name, e.g.:

source_ip:
  zone: source_zone_name

This will add the name of the matched lookup into the event field source_zone_name.

Note that if there is a field called "zone" in the lookup, its value will be used instead of the lookup name.

The lookup name is set at the creation of the lookup file. It defaults to the name of the source file, but it can be configured to some other value. See the lmiocmd ipzone from-csv command in LogMan.io Commander for details.

Example usage

Declaration file

---
define:
  name: IPEnricher
  type: enricher/ipv6

tenants:
  - some-tenant
  - another-tenant

zones:
  - tenant: lookup-zone-1.pkl
  - global: ip2location.pkl.gz

attributes:
  ip_addr1:
    country_code: sourceCountry
    city_name: sourceCity
    L: sourceLocation
  ip_addr2:
    country_code: destinationCountry
    city_name: destinationCity
    L: destinationL
  ...

Here, the enricher reads the IP address from the event attribute ip_addr1. Then it tries to find the address in its lookup ojects: first in lookup-zone-1.pkl, then in ip2location.pkl.gz. If it finds a match, it retrieves the lookup values country_code, city_name and L, and saves them in their respective event fields sourceCountry, sourceCity and sourceLocation. It proceeds analogically for the second address ip_addr2. The result can be seen below.

Input

Feb 5 10:50:01 0:0:0:0:0:ffff:1f1f:e001 %ASA-1-105043 test

The line above may be parsed into the following dictionary.

{
    'rt': 1580899801.0,
    'msg': 'test',
    'ip_addr1': '0:0:0:0:0:ffff:1f1f:e001'
}

This is passed to the IP Enricher we declared above.

Output

{
    'rt': 1580899801.0,
    'msg': 'test',
    'ip_addr1': '0:0:0:0:0:ffff:1f1f:e001',
    'sourceCountry': 'CZ',
    'sourceCity': 'Brno',
    'sourceLocation': (49.195220947265625, 16.607959747314453)
}

IP zone lookup file

The IP lookup file is a pickled Python dictionary. It can be simply created from CSV file using the lmiocmd ipzone from-csv command found in LogMan.io Commander. The CSV needs to contain a header row with column names. There needs to be an ip_from and an ip_to column, and at least one other column with desired lookup values.

For example: 

ip_from,ip_to,zone_info,latitude,longitude
127.61.100.0,127.61.111.255,my secret base,48.224673,-75.711505
127.61.112.0,127.61.112.255,my submarine,22.917923,267.490378

NOTE The zones defined in one lookup file must not overlap.

NOTE The IP zones in the CSV file are treated as closed intervals, i.e. both ip_from and ip_to fields are included in the zone they delimit.

IP2Location

This command is also able to create a lookup file from IP2Location™ CSV databases. Note that these files don't include column names, so the header row needs to be added to the CSV file manually before creating the lookup.