Mapping declarations

After all declared fields are obtained from parsers, the fields typically have to be renamed according to some schema (ECS, CEF) in a process called mapping.

Why is mapping necessary?

To store event data in Elasticsearch, it's essential that the field names in the logs align with the Elastic Common Schema (ECS), a standardized, open-source collection of field names that are compatible with Elasticsearch. The mapping process renames the fields of the parsed logs according to this schema. Mapping ensures that logs from various sources have unified, consistent field names, which enables Elasticsearch to interpret them accurately.

Important

By default, mapping works as a filter. Make sure to include all fields you want in the parsed output in the mapping declaration. Any field not specified in mapping will be removed from the event.

Writing a mapping declaration

Write mapping delcarations in YAML. (Mapping declarations do not use SP-Lang expressions.)

define:
    type: parser/mapping
    schema: /Schemas/ECS.yaml

mapping:
    <original_key>: <new_key>
    <original_key>: <new_key>
    ...

Specify parser/mapping as the type in the define section. In the schema field, specify the filepath to the schema you're using. If you use Elasticsearch, use the Elastic Common Schema (ECS).

To rename they key and change the data type of the value:

mapping:
    <original_key>:
        field: <new_key>
        type: <new_type>

Find available data types here.

To rename the key without changing the data type of the value:

mapping:
    <original_key>: <new_key>

Example

Example

For the purpose of the example, let's say that we want to parse a simple event in JSON format:

{
    "act": "user login",
    "ip": "178.2.1.20",
    "usr": "harry_potter",
    "id": "6514-abb6-a5f2"
}

and we would like the final output look like this:

{
    "event.action": "user login",
    "source.ip": "178.2.1.20",
    "user.name": "harry_potter"
}

Notice that the key names in the original event differ from the key names in the desired output.

For the initial parser declaration in this case, we can use a simple JSON parser:

10_parser.yaml

define:
type: parser/json

This parser will create a list of key-value pairs that are exactly the same as the original ones.

To change the names of individual fields, we create this mapping delcaration file, 20_mapping_ECS.yaml, in which we describe what fields to map and how:

20_mapping_ECS.yaml

---
define:
type: parser/mapping  # determine the type of declaration
schema: /Schemas/ECS.yaml  # which schema is applied

mapping:
    act: 'event.action'
    ip: 'source.ip'
    usr: 'user.name'

This declaration will produce the desired output. (Data types have not been changed.)