Mapping

After all declared fields are obtained from parsers, the fields typically have to be renamed according to some schema (ECS, CEF) in a process called mapping.

Why is mapping necessary?

To store event data in Elasticsearch, it's essential that the field names in the logs align with the Elastic Common Schema (ECS), a standardized, open-source collection of field names that are compatible with Elasticsearch. The mapping process renames the fields of the parsed logs according to this schema. Mapping ensures that logs from various sources have unified, consistent field names, which enables Elasticsearch to interpret them accurately.

Important

By default, mapping works as a filter. Make sure to include all fields you want in the parsed output in the mapping declaration. Any field not specified in mapping will be removed from the event.

Writing a mapping declaration

Write mapping delcarations in YAML. (Mapping declarations do not use SP-Lang expressions.)

define:
    type: parser/mapping
    schema: /Schemas/ECS.yaml

mapping:
    <original_key>: <new_key>
    <original_key>: <new_key>
    ...

Specify parser/mapping as the type in the define section. In the schema field, specify the filepath to the schema you're using. If you use Elasticsearch, use the Elastic Common Schema (ECS).

To rename the key and change the data type of the value:

Warning

New data type must respond to the data type specified in the schema for this field.

By specifying type: auto, the data type will be automatically determined from the schema based of field name.

mapping:
    <original_key>:
        field: <new_key>
        type: <new_type>

Find available data types here.

To rename the key without changing the data type of the value:

mapping:
    <original_key>: <new_key>

To rename the key stored in JSON object:

mapping:
    <jsonObject> </jsonKey>: <new_key>

Name of the JSON object and JSON key must be separated by space. JSON key is always starts with /, and every next level is separated by /. Same as before it is possible to change data type by specifying type field.

Example

Example

For the purpose of the example, let's say that we want to parse a simple event in JSON format:

{
    "act": "user login",
    "ip": "178.2.1.20",
    "usr": "harry_potter",
    "id": "6514-abb6-a5f2"
}

and we would like the final output look like this:

{
    "event.action": "user login",
    "source.ip": "178.2.1.20",
    "user.name": "harry_potter"
}

Notice that the key names in the original event differ from the key names in the desired output.

For the initial parser declaration in this case, we can use a simple JSON parser:

10_parser.yaml

define:
type: parser/json

This parser will create a JSON object and will store it in json field.

To change the names of individual fields, we create this mapping delcaration file, 20_mapping_ECS.yaml, in which we describe what fields to map and how:

20_mapping_ECS.yaml

---
define:
type: parser/mapping  # determine the type of declaration
schema: /Schemas/ECS.yaml  # which schema is applied

mapping:
    json /act: 'event.action'
    json /ip: 'source.ip'
        field: 'source.ip'
        type: auto
    json /usr: 'user.name'

This declaration will produce the desired output. Data type for the source.ip field will be determined automatically based on the schema and changed accordingly.