Entity Correlator

Window correlator detects incoming events based on the predicate section and stores them in data structures based on the evaluate section. If the dimension detected from the event does not produce any data, lost in the triggers section is called, otherwise seen in triggers section is called.

Example

define:
  name: Detection of user entity behavior
  description: Detection of user entity behavior
  type: correlator/entity  
  span: 5
  delay: 5m  # analysis time = delay + resolution         

logsource:
  vendor: "Microsoft"

predicate:
  !AND
  - !EQ
    - !ITEM EVENT message
    - "FAIL"  
  - !EQ
    - !ITEM EVENT device.vendor
    - "Microsoft"
  - !EQ
    - !ITEM EVENT device.product
    - "Exchange Server"

evaluate:
  dimension: [source.user] 
  by: @timestamp  # Name of event field with an event time
  resolution: 60 # unit is second
  lookup_seen: active_users
  lookup_lost: inactive_users

triggers:
  lost:
    - event:
        severity: "Low"
  seen:
    - event:
        severity: "Low"

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be correlator/entity.

Item span

Specifies the width of the window. The unit is resolution.

Item delay (optional)

Analysis happens after a specified period in seconds and this period is based on resolution.

If there is a need to prolong the period and hence delay the analysis, delay option can be specified, such as 300 (300 seconds), 1h (3600 seconds / one hour) etc.

Item aggregation_count_field

Name of the attribute, that specifies the number of events within one aggregated event, hence influencing the sum of events in analysis. Defaults to 1.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section logsource

Specifies the types of event lanes that should the incoming events be read from.

Section predicate (optional)

The predicate filters incoming events using an expression. If the expression returns True, the event will enter evaluate section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

Section evaluate

The evaluate section specifies primary key, resolution and other attributes that are applied on the incoming event. The evaluate function is to add the event into the two dimensional structure, defined by a time and a primary key.

Item dimension

Specifies simple or compound primary key (or dimension) for the event. The dimension is defined by names of the input event fields.

Example of the simple primary key:

evaluate:
  dimension: [source.ip]

Note

Tenant is added automatically to the dimension list.

Example of the compound primary key:

evaluate:
  dimension: [source.ip, destination.ip]

If exactly one dimension like DestinationHostname is a list in the original event and the correlation should happen for each one of the dimension values, the dimension should be wrapped in [ ]:

evaluate:
  dimension: [source.ip, destination.ip, [DestinationHostname] ]

Item by

Specified the name of the field of the input event that contains a date/time information, which will be used for evaluation. Default is: @timestamp.

Item resolution (optional)

Specifies the resolution of the time aggregation of the correlator. The unit is seconds.

evaluate:
  resolution: 3600  # 1 hour

Default value: 3600

Item lookup_seen

lookup_seen specifies the lookup ID where to write seen entities with the last seen time

Item lookup_lost

lookup_lost specifies the lookup ID where to write lost entities with the last analysis time

Section triggers

The triggers section specifies kinds of actions to be taken when the periodic analysis happens.

The supported actions are lost and seen.

See correlator triggers chapter for details.

seen triggers

Seen triggers are executed when the analysis finds events, that entered the window in the analyzed time.

The dimension (entity name) can be obtained via !ITEM EVENT dimension.

Timestamp of the last event, that came to the window in the specified dimension, can be obtained via !ITEM EVENT last_event_timestamp (entity updated).

Example:

  seen:
    - lookup: user_inventory
      key: !ITEM EVENT dimension
      set:
        last_seen: !ITEM EVENT last_event_timestamp

lost triggers

Lost triggers are executed when the analysis discovers, that no events came to the specified dimension in the analyzed time (entity drop).

The dimension (entity name) can be obtained via !ITEM EVENT dimension.

Example:

  lost:
    - event:
        severity: "Low"
        dimension: !ITEM EVENT dimension