Skip to content

Entity Correlator

Example

define:
  name: Detection of user entity behavior
  description: Detection of user entity behavior
  type: correlator/entity  
  span: 5
  delay: 5m  # analysis time = delay + resolution         

predicate:
  !AND
  - !EQ
    - !ITEM EVENT message
    - "FAIL"  
  - !EQ
    - !ITEM EVENT device.vendor
    - "Microsoft"
  - !EQ
    - !ITEM EVENT device.product
    - "Exchange Server"

evaluate:
  dimension: [tenant, source.user] 
  by: @timestamp  # Name of event field with an event time
  resolution: 60 # unit is second
  lookup_seen: active_users
  lookup_lost: inactive_users

triggers:
  lost:
    - event:
        !DICT
        type: "{str:any}"
        with:
          name: "User lost detected on periodic analysis"
          type: "Correlation"
          severity: "Low"
  seen:
    - event:
        !DICT
        type: "{str:any}"
        with:
          name: "User seen detected on periodic analysis"
          type: "Correlation"
          severity: "Low"

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be correlator/entity.

Item span

Specifies the width of the window. The unit is resolution.

Item delay (optional)

Analysis happens after a specified period in seconds and this period is based on resolution.

If there is a need to prolong the period and hence delay the analysis, delay option can be specified, such as 300 (300 seconds), 1h (3600 seconds / one hour) etc.

Item field_alias

Name of the field alias lookup to be loaded, so that alias names of event attributes can be used in the declaration alongside their canonical names.

Item aggregation_count_field

Name of the attribute, that specifies the number of events within one aggregated event, hence influencing the sum of events in analysis. Defaults to 1.

Item disabled

Boolean value that specifies whether the declaration is disabled/enabled for correlating. Defaults to false.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section predicate (optional)

The predicate filters incoming events using an expression. If the expression returns True, the event will enter evaluate section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in include or filters folder (this one is a global folder located at the top hierarchy of the LogMan.io library), use !INCLUDE statement:

!INCLUDE predicate_filter

where predicate_filter is the name of the file plus .yaml extension. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section evaluate

The evaluate section specifies primary key, resolution and other attributes that are applied on the incoming event. The evaluate function is to add the event into the two dimensional structure, defined by a time and a primary key.

Item dimension

Specifies simple or compound primary key (or dimension) for the event. The dimension is defined by names of the input event fields.

Example of the simple primary key:

evaluate:
  dimension: CustomerName

Example of the compound primary key:

evaluate:
  dimension: [CustomerName, DestinationAddress, DestinationHostname]

If exactly one dimension like DestinationHostname is a list in the original event and the correlation should happen for each one of the dimension values, the dimension should be wrapped in [ ]:

evaluate:
  dimension: [CustomerName, DestinationAddress, [DestinationHostname] ]

Item by

Specified the name of the field of the input event that contains a date/time information, which will be used for evaluation.

Item resolution (optional)

Specifies the resolution of the time aggregation of the correlator. The unit is seconds.

evaluate:
  resolution: 3600  # 1 hour

Default value: 3600

Item lookup_seen

lookup_seen specifies the lookup ID where to write seen entities with the last seen time

Item lookup_lost

lookup_lost specifies the lookup ID where to write lost entities with the last analysis time

Section triggers

The triggers section specifies kinds of actions to be taken when the periodic analysis happens.

The supported actions are lost and seen.

See correlator triggers chapter for details.

seen triggers

Seen triggers are executed when the analysis finds events, that entered the window in the analyzed time.

The dimension (entity name) can be obtained via !ITEM EVENT dimension.

Timestamp of the last event, that came to the window in the specified dimension, can be obtained via !ITEM EVENT last_event_timestamp (entity updated).

Example:

  seen:
    - lookup: user_inventory
      key: !ITEM EVENT dimension
      set:
        last_seen: !ITEM EVENT last_event_timestamp

lost triggers

Lost triggers are executed when the analysis discovers, that no events came to the specified dimension in the analyzed time (entity drop).

The dimension (entity name) can be obtained via !ITEM EVENT dimension.

Example:

  lost:
    - event:
        !DICT
        type: "{str:any}"
        with:
          name: "User drop detected on periodic analysis"
          type: "Correlation"
          severity: "Low"