Skip to content

Window Correlator

Example

define:
  name: Example of the Window Correlator
  description: |
    Long multi-line descriptor
    That really goes to another linw
  type: correlator/window
  field_alias: field_alias.default
  aggregation_count_field: cnt
  disabled: false

predicate:
  !AND
  - !EQ
    - !ITEM EVENT type
    - UseIt
  - !INCLUDE predicate_filter

evaluate:
  dimension: [customer.name, destination.address, [destination.hostname] ]  
  by: Timestamp
  resolution: 5  # seconds
  minimum_growth: 10  # 10 * resolution is the default size of the segment (optional)

analyze:
  when: event
  window: hopping
  aggregate: sum
  span: 12
  test:
    !GT
    - !ARG
    - 5

trigger:
  - event:
      !DICT
      type: "{str:any}"
      with:
        category.significance: "/Compromise"

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be correlator/window.

Item field_alias

Name of the field alias lookup to be loaded, so that alias names of event attributes can be used in the declaration alongside their canonical names.

Item disabled

Boolean value that specifies whether the declaration is disabled/enabled for correlating. Defaults to false.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section predicate (optional)

The predicate filters incoming events using an expression. If the expression returns True, the event will enter evaluate section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in include or filters folder (this one is a global folder located at the top hierarchy of the LogMan.io library), use !INCLUDE statement:

!INCLUDE predicate_filter

where predicate_filter is the name of the file plus .yaml extension. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section evaluate

The evaluate section specifies primary key, resolution and other attributes that are applied on the incoming event. The evaluate function is to add the event into the two dimensional structure, defined by a time and a primary key.

Item dimension

Specifies simple or compound primary key (or dimension) for the event. The dimension is defined by names of the input event fields.

Example of the simple primary key:

evaluate:
  dimension: CustomerName

Example of the compound primary key:

evaluate:
  dimension: [CustomerName, DestinationAddress, DestinationHostname]

If exactly one dimension like DestinationHostname is a list in the original event and the correlation should happen for each one of the dimension values, the dimension should be wrapped in [ ]:

evaluate:
  dimension: [CustomerName, DestinationAddress, [DestinationHostname] ]

Item by

Specified the name of the field of the input event that contains a date/time information, which will be used for evaluation.

Item event_count (optional)

Name of the attribute, that specifies the count for correlation within one event, hence influencing the "sum of events" in analysis. Defaults to 1.

Item resolution (optional)

Specifies the resolution of the time aggregation of the correlator. The unit is seconds.

evaluate:
  resolution: 3600  # 1 hour

Default value: 3600

Item saturation (optional)

Specifies the duration of the 'silent' time interval after the trigger is fired. It is specific for the dimension. The unit is resolution.

Default value: 3

Section analyze (optional)

The section analyze contains the configuration of the time window that is applied on the input events. The result of the time window analysis is subjected to the configurable test. When the test is successful (aka returns True), the trigger is fired.

Note: The section is optional, the default behavior is to fire the trigger when there is at least one event in the tumbling of the span equals 2.

Item when (optional)

Specifies when the analysis of the events in the windows should happen.

Options:

  • event (default): Analysis happens after an event comes and is evaluated, usually useful for match and arithmetic correlation
  • periodic/...: Analysis happens after a specified interval in seconds, such as periodic/10 (every 10 seconds), periodic/1h (every 3600 seconds / one hour) etc. Usually useful for UEBA evaluation.

Periodic analysis requires the time window resolution and span to be set properly, so the analysis does not happen too often.

Item window (optional)

Specifies what kind of time window to use.

Options:

  • tumbling: Fixed span (duration), non-overlapping, gap-less contiguous time intervals
  • hopping: Fixed span (duration), overlapping windows contiguous time intervals

Default value: hopping

Item span

Specifies the width of the window. The unit is resolution.

Item aggregate (optional)

Specifies what aggregation functions to be applied on events in the window.

Aggegate functions

Default value: sum

Example of the unique count:

analyze:
  window: hopping
  aggregate: unique count
  dimension: SourceAddress
  span: 6
  test:
    !GE
    - !ARG
    - 5

Trigger when 5 and more unique Source Addresses are observed.

Item test (optional)

The test is an expression that is applied on the output of the aggregate calculation. If the expression returns True, the trigger will be fired if a dimension is not already saturated. If the expression returns False, then no action is taken.

Other returned values are undefined.

Section trigger

The trigger section specifies what kinds of actions to be taken when the trigger is fired by test in the analyze section. See correlator triggers chapter for details.