Skip to content

Window Correlator

Window correlator detects incoming events based on the predicate section and stores them in data structures based on the evaluate section. If then the test rule in analyze section is matched, trigger section is called.

The following sample correlation detects more than or equal to 5 error connections between two IP addresses:

Sample

---
define:
    name: "Network T1046 Network Service Discovery"
    description: "Detects more than or equal to 5 error connections between two IP addresses"
    type: correlator/window

logsource:
    type: "Network"

mitre:
    technique: "T1046"
    tactic: "TA0007"

predicate:
    !OR
    - !EQ
        - !ITEM EVENT log.level
        - "error"
    - !EQ
        - !ITEM EVENT log.level
        - "critical"
    - !EQ
        - !ITEM EVENT log.level
        - "emergency"

evaluate:
    dimension: [source.ip, destination.ip]  
    by: "@timestamp"
    resolution: 60

analyze:
    window: hopping
    aggregate: sum
    span: 10
    test:
        !GE
        - !ARG
        - 5

trigger:
    - event:
            threat.indicator.confidence: "Medium"
            threat.indicator.ip: !ITEM EVENT source.ip
            threat.indicator.port: !ITEM EVENT source.port
            threat.indicator.type: "ipv4-addr"

Section define

This section contains the common definition and meta data.

Item name

Shorter human-readable name of this declaration.

Item type

The type of this declaration, must be correlator/window.

Item description (optional)

Longed, possibly multiline, human-readable description of the declaration.

Section logsource

Specifies the types of event lanes that should the incoming events be read from.

Section predicate

The predicate filters incoming events using an expression. If the expression returns True, the event will enter evaluate section. If the expression returns False, then the event is skipped.

Other returned values are undefined.

Include of nested predicate filters

Predicate filters are expressions located in a dedicated file, that can be included in many different predicates as their parts.

If you want to include an external predicate filter, located either in the library, use !INCLUDE statement:

!INCLUDE /predicate_filter.yaml

where /predicate_filter is the path of the file in the library. The content of predicate_filter.yaml is an expression to be included, like:

---
!EQ
- !ITEM EVENT category
- "MyEventCategory"

Section evaluate

The evaluate section specifies primary key, resolution and other attributes that are applied on the incoming event. The evaluate function is to add the event into the two dimensional structure, defined by a time and a primary key.

Item dimension

Specifies simple or compound primary key (or dimension) for the event. The dimension is defined by names of the input event fields.

Example of the simple primary key:

evaluate:
    dimension: [source.ip]

Note

Tenant is added automatically to the dimension list.

Example of the compound primary key:

evaluate:
    dimension: [source.ip, destination.ip]

If exactly one dimension like DestinationHostname is a list in the original event and the correlation should happen for each one of the dimension values, the dimension should be wrapped in [ ]:

evaluate:
    dimension: [source.ip, destination.ip, [DestinationHostname] ]

Item by

Specified the name of the field of the input event that contains a date/time information, which will be used for evaluation. Default is: @timestamp.

Item event_count (optional)

Name of the attribute, that specifies the count for correlation within one event, hence influencing the "sum of events" in analysis. Defaults to 1.

Item resolution

Specifies the resolution of the time aggregation of the correlator. The unit is seconds.

evaluate:
    resolution: 3600  # 1 hour

Default value: 3600

Item saturation (optional)

Specifies the duration of the silent time interval after the trigger is fired. It is specific for the dimension. The unit is resolution.

Default value: 3

Section analyze (optional)

The section analyze contains the configuration of the time window that is applied on the input events. The result of the time window analysis is subjected to the configurable test. When the test is successful (aka returns True), the trigger is fired.

Note: The section is optional, the default behavior is to fire the trigger when there is at least one event in the tumbling of the span equals 2.

Item when (optional)

Specifies when the analysis of the events in the windows should happen.

Options:

  • event (default): Analysis happens after an event comes and is evaluated, usually useful for match and arithmetic correlation
  • periodic/...: Analysis happens after a specified interval in seconds, such as periodic/10 (every 10 seconds), periodic/1h (every 3600 seconds / one hour) etc. Usually useful for UEBA evaluation.

Periodic analysis requires the time window resolution and span to be set properly, so the analysis does not happen too often.

Item window (optional)

Specifies what kind of time window to use.

Options:

  • tumbling: Fixed span (duration), non-overlapping, gap-less contiguous time intervals
  • hopping: Fixed span (duration), overlapping windows contiguous time intervals

Default value: hopping

Item span

Specifies the width of the window. The unit is resolution.

Item aggregate (optional)

Specifies what aggregation functions to be applied on events in the window.

Aggegate functions

Default value: sum

Example of the unique count:

analyze:
    window: hopping
    aggregate: unique count
    dimension: client.ip
    span: 6
    test:
        !GE
        - !ARG
        - 5

Trigger when 5 and more unique client.ips are observed.

Item test (optional)

The test is an expression that is applied on the output of the aggregate calculation. If the expression returns True, the trigger will be fired if a dimension is not already saturated. If the expression returns False, then no action is taken.

Other returned values are undefined.

Section trigger

The trigger section specifies what kinds of actions to be taken when the trigger is fired by test in the analyze section. See correlator triggers chapter for details.