Writing a window correlation-type detection rule

A window correlation rule is a highly versatile type of detection that can identify combinations of events over time. This example shows some of the techniques you can use when writing window correlations, but there are many more options, so this page gives you additional guidance.

Before you can write a new detection rule, you need to:

Decide what activity you are looking for, and decide the timeframe in which this activity happening is notable.
Identify which data source produces the logs that could trigger a positive detection, and identify what information those logs contain.
Decide what you want to happen when the activity is detected.

Use TeskaLabs SP-Lang to write correlation rules.

Sections of a correlaton rule

Include each of these sections in your rule:

Define: Information that describes your rule.
Predicate: The predicate section is a filter that identifies which logs to evaluate, and which logs to ignore.
Evaluate: The evaluate section sorts or organizes data to be analyzed.
Analyze: The analyze section defines and searches for the desired pattern in the data sorted by evaluate.
Trigger: The trigger section defines what happens if there is a positive detection.

To better understand the structure of a window correlation rule, consult this example.

Comments

Include comments in your detection rules so that you and others can understand what each item in the detection rule does. Add comments on separate lines from code, and begin comments with hashtags #.

Parentheses

Words in parentheses () are placeholders to show that there would normally be a value in this space. Correlation rules don't use parentheses.

Define

Always include in define:

Item in the rule	How to include
`name: "(name)"`	Name the rule. While the name has no impact on the rule's functionality, it should still be a name that's clear and easy for you and others to understand.
`description: "(description)"`	Describe the rule briefly and accurately. The description also has no impact on the rule's functionality, but it can help you and others understand what the rule is for.
`type: correlator/window`	Include this line as-is. The `type` does impact the rule's functionality. The rule uses `correlator/window` to function as a window correlator.

Predicate

The predicate section is a filter. When you write the predicate, you use SP-Lang expressions to structure conditions for the filter "allow in" only logs that are relevant to the activity or pattern that the rule is detecting.

If a log meets the predicate's conditions, it gets analyzed in the next steps of the detection rule, alongside other related logs. If a log doesn't meet the predicate's conditions, the detection rule ignores the log.

See this guide to learn more about writing predicates.

Evaluate

Any log that passes through the filter in predicate gets evaluated in evaluate. The evaluate section organizes the data so it can by analyzed. Usually, you can't spot a security threat (or other noteworthy patterns) based on just one event (for example, one failed login attempt), so you need to write detection rules to group events together to find patterns that point to security or operational issues.

The evaluate section creates an invisible evaluation window - you can think of the window as a table. The table is what the analyze section uses to detect the activity the detection rule is seeking.

You can see an example of the evaluate and analyze sections working together here.

Item in `evaluate`	How to include
`evaluate: dimension: [(field1), (field2)]`	`dimension` creates the rows in the table. In the table, the values of the specified fields are grouped into one row (see the table below).
`by: "@timestamp"`	`by` creates the columns in the table. In most cases, `@timestamp` is the right choice because window correlation rules are based around time. So, each column in the table is an interval of time, which the `resolution` specifies.
`resolution: (integer)`	The `resolution` unit is seconds. Each time interval will be the number of seconds you specify.
`saturation: 1`	The `saturation` field sets how many times the trigger can be activated before the rule stops counting events in a single cell that caused the trigger (see the table below). With a recommended saturation of 1, relevant events that happen within the same specified timeframe (`resolution`) will stop being counted after one trigger. Setting the saturation to 1 prevents additional triggers for identical behavior in the same timeframe.

Analyze

analyze uses the table created by the evaluate section to find out if the activity the detection rule is seeking has happened.

You can see an example of the evaluate and analyze sections working together here.

Item in `analyze`	How to include
`analyze: window: (hopping or tumbling)`	The window analyzes a specified number of cells in the table created by `evaluate` section, each of which represents logs in a specified timeframe. Hopping window: The window will count the values in cells, testing all adjacent combinations of cells to cover the specified time period, with overlap. A hopping window is recommended. Tumbling window: The window counts the values in cells, testing all adjacent combinations of cells to cover the specified time period, WITHOUT overlap. See the note below to learn more about hopping and tumbling windows.
`aggregate: unique count dimension: (field)`	The `aggregate` depends on the `dimension`. Use `unique count` to ensure that the rule won't count the same value of your specified field in `dimension` more than once.
`span: (integer)`	A span sets the number of cells in the table that will be analyzed at once. `span` multiplied by `resolution` is the timeframe in which the correlation rule looks for a pattern or behavior. (For example, 2*60 is a 2-minute timeframe.)
`test: !GE - !ARG VALUE - (integer)`	The `!GE` expression means "greater than or equal to," and `!ARG VALUE` refers to the output value of the `aggregate` function. The value listed under `!ARG VALUE` is the number of unique occurances of a value in a single analysis window that will trigger the correlation rule.

Hopping vs. tumbling windows

This page about tumbling and hopping windows can help you understand the different types of analysis windows.

Trigger

After identifying the suspicious activity you specified, the rule can:

Send the detection to Elasicsearch as a document. Then, you can see the detection as a log in TeskaLabs LogMan.io. You can create your own dashboard to display correlation rule detections, or find the logs in Discover.
Send a notification via email

Visit the triggers page to learn about setting up triggers to create events, and go to the notifications page to learn about sending messages from detections.