LogMan.io Correlator

TeskaLabs LogMan.io Correlator is a powerful, fast, scalable component of LogMan.io and TeskaLabs SIEM. As the Correlator makes detections possible, it is essential to effective cybersecurity.

The Correlator identifies specified activity, patterns, anomalies, and threats in real time as defined by detection rules. The Correlator works in your system's data stream, rather than on disk storage, making it a fast and uniquely scalable security mechanism.

What does the Correlator do?

The Correlator keeps track of events and when they happen in relation to a larger pattern or activity.

First, you identify the pattern, threat, or anomaly you want the Correlator to monitor for. You write a detection that defines the activity, including which types of events (logs) are relevant and how many times an event needs to occur in a defined timeframe in order to trigger a response.
The Correlator identifies the relevant incoming events, and organizes the events first by a specific attribute in the event (dimension), such as source IP address or user ID, then sorts the events into short time intervals so the number of events can be analyzed. The time intervals are also defined by the detection rule.

Note: It's most common to use the Correlator's sum function to count events that occur in a specified timeframe. However, the Correlator can also analyze using other mathematical functions.
The Correlator analyzes these dimensions and time intervals to see if the relevant events have happened in the desired timeframe. When the Correlator detects the activity, it triggers the response specified in the detection.

In other words, this microservice shares event statuses over time intervals and uses a sliding, or rolling, analysis window.

What is a sliding analysis window?

Using a sliding analysis window means that the Correlator can analyze multiple time intervals continuously. For example, when analyzing a period of 30 seconds, the Correlator shifts its analysis, which is a window of 30 seconds, to overlap previous analyses as time progresses.

This picture represents a single dimension, for example the analysis of events with the same source IP address. In a real detection rule, you'd have several rows of this table, one row for each IP address. More in the example below.

The sliding window makes it possible to analyze the overlapping 30-second timeframes 0:00-0:30, 0:10-0:40, 0:20-0:50, and 0:30-0:60, rather than just 0:00-0:30 and 0:30-0:60.

Example

Example scenario: You create a detection to alert you when 20 login attempts are made to the same user account within 30 seconds. Since this password entry rate is higher than most people could achieve on their own, this activity could indicate a brute force attack.

In order to detect this security threat, the Correlator needs to know two things:

Which events are relevant. In this case, that means failed login attempts to the same user account.
When the events (login attempts) happen in relation to each other.

Note: The following logs and images are heavily simplified to better illustrate the ideas.

1. These logs occur in the system:

What do these logs mean?

Each table you see above is a log for the event of a user having a single failed login attempt.

log.ID: The unique log identifier, as seen in the table below
timestamp: The time the event occurred
username: The Correlator will analyze groups of logs from the same users, because it wouldn't be effective in this case to analyze login attempts across all users combined.
event.message: The Correlator is only looking for failed logins, as would be defined by the detection rule.

2. The Correlator begins tracking the events in rows and columns:

Username is the dimension, as defined by the detection rule, so each user has their own row.
Log ID (A, B, C, etc.) is here in the table so you can see which logs are being counted.
The number in each cell is how many events occurred in that time interval per username (dimension).

3. The Correlator continues keeping track of events:

You can see that one account is experiencing a higher volume of failed login attempts now.

4. At the same time, the Correlator is analyzing 30-second time periods with an analysis window:

The analysis window moves across the time intervals to count the total number of events in 30-second timeframes. You can see that when the analysis window reaches the 01:20-01:50 timeframe for the username anna.s.ample, it will count more than 20 events. This would trigger a response from the Correlator, as defined by the detection (more on triggers here).

A gif to illustrate the analysis window moving

The 30-second analysis window "slides" or "rolls" along the time intervals, counting how many events occurred. When it finds 20 or more events in a single analysis, an action from the detection rule is triggered.

Memory and storage

The Correlator operates in the data stream, not in a database. This means that the Correlator is tracking events and performing analysis in real time as events occur, rather than pulling past collected events from a database to perform analysis.

In order to work in the data stream, the Correlator uses memory mapping, which allows it to function in the system's quickly accessible memory (RAM) rather than relying on disk storage.

Memory mapping provides significant benefits:

Real-time detection: Data in RAM is more quickly accessible than data from a storage disk. This makes the Correlator very fast, allowing you to detect threats immediately.
Simultaneous processing: Greater processing capacity allows for the Correlator to run many parallel detecions at once.
Scalability: The volume of data in your log collection system will likely increase as your organization grows. The Correlator can keep up. Allocating additional RAM is faster and simpler than increasing disk storage.
Persistence: If the system shuts down unexpectedly, the Correlator does not lose data. The Correlator's history is backed up to disk (SSD) often, so the data is available when the system restarts.

For more technical information, visit our Correlator reference documentation.