Declarations for defining baselines

The declarations for baselines are loaded from the Library from the folder specified in the configuration, such as /Baseliners.

Note

The Baseliner uses /Schemas/ECS.yaml by default, so /Schemas/ECS.yaml must also be present in the Library.

Declaration

This is an example of a baseline definition, located in the /Baseliners folder in the Library:

---
define:
    name: Dataset
    description: Creates baseline for each dataset and trigger alarms if the actual number deviates
    type: baseliner

baseline:
    region: Czech Republic
    period: day
    learning: 4
    classes: [workdays, weekends, holidays]
    max_history_days: 180  # (optional) Maximum days of counter history to retain
    model: negbin

evaluate:
    key: event.dataset
    timestamp: "@timestamp"

analyze:
    when: 1h  # How often to perform the analysis (1 hour is the default)
    test:
        !AND
        - !LT
            - !ARG VALUE
            - 1
        - !GT
            - !ARG MEDIAN
            - 0
        - !GT
            - !ARG SIGMA
            - 5.0

trigger:
    - event:
            # Threat description
            # https://www.elastic.co/guide/en/ecs/master/ecs-threat.html
            threat.framework: "MITRE ATT&CK"
            threat.indicator.sightings: !ITEM EVENT value
            threat.indicator.confidence: "High"
            threat.indicator.name: !ITEM EVENT dimension

    - notification:
            type: email
            to: ["myemail@example.co"]
            template: "/Templates/Email/Notification_baseliner_dimension.md"

            variables:
                name: "Logs are not coming to the dataset within the given UTC hour."
                dimension: !ITEM EVENT dimension
                hour: !ITEM EVENT hour

Sections

baseline

baseline: # (1)
    region: Czech Republic #(2)
    period: day # (3)
    learning: 4  # (4)
    classes: [workdays, weekends, holidays] # (5)
    max_history_days: 180  # (6)
    model: normal  # (7)

1. Defines how the given baseline is built.
2. Defines in which region the activity is happening (for calculating holidays and so on).
3. Defines the timespan for the baseline. The period can be either day or week.
4. Defines the number of periods (here day) that occur from the baseliner beginning to receive input until the user can see the baseline analysis. Additional details below.
5. Define which days in the week we want to monitor. Classes can include any/all: workdays, weekends, and holidays.
6. max_history_days (optional): Maximum number of days of counter history to retain for baseline building. Counter records older than this limit are automatically deleted to prevent memory leaks and excessive storage usage. If not specified, defaults to the configuration value max_baseline_history_days (default: 60 days if not configured). This parameter helps manage long-term data retention while ensuring sufficient history for accurate baseline calculations. Optional max_history_days_per_class overrides the window per temporal class (see below).
7. model (optional): Statistical model used when scoring the current hour against the learned mean and variance. One of normal (default), poisson, or negbin (Negative Binomial with automatic Poisson fallback). See Statistical model.

learning
The learning field defines the learning phase.
The learning phase is the time from the first occurrence of the dimension value in the input of the Baseliner instance until the point when the baseline is shown to the user and the analysis takes place. In the declaration, learning is the number of periods. The learning phase is calculated separately for holidays, weekends and working days. Baselines are rebuilt overnight (housekeeping).

In this example, the period is day, so learning is 4 days. Considering the calendar, a learning phase of 4 days beginning on Friday means 4 working days, and thus ends on Wednesday night.

max_history_days
The max_history_days field defines the maximum number of days of counter history to retain for baseline building. This parameter helps prevent memory leaks and excessive storage usage by automatically deleting counter records older than the specified limit.

• Default behavior: If not specified in the baseline declaration, the system uses the configuration value max_baseline_history_days from the baseliner configuration (default: 60 days if not configured).
• Automatic cleanup: During baseline building (typically overnight), counter records older than max_history_days are automatically deleted from MongoDB.
• Impact on baseline: Only counter records within the max_history_days window are used for baseline calculations. Older data is excluded to keep the baseline relevant to recent activity patterns.
• Recommendation: Set this value based on your baseline requirements. For example:
• 60-90 days: Suitable for most use cases, provides good balance between history and performance
• 180 days (6 months): For baselines that need longer-term patterns, such as seasonal variations
• 30 days: For high-volume environments where storage is a concern

Storage Management

The max_history_days parameter is particularly important for high-volume baselines or long-running systems. Without this limit, counter records can accumulate indefinitely, leading to increased storage usage and slower baseline calculations.

max_history_days_per_class (optional)
Override the history window for individual temporal classes. workdays and weekends default to max_history_days. holidays defaults to 400 days (~13 months) so a full public-holiday year fits when the global window is 60 days. Counter retention uses the longest configured window.

baseline:
    max_history_days: 60
    # optional override; holidays already default to 400:
    # max_history_days_per_class:
    #   holidays: 400

merge_holidays_into_weekends (optional, default false)
When true, calendar public holidays are classified as weekends for baseline building and analysis. Use this when holidays are too rare for a separate class within the history window (common with max_history_days: 60). Prefer classes: [workdays, weekends] together with this flag.

baseline:
    classes: [workdays, weekends]
    merge_holidays_into_weekends: true

Statistical model

baseline.model chooses one probability model for analyzing the current hour. Only that model’s scores are computed.

Choosing a model

Model	Recommended use	Suitable examples
normal (default)	Approximately symmetric continuous values, especially signed measurements, not concentrated near a hard boundary	Measurement error or residual, clock offset, temperature deviation from a set point, frequency or voltage deviation from nominal
poisson	Event counts with relatively stable variation around their mean	Regular scheduled activity, stable transaction counts, periodic status events
negbin	Event counts that are bursty or more variable than Poisson can represent	Log volumes, authentication events, network events

Guidance: use normal for approximately symmetric continuous measurements. Use poisson or negbin for event counts. If you are modelling raw event counts and are unsure whether Poisson is appropriate, use negbin: it is a Negative Binomial count model with automatic Poisson fallback for profiles that do not show sufficient overdispersion.

Do not use normal for raw event counts. A normal model can sometimes approximate bounded or skewed data away from a boundary, but counts near zero are better handled by poisson or negbin.

baseline:
    region: Czech Republic
    period: week
    learning: 2
    classes: [workdays, weekends, holidays]
    model: negbin

Understanding the analysis outputs

Configuration (baseline.*) chooses how the baseline is built and which model scores the current hour. The attributes below are computed values available in analyze.test.

Attribute	Kind	Meaning
VALUE	Observation	Current activity for this analysis window (event count, or vector norm when aggregation: vector)
MEAN, STDEV, VARIANCE, MEDIAN, MIN, MAX, SUM	Learned history	Summary statistics of comparable past hours for the same class
SIGMA	Model score	Magnitude of the anomaly on a common normal-equivalent rarity scale (see below)
DIRECTION	Model score	Whether the observation is unusually low, high, or neither (see below)
LOG_P0	Model score	Natural log of P(X ≤ 0) under the configured model (see below)
MODEL	Context	Configured model name (normal, poisson, or negbin)
HOUR, KEY, CLASS, TENANT	Context	Slot being analyzed

SIGMA

SIGMA expresses how rare the observation is under the configured model, on a common normal-equivalent scale. A larger value means a rarer observation.

• With model: normal, it is the conventional absolute z-score |VALUE − MEAN| / STDEV.
• With model: poisson or model: negbin, the probability of the observed count is converted to the equivalent one-sided normal z-score.

This keeps existing SIGMA thresholds usable across models, even though the underlying probability model differs.

The one-sided tail probability describes observations at least as extreme in one specified direction. For a direction-independent rule, the probability of reaching the same rarity in either tail is approximately twice the one-sided probability:

SIGMA	One-sided tail probability	Either-tail probability	Approximate frequency with hourly evaluation
3	1.35×10⁻³	2.70×10⁻³	once per 370 hours (~twice per month)
4	3.17×10⁻⁵	6.33×10⁻⁵	once per 15,800 hours (~once per 1.8 years)
5	2.87×10⁻⁷	5.73×10⁻⁷	once per 1.74 million hours (~once per 199 years)

For example, SIGMA = 3 means a one-sided tail probability of about 0.00135. A value at least this unusual in either direction has probability about 0.0027, or roughly one occurrence in 370 evaluations. With hourly evaluation that is about twice per month on average.

These are probability equivalences, not forecasts of a regular schedule. The frequency interpretation assumes the model is well calibrated and that comparable evaluation windows are sufficiently independent. A directional rule such as DIRECTION: LOW uses the one-sided probability; a rule concerned with equally extreme values in either direction uses the either-tail probability.

Start thresholds around 5 when false positives are costly; use 4 when earlier detection matters more. Raise the threshold if alerts are too noisy.

DIRECTION

• LOW — the observed value is unusually low
• HIGH — the observed value is unusually high
• NEUTRAL — the observation is not clearly associated with either side of the distribution

SIGMA describes the magnitude of the anomaly; DIRECTION describes whether it is on the low or high side of the baseline. That makes it easy to write rules that distinguish silence from traffic spikes.

MODEL

MODEL reports the configured statistical model. With model: negbin, an individual baseline profile may internally use the Poisson fallback when overdispersion is not present.

LOG_P0

LOG_P0 = ln(P0) where P0 = P(X ≤ 0) under the configured model.

• For event-count models (poisson, negbin), this is the probability of observing zero events.
• For normal, it is the probability of falling at or below the zero boundary (for continuous data, P(X ≤ 0) equals P(X < 0)).

More negative values mean that reaching zero is less likely under normal operation. For most detection rules, prefer SIGMA; use LOG_P0 when you need an exact probability threshold or a Bayesian outage posterior.

LOG_P0	P0	Equivalent SIGMA at zero
−6.61	≈ 1.35×10⁻³	3
−10.36	≈ 3.17×10⁻⁵	4
−15.06	≈ 2.87×10⁻⁷	5

For vector baselines and Event Codes on Windows hosts, see Detecting user activity anomalies.

Detecting silence or an outage

The primary pattern uses VALUE, MEDIAN, and SIGMA:

baseline:
    model: negbin

analyze:
    test:
        !AND
        - !LT
            - !ARG VALUE
            - 1
        - !GT
            - !ARG MEDIAN
            - 0
        - !GT
            - !ARG SIGMA
            - 5.0

This triggers when the current hour is silent, comparable historical hours are normally non-empty (MEDIAN > 0), and the silence is exceptionally unusual under the configured count model. SIGMA = 5 is a conservative starting example; lower it (for example to 4) when earlier detection is more important, or raise it when false positives are excessive. Use the rarity table above to interpret the approximate hourly frequency of that choice.

Profiles with MEDIAN == 0 (often silent historically) should usually not auto-alert; the MEDIAN guard above handles that.

Advanced: probability-based outage rules

Baseliner does not compute a Bayesian outage posterior. LOG_P0 describes how surprising silence is under healthy operation. A posterior also depends on how common outages are and how reliably an outage produces complete silence.

Let:

• O — the monitored source is in an outage state
• H — it is operating normally
• q = P(O) — prior probability that a comparable evaluation window is affected by an outage
• r = P(X=0 | O) — probability of observing zero events during an outage
• p0 = P(X=0 | H) — zero-event probability under the configured healthy-state model (exp(LOG_P0) for count models)

After observing zero events:

P(O | X=0) = (q · r) / (q · r + (1 − q) · p0)

The prior q should represent outage prevalence per comparable analysis window, not an arbitrary confidence setting. For a complete source failure, r = 1 may be reasonable; for partial or intermittent failures, r < 1 is more realistic.

For a desired posterior threshold a, the condition converts to a threshold on the healthy-state zero probability:

P(O | X=0) ≥ a  ⇔  p0 ≤ (q · r · (1 − a)) / (a · (1 − q))

LOG_P0 ≤ ln( (q · r · (1 − a)) / (a · (1 − q)) )

Example. Assume q = 0.001, complete silence during outage (r = 1), and required posterior a = 0.95. Then p0 ≤ 5.2684×10⁻⁵ and LOG_P0 ≤ −9.8512. At VALUE = 0 this is approximately SIGMA ≥ 3.878. The practical rule can use the more approachable SIGMA interface:

analyze:
    test:
        !AND
        - !EQ
            - !ARG VALUE
            - 0
        - !GE
            - !ARG SIGMA
            - 3.878

The threshold is valid only for the stated assumptions. If q or r changes, recalculate. At zero, SIGMA, LOG_P0, and the posterior are monotonic transformations of the same model evidence for fixed q and r — not three independent signals.

logsource (optional)

The logsource section specifies the types of event lanes that incoming events should be read from. It is used for categorization of the log source and is derived from Sigma rules.

logsource:
  vendor:
    - microsoft
  product:
    - windows

The logsource section can contain:

• vendor: The vendor of the log source (e.g., microsoft, linux, cisco)
• product: The product name (e.g., windows, exchange, fortigate)
• service (optional): The service name (e.g., syslog, audit, activitylogs)

Each field accepts a list of values. Events from event lanes that match any of the specified vendor, product, or service values will be considered for the baseline.

Example: Windows Events Baseline

logsource:
  vendor:
    - microsoft
  product:
    - windows

This configuration will process events from event lanes that are categorized as Microsoft Windows logs.

Example: Multiple Products

logsource:
  vendor:
    - microsoft
  product:
    - windows
    - exchange
    - office365

This configuration will process events from Microsoft Windows, Exchange, or Office 365 event lanes.

signal (optional)

The optional signal section controls whether the baseline sends a signal to Alert Management and how tickets are grouped. Use signal: default: false when the baseline is for analysis or correlation input only (no direct ticket creation). Use signal: grouping: to set which attributes are used for ticket grouping. See Signal for details.

predicate (optional)

The predicate section filters incoming events to be considered as activity in the baseline.

Write filters with TeskaLabs SP-Lang. Visit Predicates or the SP-Lang documentation for details.

Combining logsource and predicate

The logsource section filters events at the event lane level (which event lanes to read from), while the predicate section filters individual events within those lanes. Both filters are applied, so an event must match both the logsource criteria and the predicate expression to be included in the baseline.

evaluate

This section specifies which attributes from the event are going to be used in the baseline build.

evaluate:
    key: event.dataset # (1)
    timestamp: "@timestamp" # (2)

1. Specifies the attribute/entity to monitor.
2. Specifies the attribute in which the time dimension of the event activity is stored in.

key_aliases (optional, since v25.47.02)

In the evaluate section you can add key_aliases to map dimension values to a single key. This normalizes different event values so they are counted under one dimension key.

evaluate:
    key: myuser
    key_aliases:
        myuser1: myuser
        myuser2: myuser
        myuser3: myuser
    timestamp: "@timestamp"

• An event with myuser: "myuser1" is counted under the key myuser.
• An event with myuser: "myuser2" is also counted under myuser.
• Values not listed in key_aliases are left unchanged (e.g. myuser4 remains myuser4).

Aliases are applied to all dimension value types (string, integer, list after join, IP after conversion): the value is converted to a string and then the key_aliases mapping is applied.

analyze

The test section in analyze specifies when to run the trigger if the actual activity (!ARG VALUE) deviates from the baseline. Write tests in SP-Lang.

During analysis, Baseliner compares the current activity for the dimension (key), hour, and temporal class (workdays, weekends, holidays) against learned statistics for that same slot.

By default (analyze.when: 1h), the watcher runs about once per hour and scores the previous completed UTC hour against the counters stored at that moment. Triggers fire only if analyze.test passes for that snapshot. The hour is not re-analyzed later.

Computed scores (SIGMA, DIRECTION, LOG_P0, MODEL) and model selection are described under Statistical model. The silence / outage pattern using VALUE, MEDIAN, and SIGMA is the recommended starting point for count baselines.

Late events vs. the analysis UI

The baseline values API (and the analysis chart in the UI) recomputes SIGMA for historical hours from counters that exist now. That can disagree with what the watcher saw when the hour was closed.

Example. A host baseline uses model: negbin and !GT !ARG SIGMA 5. Shortly after 11:00 UTC the watcher analyzes 10:00–11:00. If events for that hour are still delayed in the pipeline, the counter is still 0, so SIGMA stays low and no complex event or signal is created. Later the same hour shows actual: 95 and SIGMA: 100 in the UI — the counters arrived after analysis. The watcher does not run again for that hour, so no alert is produced retrospectively.

When investigating a missing alert, compare watcher logs for that hour with whether counters already existed at analysis time; a high SIGMA in the chart alone does not mean a trigger ran.

Example: absolute thresholds

analyze:
    test:
        !AND #(1)
        - !LT #(2)
            - !ARG VALUE # (3)
            - 1
        - !GT # (4)
            - !ARG MEAN # (5)
            - 10.0

1. All expressions nested under AND must be true for the test to pass. Here, if the value is less than 1 and the mean is greater than 10, the trigger is run.
2. "Less than"
3. Get (!ARG) the value (VALUE). If the value is less than 1 as specified, the !LT expression is true.
4. "Greater than"
5. Get (!ARG) the mean (MEAN). If the mean is greater than 10.0 as specified, the !GT expression is true.

Example: sigma threshold (spike or silence)

baseline:
    model: negbin  # or normal / poisson

analyze:
    when: 1h
    test:
        !GT
        - !ARG SIGMA
        - 5

Combine with DIRECTION when you only care about one side (for example spikes):

analyze:
    test:
        !AND
        - !EQ
            - !ARG DIRECTION
            - HIGH
        - !GT
            - !ARG SIGMA
            - 5

Silence / dataset-down example (same as in Detecting silence or an outage):

baseline:
    model: negbin

analyze:
    test:
        !AND
        - !LT
            - !ARG VALUE
            - 1
        - !GT
            - !ARG MEDIAN
            - 0
        - !GT
            - !ARG SIGMA
            - 5.0

Available !ARG attributes

The following attributes are available in SP-Lang notation inside analyze.test:

TENANT: "str",
VALUE: "ui64",
STDEV: "fp64",
MEAN: "fp64",
SIGMA: "fp64",
DIRECTION: "str",
LOG_P0: "fp64",
MODEL: "str",
MEDIAN: "fp64",
VARIANCE: "fp64",
MIN: "ui64",
MAX: "ui64",
SUM: "ui64",
HOUR: "ui64",
KEY: "str",
CLASS: "str",

The when option defines how often to perform the analysis.

trigger

The trigger section defines the activity that is triggered to run after a successful analysis. (More about triggers.)

Baseliner creates events

Upon every analysis (every hour), Baseliner creates an event to summarize its analysis. These Baseliner-created events are available to use (as EVENT) with expressions such as !ARG and !ITEM, meaning you can pull values from the events for your trigger activities.

These Baseliner-created events include the fields:

tenant
The name of the tenant the baseline belogs to.

dimension
The dimension the baseline belongs to, as specified in evaluate.

class
The class the baseline was calculated from.
Options include: workdays, weekends, and holidays

hour
The number of the UTC hour the analysis happend in.

value
The value of the current counter of events for the given UTC hour.

Analysis scores on the same event (usable via !ITEM EVENT …) include mean, stdev, sigma, direction, log_p0, and model.

Notification trigger

A notification trigger sends a message, such as an email. See Email notifications for more details about sending email notifications and using email templates.

An example of a notification trigger:

trigger:
    - notification:
            type: email #(1)
            to: ["myemail@example.co"] # (2)
            template: "/Templates/Email/Notification_baseliner_dimension.md" # (3)

            variables: # (4)
                name: "Logs are not coming to the dataset within the given UTC hour."
                dimension: !ITEM EVENT dimension # (5)
                hour: !ITEM EVENT hour

1. Specifies an email notification
2. Recipient address
3. Filepath to the email template in the LogMan.io Library
4. Begins the section that gives directions for how to fill the blank fields from the email template. The blank fields in the template being used in this example are name, dimension, and hour.
5. Uses SP-Lang to get information (!ITEM) from the Baseliner-created EVENT (detailed below). In this case, the template field dimension will be filled with the value of dimension taken from the Baseliner-created event.

Event trigger

You can use an event trigger to create a log or event, which you'll be able to see in the TeskaLabs LogMan.io UI.

Example of an event trigger:

    - event: # (1)
            threat.framework: "MITRE ATT&CK"
            threat.indicator.sightings: !ITEM EVENT value
            threat.indicator.confidence: "High"
            threat.indicator.name: !ITEM EVENT dimension

1. This new event is a threat description using threat fields from Elasticsearch

Analysis in UI

By default, the LogMan.io UI provides displays of analyses for user and host.

Specify the analysis in the schema (default: /Schemas/ECS.yaml) like this:

  host.id:
    type: "str"
    analysis: host

  user.id:
    type: "str"
    analysis: user

  ...

If then the tenant is configured to use this schema (ECS by default), the host.id and user.id fields in Discover will show a link to the given baseline.

Analysis host uses the baseline named Host by default:

---
define:
  name: Host

Analysis user uses the baseline named User by default:

---
define:
  name: User

If a specific analysis cannot locate its associated baseline, the UI will display an empty screen for that analysis.

Note

Both baselines needed for analysis are distributed as part of the LogMan.io Common Library.