Skip to content


The Receiver archive is immutable, column-oriented, append-only data storage of the received raw logs.

Each commlink feeds data into the stream. The stream is a infinite table with fields. The stream name is composed by the received. prefix, the name of the tenant and a commlink (ie. received.mytenant.udp-8889)

The archive stream contains following fields for each log entry:

  • raw: Raw log (string, digitally signed)
  • row_id: Primary identifier of the row unique across all streams (64bit unsigned integer)
  • collected_at: Date&time of the log collection at the collector
  • received_at: Date&time of the log receival to the receiver
  • source: Description of the log source (string)

The source field contains:

  • for TCP inputs: <ip address> <port> S (S is for a stream)
  • for UDP inputs: <ip address> <port> D (D is for a datagram)
  • for file inputs: a filename
  • for other inputs: optional specification of the source

The source field for a log delivered over UDP 61562 D

The log was collected from IP address and port UDP/61562.


Every stream is divided into partitions. Partitions of the same stream can be located on different receiver instances.


Partitions can share identical periods of time. This means that data entries from the same span of time could be found in more than one partition.

Each partition has its number (part_no), starting from 0. This number monotonically increases for new partitions in the archive, across streams. The partition number is globally unique, in terms of the cluster.

The partition number is encoded into the partition name. The partition name is 6 character name, which starts with aaaaaa (aka partition #0) and continues to aaaaab (partition #1) and so on.

The parititon can be investigated in the Zookeeper:


partno: 0  # The partition number, translates to aaaaaa
count: 4307  # Number of rows in this partition
size: 142138  # Size of the partition in bytes (uncompressed)

  iso: '2023-07-01T15:22:53.265267'
  unix_ms: 1688224973265267

  iso: '2023-07-01T15:22:53.283168'
  unix_ms: 1688224973283167

  address: 49542  # Address of the collector
  identity: ABCDEF1234567890  # Identity of the collector
  stream: udp-8889
  tenant: mytenant

    type: string

        iso: '2023-06-29T20:33:18.220173'
        unix_ms: 1688070798220173
        iso: '2023-06-29T18:25:03.363870'
        unix_ms: 1688063103363870
    type: timestamp

        iso: '2023-06-29T20:33:18.549359'
        unix_ms: 1688070798549359
        iso: '2023-06-29T18:25:03.433202'
        unix_ms: 1688063103433202
    type: timestamp

        count: 2
    type: token:rle


Because the partition name is globally unique, it is possible to move partition to a shared storage, ie. NAS or a cloud storage from a different nodes of the cluster. The lifecycle is designed in a way that partition names will not collide, so data will not be overwritten by different receivers but reassembled correctly on the "shared" storage.


The partition lifecycle is defined by phases.

The ingest partitions are partitions that receives the data. Once the ingest is completed, aka rotated to the new partition, the former partition is closed. The partition cannot be reopen.

When the partition is closed, the partition lifecycle starts. Each phase is configured to point to a specific directory on the filesystem.

The lifecycle is defined on the stream level, at /lmio/receiver/db/received... entry in the ZooKeeper.


Partitions can be also moved manualy into a desired phase by the API call.

Default lifecycle

The default lifecycle consists of three phases: hot, warm and cold.

graph LR
  I(Ingest) --> H[Hot];
  H --1 week--> W[Warm];
  W --6 months--> D(Delete);
  H --immediately-->C[Cold];
  C --18 months--> CD(Delete);

The ingest is done into the hot phase. Once the ingest is completed and the partition is closed, the partition is copied into the cold phase. After a week, the partition is moved to the warm phase. It means that the partition is duplicated - one copy is in the cold phase storage, the second copy is in the warm phase storage.

The partition on the warm phase storage is deleted after 6 months.

The partition on the cold phase storage is compressed using xz/LZMA. The partition is deleted from the cold phase after 18 months.

Default lifecycle definition

  # Ingest into the hot phase
  phase: hot
  rotate_size: 30G
  rotate_time: daily


    # After one week, move the partition to warm phase
      age: 1w
      phase: warm

    # Copy the partition into the cold phase immediately after closing
      phase: cold

    # Delete the partition after 6 months
      age: 6M

    # Compress the partition immediatelly
        type: xz
      preset: 6
      threads: 4

    # Delete the partition after 18 months
      age: 18M

The phase storage tiers recommendations:

  • Hot phase should be located on SSDs
  • Warm phase should be located on HDDs
  • Cold phase is an archive, could be located on NAS or slow HDDs.


For more information, visit the Administration manual, chapter about Disk storage.

Lifecycle actions

  • move: Move the partition at specified age to the specified phase.
  • copy: Copy the partition at specified age to the specified phase.
  • delete: Delete the partition at specified age.

The age can be e.g. "3h" (three hours), "5M" (five months), "1y" (one year) and so on.

Supported age postfixes:

  • y: year, respectively 365 days
  • M: month, respectively 31 days
  • w: week
  • d: day
  • h: hour
  • m: minute


If age is not specified, then the action is taken immediately.


compress: Compress the data on receival to the phase.

Currently only type: xz is supported with following options:

preset: The xz compression preset.

The compression preset levels can be categorised roughly into three categories:

0 ... 2

Fast presets with relatively low memory usage. 1 and 2 should give compression speed and ratios comparable to bzip2 1 and bzip2 9, respectively.

3 ... 5

Good compression ratio with low to medium memory usage. These are significantly slower than levels 0-2.

6 ... 9

Excellent compression with medium to high memory usage. These are also slower than the lower preset levels.

The default is 6.

Unless you want to maximize the compression ratio, you probably don't want a higher preset level than 7 due to speed and memory usage.

threads: Maximum number of CPU threads used for a compression.

The default is 1.

Set to 0 to use as many threads as there are processor cores.


A partition rotation is a mechanism that closed ingest partitions at specific conditions. When the ingest partition is closed, new data are stored in the newly created another ingest partition. This ensures more or less even slicing of the infinite stream of the data.

The rotation is configured on the stream level by:

  • rotate_time: the period (ie daily) the partition can be in the ingest mode
  • rotate_size: the maximum size of the partition; T, G, M and k postfixes are supported using base 10.

Both options can be applied simultanously.

The default stream rotation is daily and 30G.


Only daily option is available at the moment for rotate_time.

Data vending

The data can be extracted from the archive (ie. for third party processing, migration and so one) by copying out the data directory of partitions in scope.

Use Zookeeper to identify what partitions are in scope of the vending and where they are physically located on storages.

The raw column can be directly processed by third party tools. When the data are compressed by the lifecycle configuration, the decompression can be needed.


It means that you don't need to move partition from ie. cold phase into warm or hot phase.

Replay of the data

The archived logs can be replayed to subsequent central components.


The archive is a cryptographically secured, designed for traceability and non-repudiation. Digital signatures are used to verify the authenticity and integrity of the data, providing assurance that the logs have not been tampered with and were indeed generated by the stated log source.

This digital signature-based approach to maintaining logs is an essential aspect of secure logging practices and a cornerstone of a robust information security management system. These logs are vital tools for forensic analysis during an incident response, detecting anomalies or malicious activities, auditing, and regulatory compliance.

We use following cryptographical algorithms to ensure the security of logs: SHA256, ECDSA.

The hash function, SHA256, is applied to each raw log entry. This function takes the input raw log entry and produces a fixed-size string of bytes. The output (or hash) is unique to the input data; a slight alteration in the input will produce a dramatically different output, a characteristic known as the "avalanche effect".

This unique hash is then signed using a private signing key through the ECDSA algorithm, which generates a digital signature that is unique to both the data and the key. This digital signature is stored alongside the raw log data, certifying that the log data originated from the specified log source and has not been tampered with during storage.

Digital signatures of raw columns are stored in the ZooKeeper (the canonical location) and in the filesystem, under the filename col-raw.sig. Each partition is also equipped with a unique SSL signing certificate, named signing-cert.der. This certificate, in conjunction with the digital signature, can be used to verify that the (the original raw logs) has not been altered, thus ensuring data integrity.


Please note that the associated private signing key is not stored anywhere but in the process memory for security purposes. The private key is removed as soon as the partition has finished its data ingest.

The signing certificate is issued by an internal Certificate Authority (CA). The CA's certificate is available in ZooKeeper at /lmio/receiver/ca/cert.der.

Digital signature verification

You can verify the digital signature by using the following OpenSSL commands:

$ openssl x509 -inform der -in signing-cert.der -pubkey -noout > signing-publickey.pem
$ openssl dgst -sha256 -verify signing-publickey.pem -signature col-raw.sig
Verified OK

These commands extract the public key from the certificate (signing-cert.der), and then use that public key to verify the signature (col-raw.sig) against the data file ( If the data file matches the signature, you'll see a Verified OK message.

Additionally, verify also the signing-cert.der, this certificate has to be issued by the internal CA.

Practical example

The practical example of archive applied on the log stream from Microsoft 365. The "cold" phase is stored on NAS, mounted to /data/nas with XZ compression enabled.


  • Date range: 3 months
  • Rotation: daily (typically one partition is created per day)
  • Total size: 8.3M compressed, compression ratio: 92%
  • Total file count: 1062

Content of directories

tladmin@lm01:/data/nas/receiver/received.default.o365-01$ ls -l
total 0
drwxr-x--- Jul 25 20:59 aaaebd.part
drwxr-x--- Jul 25 21:02 aaaebe.part
drwxr-x--- Jul 26 21:02 aaaebg.part
drwxr-x--- Jul 27 21:03 aaaeph.part
drwxr-x--- Jul 28 21:03 aaagaf.part
drwxr-x--- Jul 29 21:04 aaagfn.part
drwxr-x--- Jul 30 21:05 aaagjm.part
drwxr-x--- Jul 31 21:05 aaagog.part
drwxr-x--- Aug  1 21:05 aaahik.part
drwxr-x--- Aug  2 21:05 aaahmb.part
drwxr-x--- Aug  3 12:49 aaaifj.part
drwxr-x--- Aug  3 17:50 aaaima.part
drwxr-x--- Aug  3 18:46 aaaiok.part
drwxr-x--- Aug  4 18:46 aaajaf.part
drwxr-x--- Aug  5 18:46 aaajbk.part
drwxr-x--- Aug  6 18:47 aaajcj.part
drwxr-x--- Aug  7 11:33 aaajde.part
drwxr-x--- Aug  7 11:34 aaajeg.part
drwxr-x--- Aug  7 12:22 aaajeh.part
drwxr-x--- Aug  7 13:51 aaajem.part
drwxr-x--- Aug  8 09:50 aaajen.part
drwxr-x--- Aug  8 09:59 aaajfk.part
drwxr-x--- Aug  8 10:06 aaajfo.part
drwxr-x--- Oct 25 15:44 aadcne.part
drwxr-x--- Oct 26 06:23 aadcnp.part
drwxr-x--- Oct 26 09:54 aadcof.part
drwxr-x--- Oct 27 09:54 aadcpc.part
tladmin@lm01:/data/nas/receiver/received.default.o365-01/aadcpc.part$ ls -l
total 104
-r--------  1824 Oct 27 09:54
-r-------- 66892 Oct 27 09:54
-r--------  2076 Oct 27 09:54 col-raw.pos.xz
-r--------    72 Oct 27 09:54 col-raw.sig
-r--------  1864 Oct 27 09:54
-r--------    32 Oct 27 09:54
-r--------    68 Oct 27 09:54 col-source-token.pos.xz
-r--------    68 Oct 27 09:54
-r--------   496 Oct 27 09:54 signing-cert.der.xz
-r--------  1299 Oct 27 09:54 summary.yaml