Each commlink feeds data into the stream.
The stream is a infinite table with fields.
The stream name is composed by the
received. prefix, the name of the tenant and a commlink (ie.
The archive stream contains following fields for each log entry:
raw: Raw log (string, digitally signed)
row_id: Primary identifier of the row unique across all streams (64bit unsigned integer)
collected_at: Date&time of the log collection at the collector
received_at: Date&time of the log receival to the receiver
source: Description of the log source (string)
source field contains:
- for TCP inputs:
<ip address> <port> S(S is for a stream)
- for UDP inputs:
<ip address> <port> D(D is for a datagram)
- for file inputs: a filename
- for other inputs: optional specification of the source
source field for a log delivered over UDP
192.168.100.1 61562 D
The log was collected from IP address 192.168.100.1 and port UDP/61562.
Every stream is divided into partitions. Partitions of the same stream can be located on different receiver instances.
Partitions can share identical periods of time. This means that data entries from the same span of time could be found in more than one partition.
Each partition has its number (
part_no), starting from 0.
This number monotonically increases for new partitions in the archive, across streams.
The partition number is globally unique, in terms of the cluster.
The partition number is encoded into the partition name.
The partition name is 6 character name, which starts with
aaaaaa (aka partition #0) and continues to
aaaaab (partition #1) and so on.
The parititon can be investigated in the Zookeeper:
partno: 0 # The partition number, translates to aaaaaa count: 4307 # Number of rows in this partition size: 142138 # Size of the partition in bytes (uncompressed) created_at: iso: '2023-07-01T15:22:53.265267' unix_ms: 1688224973265267 closed_at: iso: '2023-07-01T15:22:53.283168' unix_ms: 1688224973283167 extra: address: 192.168.100.1 49542 # Address of the collector identity: ABCDEF1234567890 # Identity of the collector stream: udp-8889 tenant: mytenant columns: raw: type: string collected_at: summary: max: iso: '2023-06-29T20:33:18.220173' unix_ms: 1688070798220173 min: iso: '2023-06-29T18:25:03.363870' unix_ms: 1688063103363870 type: timestamp received_at: summary: max: iso: '2023-06-29T20:33:18.549359' unix_ms: 1688070798549359 min: iso: '2023-06-29T18:25:03.433202' unix_ms: 1688063103433202 type: timestamp source: summary: token: count: 2 type: token:rle
Because the partition name is globally unique, it is possible to move partition to a shared storage, ie. NAS or a cloud storage from a different nodes of the cluster. The lifecycle is designed in a way that partition names will not collide, so data will not be overwritten by different receivers but reassembled correctly on the "shared" storage.
The partition lifecycle is defined by phases.
The ingest partitions are partitions that receives the data. Once the ingest is completed, aka rotated to the new partition, the former partition is closed. The partition cannot be reopen.
When the partition is closed, the partition lifecycle starts. Each phase is configured to point to a specific directory on the filesystem.
The lifecycle is defined on the stream level, at
/lmio/receiver/db/received... entry in the ZooKeeper.
Partitions can be also moved manualy into a desired phase by the API call.
The default lifecycle consists of three phases: hot, warm and cold.
graph LR I(Ingest) --> H[Hot]; H --1 week--> W[Warm]; W --6 months--> D(Delete); H --immediately-->C[Cold]; C --18 months--> CD(Delete);
The ingest is done into the hot phase. Once the ingest is completed and the partition is closed, the partition is copied into the cold phase. After a week, the partition is moved to the warm phase. It means that the partition is duplicated - one copy is in the cold phase storage, the second copy is in the warm phase storage.
The partition on the warm phase storage is deleted after 6 months.
The partition on the cold phase storage is compressed using xz/LZMA. The partition is deleted from the cold phase after 18 months.
Default lifecycle definition
ingest: # Ingest into the hot phase phase: hot rotate_size: 30G rotate_time: daily lifecycle: hot: # After one week, move the partition to warm phase move: age: 1w phase: warm # Copy the partition into the cold phase immediately after closing copy: phase: cold warm: # Delete the partition after 6 months delete: age: 6M cold: # Compress the partition immediatelly compress: type: xz preset: 6 threads: 4 # Delete the partition after 18 months delete: age: 18M
The phase storage tiers recommendations:
- Hot phase should be located on SSDs
- Warm phase should be located on HDDs
- Cold phase is an archive, could be located on NAS or slow HDDs.
For more information, visit the Administration manual, chapter about Disk storage.
move: Move the partition at specified
ageto the specified
copy: Copy the partition at specified
ageto the specified
delete: Delete the partition at specified
age can be e.g. "3h" (three hours), "5M" (five months), "1y" (one year) and so on.
y: year, respectively 365 days
M: month, respectively 31 days
age is not specified, then the action is taken immediately.
compress: Compress the data on receival to the phase.
type: xz is supported with following options:
preset: The xz compression preset.
The compression preset levels can be categorised roughly into three categories:
0 ... 2
Fast presets with relatively low memory usage. 1 and 2 should give compression speed and ratios comparable to bzip2 1 and bzip2 9, respectively.
3 ... 5
Good compression ratio with low to medium memory usage. These are significantly slower than levels 0-2.
6 ... 9
Excellent compression with medium to high memory usage. These are also slower than the lower preset levels.
The default is 6.
Unless you want to maximize the compression ratio, you probably don't want a higher preset level than 7 due to speed and memory usage.
threads: Maximum number of CPU threads used for a compression.
The default is 1.
Set to 0 to use as many threads as there are processor cores.
A partition rotation is a mechanism that closed ingest partitions at specific conditions. When the ingest partition is closed, new data are stored in the newly created another ingest partition. This ensures more or less even slicing of the infinite stream of the data.
The rotation is configured on the stream level by:
rotate_time: the period (ie
daily) the partition can be in the ingest mode
rotate_size: the maximum size of the partition;
kpostfixes are supported using base 10.
Both options can be applied simultanously.
The default stream rotation is
daily option is available at the moment for
The data can be extracted from the archive (ie. for third party processing, migration and so one) by copying out the data directory of partitions in scope.
Use Zookeeper to identify what partitions are in scope of the vending and where they are physically located on storages.
raw column can be directly processed by third party tools.
When the data are compressed by the lifecycle configuration, the decompression can be needed.
It means that you don't need to move partition from ie. cold phase into warm or hot phase.
Replay of the data
The archived logs can be replayed to subsequent central components.
The archive is a cryptographically secured, designed for traceability and non-repudiation. Digital signatures are used to verify the authenticity and integrity of the data, providing assurance that the logs have not been tampered with and were indeed generated by the stated log source.
This digital signature-based approach to maintaining logs is an essential aspect of secure logging practices and a cornerstone of a robust information security management system. These logs are vital tools for forensic analysis during an incident response, detecting anomalies or malicious activities, auditing, and regulatory compliance.
The hash function, SHA256, is applied to each raw log entry. This function takes the input raw log entry and produces a fixed-size string of bytes. The output (or hash) is unique to the input data; a slight alteration in the input will produce a dramatically different output, a characteristic known as the "avalanche effect".
This unique hash is then signed using a private signing key through the ECDSA algorithm, which generates a digital signature that is unique to both the data and the key. This digital signature is stored alongside the raw log data, certifying that the log data originated from the specified log source and has not been tampered with during storage.
Digital signatures of
raw columns are stored in the ZooKeeper (the canonical location) and in the filesystem, under the filename
Each partition is also equipped with a unique SSL signing certificate, named
This certificate, in conjunction with the digital signature, can be used to verify that the
col-raw.data (the original raw logs) has not been altered, thus ensuring data integrity.
Please note that the associated private signing key is not stored anywhere but in the process memory for security purposes. The private key is removed as soon as the partition has finished its data ingest.
The signing certificate is issued by an internal Certificate Authority (CA).
The CA's certificate is available in ZooKeeper at
Digital signature verification
You can verify the digital signature by using the following OpenSSL commands:
$ openssl x509 -inform der -in signing-cert.der -pubkey -noout > signing-publickey.pem $ openssl dgst -sha256 -verify signing-publickey.pem -signature col-raw.sig col-raw.data Verified OK
These commands extract the public key from the certificate (
signing-cert.der), and then use that public key to verify the signature (
col-raw.sig) against the data file (
col-raw.data). If the data file matches the signature, you'll see a
Verified OK message.
Additionally, verify also the
signing-cert.der, this certificate has to be issued by the internal CA.
The practical example of archive applied on the log stream from Microsoft 365.
The "cold" phase is stored on NAS, mounted to
/data/nas with XZ compression enabled.
- Date range: 3 months
- Rotation: daily (typically one partition is created per day)
- Total size: 8.3M compressed, compression ratio: 92%
- Total file count: 1062
Content of directories
tladmin@lm01:/data/nas/receiver/received.default.o365-01$ ls -l total 0 drwxr-x--- Jul 25 20:59 aaaebd.part drwxr-x--- Jul 25 21:02 aaaebe.part drwxr-x--- Jul 26 21:02 aaaebg.part drwxr-x--- Jul 27 21:03 aaaeph.part drwxr-x--- Jul 28 21:03 aaagaf.part drwxr-x--- Jul 29 21:04 aaagfn.part drwxr-x--- Jul 30 21:05 aaagjm.part drwxr-x--- Jul 31 21:05 aaagog.part drwxr-x--- Aug 1 21:05 aaahik.part drwxr-x--- Aug 2 21:05 aaahmb.part drwxr-x--- Aug 3 12:49 aaaifj.part drwxr-x--- Aug 3 17:50 aaaima.part drwxr-x--- Aug 3 18:46 aaaiok.part drwxr-x--- Aug 4 18:46 aaajaf.part drwxr-x--- Aug 5 18:46 aaajbk.part drwxr-x--- Aug 6 18:47 aaajcj.part drwxr-x--- Aug 7 11:33 aaajde.part drwxr-x--- Aug 7 11:34 aaajeg.part drwxr-x--- Aug 7 12:22 aaajeh.part drwxr-x--- Aug 7 13:51 aaajem.part drwxr-x--- Aug 8 09:50 aaajen.part drwxr-x--- Aug 8 09:59 aaajfk.part drwxr-x--- Aug 8 10:06 aaajfo.part .... drwxr-x--- Oct 25 15:44 aadcne.part drwxr-x--- Oct 26 06:23 aadcnp.part drwxr-x--- Oct 26 09:54 aadcof.part drwxr-x--- Oct 27 09:54 aadcpc.part
tladmin@lm01:/data/nas/receiver/received.default.o365-01/aadcpc.part$ ls -l total 104 -r-------- 1824 Oct 27 09:54 col-collected_at.data.xz -r-------- 66892 Oct 27 09:54 col-raw.data.xz -r-------- 2076 Oct 27 09:54 col-raw.pos.xz -r-------- 72 Oct 27 09:54 col-raw.sig -r-------- 1864 Oct 27 09:54 col-received_at.data.xz -r-------- 32 Oct 27 09:54 col-source-token.data.xz -r-------- 68 Oct 27 09:54 col-source-token.pos.xz -r-------- 68 Oct 27 09:54 col-source.data.xz -r-------- 496 Oct 27 09:54 signing-cert.der.xz -r-------- 1299 Oct 27 09:54 summary.yaml