Skip to content

Receiver internals

Architecture

Architecture of the receiver

MessagePack used for log/event delivery both in instant and syncback delivery.

The CommLink is also used for a bi-directionaly delivery of JSON messages, such as collector metrics that are shipped into a InfluxDB at the central cluster.

Archive structure

  • Stream: identified by Stream Name
    • Partition: identified by Partition Number
      • Row: identified by Row Number respectively by Row Id
        • Column: identified by the name.

The stream is an infinite data structure divided into partitions. The partition is created and populated through the ingesting of rows. Once the partition reaches a specific size or age, it is rotated, which means the partition is closed, and a new ingesting partition is created. Once the partition is closed, it cannot be reopened and remains read-only for the rest of its lifecycle.

The partition contains rows that are further divided into columns. Column structure is fixed in one partition but can be different in other partitions in the same stream.

Archive filesystem

The stream respective partition data are stored on the filesystem in the directory that is specified by the lifecycle phase.

Content of the /data/ssd/receiver (hot phase)

+ received.mytenant.udp-8889
  + aaaaaa.part
    + summary.yaml
    + signing-cert.der
    + col-raw.data
    + col-raw.pos
    + col-raw.sig
    + col-collected_at.data
    + col-received_at.data
    + col-source.data
    + col-source-token.data
    + col-source-token.pos
  + aaaaab.part
    + summary.yaml
    + ...
  + aaaaac.part
    + summary.yaml
    + ...
+ received.mytenant.tcp-7781
...

The partition directory aaaaaa.part contains the whole data content of the partition. The structure is the same for every phase of the lifecycle.

The file summary.yaml contains non-canonical information about the partition. The canonical version of the information is in the Zookeeper.

The file signing-cert.der contains the SSL certificate for verification of col-*.sig digital signatures. The signing certificate is unique for the partition.

col-*.data files contain the data for the given field.

Partition Number / part_no

Unsigned 24bit integer, range 0..16,777,216, max. 0xFF_FF_FF.

The partition number is provided by the shared counter in Zookeeper, located at /lmio/receiver/db/part.counter. It means each partition has a unique number regardless of which stream it belongs to.

Reasoning

10[years] * 365[days-in-year] * 24[hours-in-day] * 10[safety] = 876000 partitions (5% of the max.)

The partition number is frequently displayed as a string of 6 characters, such as aaaaaa. It is Base-16 encoded version of this integer, using abcdefghijklmnop characters.

Row Number / row_no

Position within the partition.

Unsigned 40bit unsigned integer 0xFF_FF_FF_FF_FF (range 0..1,099,511,627,775)

Reasoning

1.000.000 EPS per 24 hours * 10[safety] = 860,400,000,000

Row Id / row_id

Global 64bit unique identifier of the row within the stream.

Because the row_id is composed by the part_no, it guarantees its global uniqueness, not just within a single stream, but across all streams.

Calculation

row_id = (part_no << 24) | row_no

 row_id:
+------------------------+--------------------------+
|        part_no         |          row_no          |
+------------------------+--------------------------+

Column type string

The column type string uses two types of files:

  • col-<columne name>.data: The raw byte data of the entries in the column. Each entry is sequentially added to the end of the file, hence making it append-only. As such, entries are stored without any explicit delimiters or record markers. Instead, the location of each entry is tracked in a companion col-<columne name>.pos file.

  • col-<columne name>.pos: The starting byte positions of each entry in the corresponding col-<columne name>.data file. Each position is stored as a 32-bit unsigned integer. The positions are stored sequentially in the order the entries are added to the col-<columne name>.data. The Nth integer in the col-<columne name>.pos file indicates the starting position of the Nth entry in the col-<columne name>.data file. The length of the Nth entry is a difference betwenn Nth integer and (N+1)th integer in the col-<columne name>.pos file.

Column type string

Sequential data access

Sequential access of data involves reading data in the order it is stored in the column files.

Below are the steps to sequentially access data:

  1. Open both the col-<column name>.pos and col-<column name>.data files. Read the first 32-bit integer from col-<column name>.pos. Initialize a current position with a value of 0.

  2. Read the next 32-bit integer, referred to here as the position value, from col-<column name>.pos. Calculate the length of the data entry by subtracting the current position from the position value. After that, update the current position to the newly read position value.

  3. Read the data from col-<column name>.data using the length calculated in step 2. This length of data corresponds to the actual content of the database row.

  4. Repeat steps 2 and 3 to read subsequent rows. Continue this process until you reach the end of the col-<column name>.pos file, which would indicate that all data rows have been read.

Random data access

To access the Nth row in a column, you would follow these steps:

  1. Seek the col-<column name>.pos file to the (N-1)th position or to 0 if N == 0. Each position corresponds to a 32-bit integer, so the Nth position corresponds to the Nth 32-bit integer. For example, to get to the 6th row, you would seek to the 5th integer (20 bytes from the start, because each integer is 4 bytes).

  2. Read one or two 32-bit integers from the col-<column name>.pos file. If N == 0, read only one integer, the position is assumed to be 0. The first integer indicates the start position of the desired entry in the col-<column name>.data file. For N > 0, read two integers. The second integer indicates the start position of the next entry. The difference between the second and first integers gives the length of the desired entry.

  3. Seek to the position in the col-<column name>.data file pointed to by the first integer read in the previous step.

  4. Read the entry from the col-<column name>.data file using the calculated length from step 2.

Column type timestamp

The column type timestamp uses one type of file: col-<column name>.data. Each entry in this column is a 64-bit Unix timestamp representing a date and time in microsecond precision.

Info

The Unix timestamp is a way to track time as a running total of seconds that have elapsed since 1970-01-01 00:00:00 UTC, not counting leap seconds.
The microsecond precision allows tracking time even more accurately.

The timestamp column summarizes the minumum and maximum timestamp in each partition. The summary is is in the Zookeeper and in the summary.yaml file on the filesystem.

Sequential Data Access

Sequential access to a timestamp column involves reading each timestamp in the order they're stored:

  1. Open the col-<column name>.data file.
  2. Read a 64-bit integer from the col-<column name>.data file. This integer is your Unix timestamp in microseconds.
  3. Repeat step 2 until you reach the end of the file, which means you've read all timestamps.
  4. Close the file.

Random Data Access

To access a timestamp at a specific row (Nth position):

  1. Seek to the Nth position in the col-<column name>.data file. As each timestamp is a 64-bit (or 8-byte) integer, to get to the Nth timestamp, you would seek to the N * 8th byte.
  2. Read a 64-bit integer from the col-<column name>.data file. This is your Unix timestamp in microseconds for the Nth row.

Column type token

A column of the type token is designed to store string data in an optimized format. It is particularly suited for scenarios where a column contains a relatively small set of distinct, repetitive values.

Instead of directly storing the actual string data, the token column type encodes each string into an integer. This encoding process is accomplished via an index that is constructed based on all unique string values within the column partition. Each unique string is assigned a unique integer identifier in this index, and these identifiers replace the original strings in the token column.

The index itself is represented by the position of the string in a pair of associated files: col-<column name>-token.data and col-<column name>-token.pos. See string column type for more details.

This approach provides significant storage space savings and boosts query efficiency for columns with a limited set of frequently repeated values. Moreover, it allows faster comparison operations as integer comparisons are typically quicker than string comparisons.

Danger

Please note that this approach may not yield benefits if the number of unique string values is large or if the string values are mostly unique. The overhead of maintaining the index could outweigh the storage and performance advantages of using compact integer storage.

The column type token uses three types of files:

  • col-<column name>.data: The index of the column values, each represented as a 16-bit unsigned integer.
  • col-<column name>-token.data & col-<column name>-token.pos: The index, using the same structure as the string column type. The position of a string in these files represents the encoded value of the string in the token column.

Sequential Data Access

Sequential access of a token column involves reading each token in the order they are stored. This is accomplished using the indices stored in the col-<column name>.data file and translating them into the actual string values using the col-<column name>-token.data and col-<column name>-token.pos files.

Here are the steps to sequentially access data:

  1. Open the col-<column name>.data file. This file contains 16-bit unsigned integers that serve as indices into the token string list.
  2. Read a 16-bit unsigned integer from the col-<column name>.data file. This is the index of your token string.
  3. Apply the "Random data access" from string column type on ol-<column name>-token.data and col-<column name>-token.pos files to fetch the string value.
  4. Repeat steps 2 and 3 until you reach the end of the col-.data file, which indicates that all tokens have been read.
  5. Close all files.

Random Data Access

Random access in a token column allows you to retrieve any entry without needing to traverse the previous entries. This can be particularly beneficial in scenarios where you only need to fetch certain specific entries and not the entire data set.

To access a token at a specific row (Nth position):

  1. Seek to the Nth position in the col-<column name>.data file. As each index entry is a 16-bit (or 2-byte) integer, to get to the Nth entry, you would seek to the N * 2nd byte.
  2. Read a 16-bit integer from the col-<column name>.data file. This is your index entry for the Nth row.
  3. Apply the "Random data access" from string column type on col-<column name>-token.data and col-<column name>-token.pos files to fetch the string value.

Column type token:rle

A column of the type token:rle extends the token column type by adding Run-Length Encoding (RLE) to further optimize storage. This type is particularly suitable for columns that have many sequences of repeated values.

Like the token type, token:rle encodes string values into integer tokens. However, instead of storing each of these integer tokens separately, it utilizes Run-Length Encoding to compress sequences of repeated tokens into a pair of values: the token itself and the number of times it repeats consecutively.

The RLE compression is applied to the col-<column name>.data file, turning a sequence of identical token indices into a pair: (token index, repeat count).

This approach provides significant storage savings for columns where values repeat in long sequences. It also allows faster data access and query execution by reducing the amount of data to be read from disk.

Danger

Keep in mind that this approach will not yield benefits if the data doesn't contain long sequences of repeated values. In fact, it may lead to increased storage usage and slower query execution as the overhead of maintaining and processing RLE pairs might outweigh the compression benefits.

The token:rle column type uses the same three types of files as the token type:

  • col-<column name>.data: RLE compressed indices of the column values, each entry as a pair: (16-bit unsigned integer token index, 16-bit unsigned integer repeat count).

  • col-<column name>-token.data & col-<column name>-token.pos: Te index, using the same structure as the string column type. The position of a string in these files represents the encoded value of the string in the token column.

Sequential data access

Sequential access of a token:rle column involves reading each RLE-compressed token pair in the order they are stored. This is accomplished using the indices stored in the col-<column name>.data file and translating them into actual string values using col-<column name>-token.data and col-<column name>-token.pos files.

Here are the steps to sequentially access data:

  1. Open the col-<column name>.data file. This file contains pairs of 16-bit unsigned integers representing the token index and the run length.
  2. Read a pair of 16-bit unsigned integers from the col-<column name>.data file. The first integer is the index of your token string, and the second integer indicates how many times this token repeats consecutively (run length).
  3. Use the token index to locate the string in the col-<column name>-token.data and col-<column name>-token.pos files, following the process described for the string column type (Random Data Access).
  4. Repeat the value from step 3 as many times as indicated by the run length.
  5. Repeat steps 2 to 4 until you reach the end of the col-<column name>.data file, which indicates that all tokens have been read.
  6. Close all files.

Random data access

To access a token at a specific row (Nth position):

  1. Open the col-<column name>.data file. This file contains pairs of 16-bit unsigned integers that serve as indices into the token string list and their corresponding run lengths.
  2. Traverse the col-<column name>.data file pair by pair, summing the run lengths until the sum equals or exceeds N. The pair at which this occurs corresponds to the token that you are looking for.
  3. Read the 16-bit integer token index from this pair.
  4. Apply the "Random data access" from string column type on col-<column name>-token.data and col-<column name>-token.pos files to fetch the string value.