Skip to content

Collector

Collector provisioning

The collector instance needs to be provisioned prior the collector is authorized to send logs to the TeskaLabs LogMan.io. The provisioning is done exactly once during the collector life cycle.

Note

TeskaLabs LogMan.io Receiver operates Certificate Authority. The provisioning process is the approval of the CSR finished by the issuance of the client SSL certificate for the collector. This client certificate is used by the collector for its authentication.

The provisioning starts at the collector. The minimal collector YAML configuration specifies the URL of the LogMan.io entry endpoint for commlinks.

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

When the collector is started, it submits its enrolment request to the receiver. The collector also prints output similar to this one

...
Waiting for an approval, the identity: 'ABCDEF1234567890'
Waiting for an approval, the identity: 'ABCDEF1234567890'

It means that the collector has an unique identity ABCDEF1234567890 and that the receiver awaits an approval of this collector.

On the receiver side, the approval is granted by a following call:

curl -X 'PUT' \
  http://lmio-receiver/provision/ABCDEF1234567890 \
  -H 'Content-Type: application/json' \
  -d '{"tenant": "mytenant"}'

Warning

Speficy a correct tenant in the request, instead of mytenant value.

Hint

Approval can be granted also using web browser at the "Approve a CSR received from the collector" at http://lmio-receiver/doc

Mind that ABCDEF1234567890 needs to be replaced by the real identity from the output of the collector. The tenant has to be specified in the request as well.

When this call is executed, the collector informs that it is provisioned and ready:

Waiting for an approval, the identity: 'ABCDEF1234567890'
29-Jun-2023 02:05:35.276253 NOTICE lmiocollector.commlink.identity.service The certificate received!
29-Jun-2023 02:05:35.277731 NOTICE lmiocollector.commlink.identity.service [sd identity="ABCDEF1234567890"] Ready.
29-Jun-2023 02:05:35.436872 NOTICE lmiocollector.commlink.service [sd url="https://recv.logman.example.com/commlink/v2301"] Connected.

Certificates of provisioned clients are stored in the ZooKeeper at /lmio/receiver/clients.

Info

The tenant name is stored in the generated SSL client certificate.

CSRs that are not provisioned within 2 days are removed. The provisioning procedure can be restarted once the collector submits a new CSR.

Removing the collector

For removal of the provisioned collector at the receiver side, delete the relevant entry from a ZooKeeper folder /lmio/receiver/clients. This means that you revoked a grant of the collector to connect to a receiver.

Warning

The deletion will not affect currently connected collectors. The automated disconnection is on the product roadmap.

For removing at the collector side, delete ssl-cert.pem and ssl-key.pem when the collector is stopped. The collector will start new enrollment under a new identity when started. This action is called a reset of the collector identity.

Collector configuration

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

input:..:LogSource1:
  output: logsource-1

output:CommLink:logsource-1: {}

...

Section connection:CommLink:commlink:

This section configures a communication link to the central part of the TeskaLabs LogMan.io.

The configuration can be also provided by the application configuration file. If the section [commlink] is present, items from there are loaded before applying values from YAML.

Example

Empty YAML specification:

connection:CommLink:commlink: {}
...

URL is used from the application configuration:

[commlink]
url=https://recv.logman.example.com/
...

Option url

Mandatory value with URL of the central part of LogMan.io. It must use https:// protocol, not http://.

Typical values are:

  • https://recv.logman.example.com/ - for a dedicated NGINX server for receiving logs
  • https://logman.example.com/lmio-receiver - for a single DNS domain on NGINX server

Can be also provided in the environment variable LMIO_COMMLINK_URL.

Option insecure

Optional (default: no) boolean value that allows insecure server connections if set to yes. This option allows a use of self-signed server SSL certificates.

Danger

Don't use insecure option in the production setups.

Advanced SSL configuration options

The following configuration options specify the SSL (HTTPS) connection:

  • cert: Path to the client SSL certificate
  • key: Path to the private key of the client SSL certificate
  • password: Private key file password (optional, default: none)
  • cafile: Path to a PEM file with CA certificate(s) to verify the SSL server (optional, default: none)
  • capath: Path to a directory with CA certificate(s) to verify the SSL server (optional, default: none)
  • ciphers: SSL ciphers (optional, default: none)
  • dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)
  • verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional); for more information, see: github.com/TeskaLabs/asab

Section output:CommLink:<stream>:

<stream> The stream name in the archive and in the Apache Kafka topics.

Logs will be fed into the stream name received.<tenant>.<stream>.

{} means at the end that there are no options for this output.

Note

Generic options for output: applies as well. Such as debug: true for a troubleshooting.

Multiple sources

The collector can handle multiple log sources (event lanes) from the one instance. For each source, add input:.. and output:CommLink:... section to the configuration.

Example

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

# First (TCP) log source
input:Stream:LogSource1:
  address: 8888  # Listen on TCP/8888
  output: tcp-8888

output:CommLink:tcp-8888: {}

# Second (UDP) log source
input:Datagram:LogSource2:
  address: 8889  # Listen on UDP/8889
  output: udp-8889

output:CommLink:udp-8889: {}

# Third (UDP + TCP) log source
input:Stream:LogSource3s:
  address: 8890  # Listen on TCP/8890
  output: p-8890

input:Datagram:LogSource3d:
  address: 8890  # Listen on UDP/8890
  output: p-8890

output:CommLink:p-8890: {}

Warning

Log sources collected by a one instance of the collector must share one tenant.

Delivery methods

When a collector is online, logs and other events are deliverted instantly over the Websocket.

When a collector is offline, logs are stored in the offline buffer and once the collector become online, buffered logs are synced back. This delivery method is called syncback. Buffered logs are uploaded using HTTP PUT request.

Offline buffer

When the collector is not connected to a receiver, logs are stored in the collector local buffer and uploaded to the receiver as soon as the connectivity is restored.

Buffered logs are compressed using xz when stored in the offline buffer.

The local buffer is a directory on the filesystem, the location of this folder can be configured:

[general]
buffer_dir=/var/lib/lmio-receiver/buffer

Warning

The collector monitors an available disk capacity in this folder and it will stop buffering logs when less than 5% of the disk space is free.

Reconnection during housekeeping

The collector reconnects every day during housekeeping - typically at 4:00 in the morning. This is to restore balanced distribution of connected collectors across the cluster.