Skip to content

Collector

Collector provisioning

The collector instance needs to be provisioned prior the collector is authorized to send logs to the TeskaLabs LogMan.io. The provisioning is done exactly once during the collector life cycle.

Note

TeskaLabs LogMan.io Receiver operates Certificate Authority. The provisioning process is the approval of the CSR finished by the issuance of the client SSL certificate for the collector. This client certificate is used by the collector for its authentication.

The provisioning starts at the collector. The minimal collector YAML configuration specifies the URL of the LogMan.io entry endpoint for commlinks.

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

When the collector is started, it submits its enrolment request to the receiver. The collector also prints output similar to this one

...
Waiting for an approval, the identity: 'ABCDEF1234567890'
Waiting for an approval, the identity: 'ABCDEF1234567890'

It means that the collector has an unique identity ABCDEF1234567890 and that the receiver awaits an approval of this collector.

On the receiver side, the approval is granted by a following call:

curl -X 'PUT' \
  http://lmio-receiver/provision/ABCDEF1234567890 \
  -H 'Content-Type: application/json' \
  -d '{"tenant": "mytenant"}'

Warning

Speficy a correct tenant in the request, instead of mytenant value.

Hint

Approval can be granted also using web browser at the "Approve a CSR received from the collector" at http://lmio-receiver/doc

Mind that ABCDEF1234567890 needs to be replaced by the real identity from the output of the collector. The tenant has to be specified in the request as well.

When this call is executed, the collector informs that it is provisioned and ready:

Waiting for an approval, the identity: 'ABCDEF1234567890'
29-Jun-2023 02:05:35.276253 NOTICE lmiocollector.commlink.identity.service The certificate received!
29-Jun-2023 02:05:35.277731 NOTICE lmiocollector.commlink.identity.service [sd identity="ABCDEF1234567890"] Ready.
29-Jun-2023 02:05:35.436872 NOTICE lmiocollector.commlink.service [sd url="https://recv.logman.example.com/commlink/v2301"] Connected.

Certificates of provisioned clients are stored in the ZooKeeper at /lmio/receiver/clients.

Info

The tenant name is stored in the generated SSL client certificate.

Roadmap

The provisioning will be avaialble from the Web UI too.

Revoke the collector enrollment

The revovation of the collector enrollment means a removal of the grant of the collector to connect to a receiver.

For revoking the enrolment at the receiver side, delete the relevant entry from a ZooKeeper folder /lmio/receiver/clients.

Warning

The deletion will not affect currently connected collectors. The automated disconnection is on the product roadmap.

For revoking the enrolment at the colletor side, delete ssl-cert.pem and ssl-key.pem when the collector is stopped. The collector will start new enrollment under a new identity when started.

Collector configuration

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

input:..:LogSource1:
  output: udp-8889

output:CommLink:udp-8889: {}

...

Section connection:CommLink:commlink:

This section configures a communication link to the central part of the TeskaLabs LogMan.io.

Option url

Mandatory value with URL of the central part of LogMan.io. It must use https:// protocol, not http://.

Option insecure

Optional (default: no) boolean value that allows insecure server connections if set to yes. This option allows a use of self-signed server SSL certificates.

Danger

Don't use insecure option in the production setups.

Advanced SSL configuration options

The following configuration options specify the SSL (HTTPS) connection:

  • cert: Path to the client SSL certificate
  • key: Path to the private key of the client SSL certificate
  • password: Private key file password (optional, default: none)
  • cafile: Path to a PEM file with CA certificate(s) to verify the SSL server (optional, default: none)
  • capath: Path to a directory with CA certificate(s) to verify the SSL server (optional, default: none)
  • ciphers: SSL ciphers (optional, default: none)
  • dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)
  • verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional); for more information, see: github.com/TeskaLabs/asab

Section output:CommLink:<stream>:

<stream> The stream name in the archive and in the Apache Kafka topics.

{} means at the end that there are no options for this output.

Note

Output generic options for output: applies as well.

Multiple sources

The collector can handle multiple log sources (event lanes) from the one instance. For each source, add input:.. and output:CommLink:... section to the configuration.

Example

connection:CommLink:commlink:
  url: https://recv.logman.example.com/

# First log source
input:..:LogSource1:
  output: udp-8888

output:CommLink:udp-8888: {}

# Second log source
input:..:LogSource2:
  output: udp-8889

output:CommLink:udp-8889:
  debug: yes

Warning

Log sources collected by a one instance of the collector must share one tenant.

Collector reconnection and caching

When the collector is offline, logs are stored in the collector cache and uploaded to the receiver as soon as the connectivity is restored. The cached log upload is done via HTTP PUT request.

The websocket connection is restored too.

Note

Websocket connection utilizes server-side generated PING packets to keep the websocket open.