Skip to content

Forwarding to Kafka

If the Apache Kafka is configured in the configuration, every received log event is forwarded by LogMan.io Receiver to Kafka topic. The topic is created automatically, when the first message is forwarded.

The name of the Apache Kafka topic is derived from the stream name: received.<tenant>.<stream>, it is the same as the name in the archive.

Example of Kafka topics

received.mytenant.udp-8889
received.mytenant.tcp-7781

The raw log event is sent in the Kafka message body.

Following information is added to a message header:

  • row_id: Row Id, the globally unique identifier of the event in the archive. Not present if the archive is disabled. 64bit binary big endian unsigned integer.
  • collected_at: The event collection date&time, Unix timestamp in microseconds as a string.
  • received_at: The event receival date&time, Unix timestamp in microseconds as a string.
  • source: The event source, string.
  • tenant: The name of the tenant which received this message, string. (OBSOLETE)

Roadmap

Automated setup of the Kafka topic (such as number of Kafka partitions) will be implemented in the future releases.