Skip to content

Collecting logs using rsyslog

rsyslog is a high-performance, open-source, modular syslog daemon commonly installed on Linux systems, designed to collect, parse, and ship system and application logs. It's a cornerstone of reliable log pipelines because it supports modern protocols and formats, strong transport security (TLS), and robust buffering with disk-assisted queues that prevent data loss during network interruptions. With its flexible rules engine, rsyslog can enrich events, drop noise, tag sources, and fan-out to multiple destinations—making it ideal for forwarding logs to a TeskaLabs LogMan.io at scale.

This page shows how to configure rsyslog to forward logs to the TeskaLabs LogMan.io Collector, both over plain TCP and, for stronger security, over TLS with one-way or mutual (mTLS) authentication.

Setup rsyslog log forwarding

To set up rsyslog log forwarding to the LogMan.io Collector, follow these steps:

  1. Verify connectivity to the LogMan.io Collector using the netcat command:

    nc -vz <Collector IP address or hostname> <port>

    If the connection is successful, you should see a message indicating that the connection was successful (e.g., "Connection to <IP address> <port> port [tcp/*] succeeded!").

  2. Check if the rsyslog service is running:

    sudo systemctl status rsyslog.service

    If the service is not running, start it with:

    sudo systemctl start rsyslog.service

    If rsyslog is not installed, you can install it using the package manager for your Linux distribution.

    sudo apt install rsyslog rsyslog-gnutls

    rsyslog-gnutls is required for TLS/mTLS

    The rsyslog-gnutls package provides the GnuTLS network stream driver (gtls) that is needed for both one-way TLS and mTLS configurations below.

  3. Create the configuration file for rsyslog to forward logs to LogMan.io Collector:

    /etc/rsyslog.d/teskalabs-logman-io.conf
    *.* action(
    type="omfwd"
    protocol="tcp"
    target="<IP address of the collector>"
    port="514"
    KeepAlive="on"
    queue.type="LinkedList"
    queue.size="10000"

    # Set the format of the syslog message to:
    # <PRI>VERSION TIMESTAMP-RFC3339 HOSTNAME APPNAME PROCID MSGID STRUCTURED_DATA MESSAGE
    template="RSYSLOG_SyslogProtocol23Format"
)

    Note that the default smart syslog port at LogMan.io Collector will auto-detect incoming TLS/SSL connection, so the plain TCP configuration above also works when a TLS stream driver is added (see below).

  4. Validate the configuration for syntax errors:

    sudo rsyslogd -N1

  5. Apply the changes by restarting the rsyslog service:

    sudo systemctl restart rsyslog.service

  6. Test the configuration by:

    logger -t rsyslog-test "Hello from $(hostname)"

Enable MARK messages

Enable the MARK module in /etc/rsyslog.conf to emit periodic -- MARK -- messages, which makes it easy to spot silent or disconnected senders.

/etc/rsyslog.conf
module(load="immark")

Note that this works only on Linux distributions that support the MARK module.

Securing the connection with TLS

The LogMan.io Collector's syslog input is TLS-capable and auto-detects incoming TLS connections. There are two levels of security:

  • One-way TLS (server authentication) — the rsyslog client verifies that it talks to the real Collector, but the Collector does not verify who the client is.
  • Mutual TLS (mTLS) — both sides authenticate: the client verifies the Collector, and the Collector verifies the client using a client certificate. This is the recommended setup for production environments.

One-way TLS (server authentication)

This configuration lets the rsyslog client verify the Collector's server certificate against your Certificate Authority (CA).

  1. Verify the TLS/SSL connection to the LogMan.io Collector using the openssl command:

    echo | openssl s_client -connect <IP>:<port> -servername <IP> -CAfile /path/to/ca.crt 2>/dev/null | grep "Verify return code"

    A Verify return code: 0 (ok) confirms the server certificate validates against the provided CA.

  2. Configure rsyslog to use TLS for secure log forwarding to the LogMan.io Collector:

    /etc/rsyslog.d/teskalabs-logman-io.conf
    global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/path/to/ca.crt"  # Replace with the actual path to your CA certificate
)

*.* action(
    type="omfwd"
    protocol="tcp"
    target="<IP address of the collector>"
    port="514"
    KeepAlive="on"
    queue.type="LinkedList"
    queue.size="10000"

    # TLS/SSL options
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="<IP address of the collector>"  # Must match a name/IP in the Collector's server certificate

    # RFC5424 format: <PRI>VERSION TIMESTAMP-RFC3339 HOSTNAME APPNAME PROCID MSGID STRUCTURED_DATA MESSAGE
    template="RSYSLOG_SyslogProtocol23Format"
)

  3. Validate the configuration:

    sudo rsyslogd -N1

  4. Apply the changes:

    sudo systemctl restart rsyslog.service

  5. Test the configuration:

    logger -t rsyslog-test "Hello from $(hostname)"

Setup mTLS (mutual TLS) communication

mTLS authenticates both ends of the connection. The rsyslog client presents a client certificate (signed by the Collector's Root CA), and the Collector verifies it. In return, the client verifies the Collector's server certificate. This prevents unauthorized hosts from sending logs to the Collector and prevents the client from being tricked into forwarding logs to an impostor.

About the Collector certificates

The LogMan.io Collector maintains its own internal Certificate Authority (CA). The CA and the TLS server certificate of the Collector are stored in the Collector's data directory, for example:

<collector data dir>/ca/ca-cert.pem            # Root CA certificate
<collector data dir>/ca/ca-key.pem             # Root CA private key (never leaves the Collector)
<collector data dir>/ca/ssl-server-cert.pem    # Collector TLS server certificate
<collector data dir>/ca/ssl-server-key.pem     # Collector TLS server private key

To enable mTLS on the Collector, its syslog input must be configured to request and verify a client certificate. In the Collector's syslog configuration (e.g. config/syslog-514.yaml), add a cafile pointing to the Root CA certificate and set verify_mode: CERT_REQUIRED on the input:SmartStream section:

config/syslog-514.yaml
input:SmartDatagram:smart-udp-514:
  address: '514'
  output: smart
  smart: {}
input:SmartStream:smart-tcp-514:
  address: '514'
  output: smart
  smart: {}
  # mTLS: require and verify a client certificate for TCP/514 connections.
  # The rsyslog client must present a certificate signed by the Collector Root CA.
  cafile: /data/ca/ca-cert.pem      # absolute path inside the Collector container
  verify_mode: CERT_REQUIRED        # reject clients that do not present a valid client certificate

After changing the configuration, restart the Collector for the change to take effect.

Verifying mTLS on the Collector

When mTLS is enforced, a TLS client that connects without a client certificate is rejected by the Collector with a message similar to:

[SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate

This is expected and confirms that the Collector requires client certificates.

Create a client certificate for the rsyslog host

The rsyslog client needs a certificate signed by the Collector's Root CA. Generate a key and certificate request (CSR) on the client host:

mkdir -p /etc/rsyslog.d/certs && cd /etc/rsyslog.d/certs

# 1. Generate a private key
openssl ecparam -name prime256v1 -genkey -noout -out client-key.pem   # or: openssl genrsa -out client-key.pem 2048
chmod 600 client-key.pem

# 2. Create a CSR with client-authentication usage
openssl req -new -key client-key.pem -out client.csr \
  -subj "/CN=<client hostname>/O=TeskaLabs/OU=LogMan.io Collector"

# 3. Sign the CSR with the Collector's Root CA and set the clientAuth Extended Key Usage
openssl x509 -req -in client.csr \
  -CA <path-to-ssl-server-ca>/ca-cert.pem \
  -CAkey <path-to-ssl-server-ca>/ca-key.pem \
  -CAcreateserial -days 365 -out client-cert.pem \
  -extfile <(printf "basicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth\n")
chmod 600 client-cert.pem

# 4. Copy the Root CA certificate onto the client host
cp <path-to-ssl-server-ca>/ca-cert.pem /etc/rsyslog.d/certs/ca-cert.pem
chmod 644 /etc/rsyslog.d/certs/ca-cert.pem

Keep the Collector's CA private key secure

The Collector's ca-key.pem is a highly sensitive secret. Sign client certificates in a controlled environment and only distribute the signed client certificate (plus the public ca-cert.pem) to client hosts. Never copy the CA private key to the rsyslog client.

Configure rsyslog for mTLS forwarding

Create (or replace) the rsyslog forwarding configuration with the following mTLS settings:

/etc/rsyslog.d/teskalabs-logman-io.conf
global(
    DefaultNetstreamDriver="gtls"

    # CA that signed the Collector's server certificate (used to verify the Collector).
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca-cert.pem"

    # mTLS: the client certificate + key presented to the Collector.
    # Must be signed by the Collector's Root CA.
    DefaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/client-cert.pem"
    DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/client-key.pem"
)

# Forward all logs via mTLS (TLS with mutual client authentication) to the Collector.
*.* action(
    type="omfwd"
    protocol="tcp"
    target="<IP address of the collector>"
    port="514"
    KeepAlive="on"
    queue.type="LinkedList"
    queue.size="10000"

    # TLS / mTLS options
    StreamDriver="gtls"
    StreamDriverMode="1"                    # establish TLS immediately on connect
    StreamDriverAuthMode="x509/name"        # server (name) + client certificate authentication
    StreamDriverPermittedPeers="<collector server cert CN or SAN>"  # must match the Collector server certificate

    # RFC5424 format: <PRI>VERSION TIMESTAMP-RFC3339 HOSTNAME APPNAME PROCID MSGID STRUCTURED_DATA MESSAGE
    template="RSYSLOG_SyslogProtocol23Format"
)

Notes:

  • DefaultNetstreamDriverCertFile and DefaultNetstreamDriverKeyFile are the client certificate and key created above. They are what makes the connection mutual.
  • StreamDriverAuthMode="x509/name" verifies the server certificate and presents the client certificate to the Collector (use the same value that is used for one-way TLS—the client cert/key files are what turn it into mTLS).
  • StreamDriverPermittedPeers must match a name or IP address present in the Collector's server certificate (its CN and/or Subject Alternative Name). If the Collector's server certificate is issued for an IP address (e.g. CN=192.0.2.1), use that value here. The path to the matching server certificate is stored in the Collector's CA data directory under ca/ssl-server-cert.pem.

Validate and enable

  1. Validate the configuration:

    sudo rsyslogd -N1

  2. Apply the changes:

    sudo systemctl restart rsyslog.service

  3. Test the mTLS handshake from the client against the Collector:

    echo | openssl s_client -connect <IP>:<port> \
  -CAfile /etc/rsyslog.d/certs/ca-cert.pem \
  -cert /etc/rsyslog.d/certs/client-cert.pem \
  -key  /etc/rsyslog.d/certs/client-key.pem \
  2>/dev/null | grep -E "Verify return code|subject|issuer"

    A Verify return code: 0 (ok) with the Collector's certificate subject/issuer indicates a successful mutual TLS handshake.

  4. Confirm logs are delivered end-to-end:

    logger -t rsyslog-mtls-test "Hello from $(hostname)"

    Then check the Collector logs for a SSL detected. notice and a Connected. event from the client's IP address.

Troubleshooting

Symptom Likely cause / fix
rsyslogd: error during parsing file ... syntax error on token The config values are missing double quotes. Quote every value, e.g. DefaultNetstreamDriver="gtls".
[SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] on the Collector The client connected without a client certificate. Add DefaultNetstreamDriverCertFile / DefaultNetstreamDriverKeyFile (mTLS is enforced on the Collector).
Verify return code: 20 (unable to get local issuer certificate) The client does not trust the Collector's CA. Check DefaultNetstreamDriverCAFile / -CAfile points to the Collector Root CA (ca-cert.pem).
Verify return code: 18 (self-signed certificate) The CA used for validation is not the CA that signed the Collector's server certificate.
Connection succeeds but no logs appear Confirm the StreamDriverPermittedPeers matches a CN/SAN in the Collector's server certificate, and verify the Collector actually ingests (check Collector logs for Connected./SSL detected.).