Collecting logs using rsyslog¶

rsyslog is a high-performance, open-source, modular syslog daemon commonly installed on Linux systems, designed to collect, parse, and ship system and application logs. It’s a cornerstone of reliable log pipelines because it supports modern protocols and formats, strong transport security (TLS), and robust buffering with disk-assisted queues that prevent data loss during network interruptions. With its flexible rules engine, rsyslog can enrich events, drop noise, tag sources, and fan-out to multiple destinations—making it ideal for forwarding the log to a TeskaLabs LogMan.io at scale.

Setup rsyslog log forwarding¶

To set up rsyslog log forwarding to the LogMan.io Collector, follow these steps:

Verify connectivity to the LogMan.io Collector using the netcat command:
```
nc -vz <Collector IP address or hostname> <port>
```
If the connection is successful, you should see a message indicating that the connection was successful (e.g., "Connection to port [tcp/*] succeeded!").
Check if the rsyslog service is running:
```
sudo systemctl status rsyslog.service
```
If the service is not running, start it with:
```
sudo systemctl start rsyslog.service
```
If rsyslog is not installed, you can install it using the package manager for your Linux distribution.
```
sudo apt install rsyslog rsyslog-gnutls
```

Create the configuration file for rsyslog to forward logs to LogMan.io Collector:

/etc/rsyslog.d/teskalabs-logman-io.conf

*.* action(
    type="omfwd"
    protocol="tcp"
    target="<IP address of the collector>"
    port="514"
    KeepAlive="on"
    queue.type="LinkedList"
    queue.size="10000"

    # TLS/SSL options
    StreamDriver="gtls"
    StreamDriverMode="1"  # TLS/SSL connection is established immediately when connecting to the server
    StreamDriverAuthMode="anon"  # The client will not authenticate itself to the server, and the server will not authenticate itself to the client

    # Set the format of the syslog message to: <PRI>VERSION TIMESTAMP-RFC3339 HOSTNAME APPNAME PROCID MSGID STRUCTURED_DATA MESSAGE
    template="RSYSLOG_SyslogProtocol23Format"
)

Note that the default smart syslog port at LogMan.io Collector will auto-detect incoming TLS/SSL connection.

Validate the configuration for syntax errors:
```
sudo rsyslogd -N1
```
Apply the changes by restarting the rsyslog service:
```
sudo systemctl restart rsyslog.service
```

Test the configuration by:

logger -t rsyslog-test "Hello from $(hostname)"

Enable MARK messages

Enable the MARK module in /etc/rsyslog.conf to emit periodic -- MARK -- messages, which makes it easy to spot silent or disconnected senders.

/etc/rsyslog.conf

module(load="immark")

Note that this works only on Linux distributions that support the MARK module.

Setup rsyslog client authentication¶

To set up authentication for rsyslog client, you can use TLS/SSL certificates to secure the connection between the rsyslog client and the LogMan.io Collector.

Verify the TLS/SSL connection to the LogMan.io Collector using the openssl command:

echo | openssl s_client -connect <IP>:<port> -servername <IP> -CAfile /path/to/ca.crt 2>/dev/null | grep "Verify return code"

The following configuration example demonstrates how to configure rsyslog to use TLS/SSL for secure log forwarding to the LogMan.io Collector:

/etc/rsyslog.d/teskalabs-logman-io.conf

global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/path/to/ca.crt"  # Replace with the actual path to your CA certificate
)

*.* action(
type="omfwd"
protocol="tcp"
target="<IP address of the collector>"
port="514"
KeepAlive="on"

queue.type="LinkedList"
queue.size="10000"

# TLS/SSL options
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="<IP address of the collector>"  # Replace with the actual IP address or hostname of the LogMan.io Collector

# Set the format of the syslog message to: <PRI>VERSION TIMESTAMP-RFC3339 HOSTNAME APPNAME PROCID MSGID STRUCTURED_DATA MESSAGE
template="RSYSLOG_SyslogProtocol23Format"
)

Validate the configuration:
```
sudo rsyslogd -N1
```
Apply the changes:
```
sudo systemctl restart rsyslog.service
```

Test the configuration:

logger -t rsyslog-test "Hello from $(hostname)"