Přesměrování logů z Collectoru

LogMan.io Collector může přijaté logy přesměrovat do jiných systémů — další instance kolektoru, SIEM nebo nástroje s TCP/UDP. Vytvoříte pipeline, kde se logy přijmou, volitelně zpracují a přesměrují na více cílů.

Přehled

Přesměrování logů umožňuje:

• Replikovat logy na více cílů pro redundanci nebo compliance
• Přesměrovat logy na další kolektory v distribuovaném nasazení
• Odesílat logy do externích SIEM a nástrojů pro správu logů
• Vytvářet pipeline, kde logy procházejí více fázemi zpracování

Mechanismus přidá komponentu forward: do pipeline. Forward dostává události ze stejného výstupu jako vstup — lokální zpracování a vzdálené přesměrování současně.

Základní konfigurace přesměrování

Přesměrování TCP

Pro přesměrování logů přes TCP na jiný systém:

# Input receiving logs
input:TCP:TestTCPInput:
  address: 0.0.0.0:9999
  output: print1

# Forward logs to another system via TCP
forward:TCP:TestTCPForward:
  address: 127.0.0.1:9998
  output: print1

# Local output (optional)
output:Print:print1: {}

V tomto příkladu:

1. Input (input:TCP:TestTCPInput) přijímá logy na portu 9999 ze všech rozhraní (0.0.0.0)
2. Forward (forward:TCP:TestTCPForward) přesměruje stejné logy na 127.0.0.1:9998 přes TCP
3. Output (output:Print:print1) také zpracuje logy lokálně (výpis do konzole)

Forward a output používají stejné ID výstupu (print1) — oba dostávají události ze vstupu.

Přesměrování UDP

Pro přesměrování přes UDP:

# Input receiving logs
input:UDP:TestUDPInput:
  address: 0.0.0.0:9999
  output: print2

# Forward logs to another system via UDP
forward:UDP:TestUDPForward:
  address: 127.0.0.1:9998
  output: print2

# Local output (optional)
output:Print:print2: {}

Konfigurace je podobná TCP, ale používá UDP.

Parametry konfigurace forward

Adresa

Parametr address určuje cíl přesměrování logů:

forward:TCP:myforward:
  address: <IP_ADDRESS>:<PORT>
  output: <OUTPUT_ID>

• IP_ADDRESS: The destination IP address or hostname (IPv4 or IPv6 supported)
• PORT: The destination port number

Examples:

address: 192.168.1.100:514        # IPv4 address
address: [2001:db8::1]:514        # IPv6 address (use brackets)
address: collector.example.com:514 # Hostname

Reference výstupu

Parametr output musí odkazovat na stejné ID výstupu jako vstup:

input:TCP:myinput:
  output: shared_output  # Input uses this output

forward:TCP:myforward:
  output: shared_output  # Forward also uses the same output

output:Print:shared_output: {}  # Local output definition

Forward a lokální výstup tak dostanou stejné události ze vstupu.

Přesměrování se smart classification

You can combine forwarding with smart classification to route logs based on source IP addresses, ports, and protocols. This is useful when you want to forward all classified logs to an external system while also processing them locally.

Example: Forwarding Classified Logs

This example shows how to use smart classification with forwarding for both UDP and TCP inputs:

classification:
  smart_syslog: &smart_syslog  # YAML anchor referencing to both SmartDatagram and SmartStream inputs

    # All events are sent to stream 'generic'
    generic:
      - ip: "*"

# Listen on the UDP 514
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_syslog
  output: smart-udp

forward:UDP:SmartUDPForward:
  address: 127.0.0.1:9998
  output: smart-udp

# Connection to LogMan.io
output:CommLink:smart-udp: {}

# Listen on the TCP 514
input:SmartStream:TCP514:
  address: 514
  smart: *smart_syslog
  output: smart-tcp

forward:TCP:SmartTCPForward:
  address: 127.0.0.1:9998
  output: smart-tcp

# Connection to LogMan.io
output:CommLink:smart-tcp: {}

In this configuration:

1. Smart Classification: The smart_syslog classification map (defined with YAML anchor &smart_syslog) routes all incoming logs to the generic stream based on source IP.

2. UDP Input: Receives logs on UDP port 514, classifies them using the smart map, and sends them to output smart-udp.

3. UDP Forward: Forwards all UDP logs to 127.0.0.1:9998 while also sending them to LogMan.io.

4. TCP Input: Receives logs on TCP port 514, classifies them using the same smart map, and sends them to output smart-tcp.

5. TCP Forward: Forwards all TCP logs to 127.0.0.1:9998 while also sending them to LogMan.io.

Both UDP and TCP inputs use the same smart classification map (via YAML anchor), but have separate output IDs (smart-udp and smart-tcp) to allow independent forwarding configurations.

Example: Advanced Classification with Forwarding

For more complex scenarios where you want to classify logs into different streams and forward them:

classification:
  smart_forward: &smart_forward
    # Classify logs from specific IP ranges
    fortinet-fortigate-1:
      - ip: "192.168.1.0/24"
        port: "*"
        protocol: UDP

    linux-syslog-1:
      - ip: "10.0.0.0/8"
        port: "*"
        protocol: TCP

    # All other logs go to generic stream
    generic:
      - ip: "*"
        port: "*"
        protocol: "*"

# Connection to LogMan.io
connection:CommLink:commlink:
  url: https://recv.logman.example.com/

# Input with smart classification
input:SmartDatagram:UDP514:
  address: 514
  smart: *smart_forward
  output: smart_forward

# Forward all classified logs to external SIEM
forward:TCP:siem_forward:
  address: siem.example.com:514
  output: smart_forward

# Also send to local LogMan.io
output:CommLink:smart_forward: {}

Smart Classification and Forwarding

When using smart classification with forwarding, the forward component receives all logs from the smart classification output. The smart classification determines which stream logs belong to in LogMan.io, while the forward component sends all logs to the external destination. If you need to forward only specific log streams to different destinations, you can use separate inputs with different forward configurations, or use transform filters to create separate output paths.

Kompletní příklad: více cílů

Here is a complete example that receives logs, processes them locally, and forwards them to multiple destinations:

# Connection to LogMan.io
connection:CommLink:commlink:
  url: https://recv.logman.example.com/

# Input receiving syslog
input:TCP:syslog_input:
  address: 0.0.0.0:514
  output: syslog_pipeline

# Forward to external SIEM system
forward:TCP:siem_forward:
  address: siem.company.com:514
  output: syslog_pipeline

# Forward to backup collector
forward:UDP:backup_collector:
  address: 192.168.1.200:1514
  output: syslog_pipeline

# Send to local LogMan.io
output:CommLink:syslog_pipeline: {}

In this configuration:

1. Logs are received on TCP port 514
2. Logs are forwarded to an external SIEM system via TCP
3. Logs are also forwarded to a backup collector via UDP
4. Logs are sent to the local LogMan.io instance via CommLink

All three destinations receive the same logs simultaneously.

Případy použití

1. Replikace logů pro compliance

Forward logs to a compliance archive system while also processing them in LogMan.io:

input:TCP:compliance_input:
  address: 0.0.0.0:514
  output: compliance_pipeline

# Forward to compliance archive
forward:TCP:compliance_archive:
  address: archive.company.com:514
  output: compliance_pipeline

# Process in LogMan.io
output:CommLink:compliance_pipeline: {}

2. Distribuované nasazení kolektorů

Forward logs from edge collectors to a central collector:

# Edge collector configuration
input:UDP:edge_input:
  address: 0.0.0.0:514
  output: edge_pipeline

# Forward to central collector
forward:TCP:central_collector:
  address: central-collector.company.com:514
  output: edge_pipeline

# Optional: also process locally
output:CommLink:edge_pipeline: {}

3. Integrace SIEM

Forward security logs to an external SIEM while maintaining local processing:

input:TCP:security_input:
  address: 0.0.0.0:1514
  output: security_pipeline

# Forward security events to SIEM
forward:TCP:siem_forward:
  address: siem.company.com:514
  output: security_pipeline

# Process in LogMan.io for correlation
output:CommLink:security_pipeline: {}

Řešení problémů

Verify Network Connectivity

Test if the collector can reach the forward destination:

# Test TCP connectivity
nc -v destination-ip destination-port

# Test UDP connectivity
nc -u -v destination-ip destination-port

Check Forward Configuration

Ensure that:

1. The output parameter in the forward component matches the output ID used by the input
2. The destination address and port are correct
3. Network firewall rules allow outbound connections to the destination
4. The destination system is listening on the specified port

Monitor Forward Status

Check collector logs for forward-related errors:

# View collector logs
docker logs -f lmio-collector

# Look for forward connection errors
docker logs lmio-collector | grep -i forward

Common Issues

1. Logs not reaching destination:
2. Verify network connectivity between collector and destination
3. Check firewall rules allow outbound connections
4. Ensure destination is listening on the correct port

5. Verify the forward address format is correct

6. Connection refused errors:

7. Destination may not be listening on the specified port
8. Firewall may be blocking the connection

9. Check destination system logs

10. Partial log forwarding:

11. UDP may drop packets if network is congested (consider TCP for reliability)
12. Check if destination system has sufficient capacity
13. Monitor collector resource usage

Doporučené postupy

1. Use TCP for reliability: TCP ensures reliable delivery, while UDP is faster but may lose packets
2. Monitor forward destinations: Set up monitoring to ensure forward destinations are available
3. Use transforms for filtering: If you need to forward only specific logs, use transform filters
4. Test connectivity: Verify network connectivity before deploying forward configurations
5. Consider load balancing: For high-volume scenarios, consider forwarding to multiple destinations with load balancing

Související dokumentace

• Collector Configuration
• Collector Inputs
• Collector Outputs
• Collector Transformations
• Syslog Collection with Smart Classification