Skip to content

Collecting from Beats or Logstash

Supported Beats

TeskaLabs LogMan.io supports collecting logs from a wide range of Elastic Beats, which are lightweight data shippers for various log sources and platforms. The most commonly used Beats include:

  • Winlogbeat: Collects Windows Event, including Application, System, Security, and many other channels. Useful for monitoring Windows servers and workstations.
  • Filebeat: Collects and ships log from files. Common use cases include:
    • Microsoft DHCP: Collects DHCP server logs from Windows servers.
    • Microsoft DNS: Collects DNS debug logs from Windows DNS servers.
    • Microsoft IIS: Collects web server logs from Microsoft IIS.
    • Custom application logs: Collects logs from any application or service that writes to a file.
  • Auditbeat: Collects audit logs from Linux systems, such as file integrity monitoring and user activity.
  • Other Beats: Any other community or official Beats (e.g., Metricbeat, Packetbeat) can also be integrated.

Beats send their data to the LogMan.io Collector using the Lumberjack protocol (the same protocol used by Logstash). This allows for secure and reliable log shipping from a variety of sources and platforms.

Configuration of TeskaLabs LogMan.io

The basic configuration snippet of the collector:

input:Lumberjack:logstash:
  output: ...

logstash is an input identified and can be replaced by anything unique within the configuration file

address specifies an network interface and/or a port that will be used for listening. The default value is '5044', to listen on tcp/5044.

Optional smart attribute can be used to specify a smart map, when connected to the output:CommLink.

Tip

The protocol used for this extraction is called Lumberjack and by default runs on tcp/5044 with optional SSL. Log source classes input:Lumberjack:, input:Logstash: and input:Beats: are actually synonyms.

SSL configuration

The incoming SSL is detected automatically.

The following configuration options specify the SSL connection:

  • cert: Path to the client SSL certificate
  • key: Path to the private key of the client SSL certificate
  • password: Private key file password (optional, default: none)
  • cafile: Path to a PEM file with CA certificate(s) to verify the SSL server (optional, default: none)
  • capath: Path to a directory with CA certificate(s) to verify the SSL server (optional, default: none)
  • cadata: one or more PEM-encoded CA certificates to verify the SSL server (optional, default: none)
  • ciphers: SSL ciphers (optional, default: none)
  • dh_params: Diffie–Hellman (D-H) key exchange (TLS) parameters (optional, default: none)
  • verify_mode: One of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED (optional); for more information, see: github.com/TeskaLabs/asab

Example Configuration of Beats

This log source can be used to collect logs using Beats family of tools. Beats are lightweight data shippers from Elastic.

Configuration snippet:

output.logstash:
  hosts: ["<collector>:5044"]

Winlogbeat

winlogbeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System
    ignore_older: 72h

  - name: Security
    ignore_older: 72h

  - name: Setup
    ignore_older: 72h

  - name: Microsoft-Windows-Sysmon/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Defender/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-GroupPolicy/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-TaskScheduler/Operational
    ignore_older: 72h

  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
    ignore_older: 72h

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800
    ignore_older: 72h

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106
    ignore_older: 72h

  - name: ForwardedEvents
    tags: [forwarded]

logging.to_files: true
logging.files:
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7

Security logs

Make sure the user, under which the Winlogbeat service runs, has a permission to access Security logs.

To do so, the user has to be part of the Event Log Readers group.

Step-by-Step (via lusrmgr.msc)

  1. Open the Run dialog:

  2. Press Win + R on your keyboard.

  3. Type lusrmgr.msc and press Enter.

  4. Navigate to Groups:

  5. In the left pane, click on Groups.

  6. Find and open the group:

  7. In the right pane, double-click on Event Log Readers.

  8. Add a new member:

  9. In the window that appears, click on the Add... button.

  10. Select the user or service:

  11. In the Enter the object names to select field, type the name of the user or service account you want to add.

    • For a service account like Winlogbeat, you might use:

    NT SERVICE\Winlogbeat
    * For a regular local user, simply type the username (e.g. winlogbeat_user).

  12. Check and confirm:

  13. Click Check Names to validate the input.

  14. Click OK to confirm.

  15. Apply and close:

  16. Click Apply and then OK to close all windows.

  17. Restart the Winlogbeat service:

  18. Open PowerShell or Command Prompt as administrator.

  19. Run:

    Restart-Service winlogbeat

Filebeat

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  - type: filestream
    fields:
      stream: <stream name>
    paths:
      - /path/to/the/file.log

Tip

<stream name> is for example my-app-log. You don't need to include tenant or any other prefix.

Microsoft DHCP

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:

  # Microsoft DHCP IPv4
  - type: filestream
    id: microsoft-dhcp-ipv4
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1780
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpSrvLog-*.log
    include_lines:
      - "^[0-9]+,"

  # Microsoft DHCP IPv6
  - type: filestream
    id: microsoft-dhcp-ipv6
    fields:
      stream: microsoft-dhcp-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1238
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\DHCP\DhcpV6SrvLog-*.log
    include_lines:
      - "^[0-9]+,"

Microsoft DNS

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  # Microsoft DNS Debug Log
  - type: filestream
    id: microsoft-dns
    fields:
      stream: microsoft-dns-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1052
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\Windows\System32\dns\debug.log
    include_lines:
      - "^[0-9]+[/-:][0-9]+"

Microsoft IIS

filebeat.yaml
output.logstash:
  hosts: ["<collector>:5044"]

filebeat.inputs:
  # Microsoft IIS log
  - type: filestream
    id: microsoft-iis
    fields:
      stream: microsoft-iis-filebeat-v1
    prospector.scanner.fingerprint:
      enabled: true
      offset: 1052
      length: 64
    file_identity.fingerprint: ~
    paths:
      - C:\inetpub\logs\LogFiles\W3SVC*\*.log