Collecting logs from Microsoft Windows

There are multiple ways how to collect logs respectively Windows Events from Microsoft Windows.

Collecting by Window Event Collector (WEC/WEF)

Agent-less Window Event Collector (WEC) ship logs from Windows machines via Windows Event Forwarding (WEF) service to LogMan.io collector. LogMan.io Collector acts as Window Event Collector (WEC). The WEF configuration can be provided using Group Policy either managed centrally the Active Directory server or using Local Group Policy. This strategy doesn't require any configuration on Windows machines.

Tip

We recommend this method for collecting Windows Event logs.

Schema: Event flow of WEC/WEF collection in TeskaLabs LogMan.io.

WEC configuration options

Input specification: input:WEC:

listen:  # Where to expose the server, format <ADDRESS> <PORT> <OPTIONAL SSL>, i. e. "0.0.0.0 5985" for HTTP with Kerbros or "0.0.0.0 5986 ssl" for HTTPS
queries:  # The Windows Event queries separated by new lines, which determine which Windows Events should be loaded in subscriptions
output:  # Which output to send the incoming events to
last_value_storage:  # Persistent storage for the current last value (default: ./var/last_value_storage)
read_existing_events:  # (optional) Notifies Windows machines if they should send existing events (true/false, default: true)
connection_retries:  # (optional) How many retries in a row is acceptable from Windows machines (default: 60)
connection_retries_wait:  # (optional) How long in seconds to wait for connection retry (default: 10.0)
heartbeat:  # (optional) How often in seconds the heart beat should be called upon subscriptions (default: 60)
backlog:  # (optional) Specify the number of pending connections the queue will hold (default: 128)
servertokens:  # (optional) Controls whether 'Server' response header field is included ('full') or faked 'prod' (default: full)
cors: # (optional) Specify CORS attributes (default: none)

The queries setting with Windows Event queries may look as follows (query name followed by its definition):

  queries:
    System: "*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]"
    Security: "*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]"

The queries can be specified for every Window Event log type, including:

Application for application logs
System for system logs
Security for security logs

The queries can be limited for certain source IP addresses or machine IDs using YAML list of the identificators at the end of the query. The machine ID is unique to each Windows machine and it is based on the domain hostname.

  queries:
    System:
      "*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]"

    Security:
      "*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]"
    ...
    Application:
      "*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]":
        - 192.168.111.128
        - 192.168.111.128
        - "APP.example.com"
    ...

In this way, only Windows machines with IPs 192.168.111.128, 192.168.111.128 and with machine ID APP.example.com will send application logs (their subscription will contain the application query).

There are two authentication options for WEC:

Kerberos and Active Directory
Mutual SSL authentication using HTTPS certificate

Kerberos authentication

Kerberos authentication utilizes Active Directory domain respective Kerberos tickets issued by the domain controller for authentication and encryption of the log forwarding. WEC server (LogMan.io collector) listens on HTTP server on port tcp/5985. This authentication require Active Directory.

LogMan.io Collector WEC configuration options for Kerberos Authentication

listen: 0.0.0.0 5985
last_value_storage:  # Persistent storage for the current last value (default: ./var/last_value_storage)
read_existing_events:  # (optional) Notifies Windows machines if they should send existing events (true/false, default: true)
connection_retries:  # (optional) How many retries in a row is acceptable from Windows machines (default: 60)
connection_retries_wait:  # (optional) How long in seconds to wait for connection retry (default: 10.0)
heartbeat:  # (optional) How often in seconds the heart beat should be called upon subscriptions (default: 60)
backlog:  # (optional) Specify the number of pending connections the queue will hold (default: 128)
servertokens:  # (optional) Controls whether 'Server' response header field is included ('full') or faked 'prod' (default: full)
cors: # (optional) Specify CORS attributes (default: none)
output:  # Which output to send the incoming events to

Kerberos Authentication setup

Prerequisites

Microsoft Active Directory Domain Controller (in this example with IP address 192.168.0.1 and hostname ad.example.om, domain name example.om / EXAMPLE.OM)
Linux host machine with LogMan.io collector (in this example with IP address 192.168.0.2 and hostname lmio-wec)
Date and time of the LogMan.io collector MUST be NTP synchronized
LogMan.io collector SHOULD use the DNS server of the Active Directory

Configuration on the Windows servers

1. Create a new user in Active Directory

Navigate to Administrative Tools > Active Directory Users and Computers > example.om > Users

Right click and choose New > User

Enter following information:

First name: lmio-wec
Full name: TeskaLabs LogMan.io
User logon name: lmio-wec

Set a password for the user. This example uses Password01!.

Uncheck "User must change password at next logon".

Check "Password never expires".

Hit "Next" and then "Finish" button to create the user.

Finally do the right click on the new user, click Properties, and open the Account tab: * Check "This account supports Kerberos AES 128 bit encryption". * Check "This account supports Kerberos AES 256 bit encryption".

The new user lmio-wec is now ready.

2. On the DNS server, create an A record for lmio-wec.example.om

Go to Administrative Tools > DNS > Forward Lookup Zones > example.om

Right click and choose "New Host (A or AAAA)…"

Add a record with name lmio-wec and IP address 192.168.0.2

Check the "Create associated pointer (PTR) record option"

Hit "Add Host" button to finish.

Tip

You can verify this DNS settings by ping command.

3. On the Active Directory Domain Controller Server's command prompt configure principal name for the host and the service running on the LogMan.io collector by executing following commands

Process is case-sensitive

When you create keytabs, make sure to CAPITALIZE anything you see capitalized in our examples (such as host/lmo-wec.example.om@EXAMPLE.OM). If you don't capitalize correctly, this process won't work.

Create a host principal name and the associated keytab file:

> ktpass /princ host/lmio-wec.example.om@EXAMPLE.OM /pass Password01! /mapuser EXAMPLE\lmio-wec -pType KRB5_NT_PRINCIPAL /out host-lmio-wec.keytab /crypto AES256-SHA1

Create a service principal name and the associated keytab file:

> ktpass /princ http/lmio-wec.example.om@EXAMPLE.OM /pass Password01! /mapuser EXAMPLE\lmio-wec -pType KRB5_NT_PRINCIPAL /out http-lmio-wec.keytab /crypto AES256-SHA1

Tips

You will need to use the password you set for a user lmio-wec.
The domain following the @ symbol is the Active Directory domain of the computer from which you want to collect logs (lmio-wec.example.om@EXAMPLE.OM).

Configuration in LogMan.io Collector

4. Copy generated keytabs from the Windows Active Directory Domain Controller Server to LogMan.io collector machine /tmp directory

5. Install Kerberos client on LogMan.io collector

For Linux Ubuntu, use:

# sudo apt update -y && sudo apt install krb5-user -y

6. Edit the Kerberos configuration file on LogMan.io collector

Kerberos configuration file is usually located at /etc/krb5.conf.

Uncomment section [domain_realm] and add:

.example.om = EXAMPLE.OM
example.om = EXAMPLE.OM

Uncomment section [realms] and add:

EXAMPLE.OM= {
kdc = example.om
admin_server = example.om
}

7. Use ktutil to merge the two keytab files generated from ktpass commands on LogMan.io collector

# ktutil 
 1. ktutil:  rkt /tmp/host-lmio-wec.keytab 
 2. ktutil:  rkt /tmp/http-lmio-wec.keytab 
 3. ktutil:  wkt /tmp/lmio-result.keytab
 4. ktutil:  q

8. Validate the merged keytab on LogMan.io collector

# klist -e -k -t /tmp/lmio-result.keytab

 Keytab name: FILE:/tmp/lmio-result.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
3 08/04/22 15:06:24 host/lmio-wec.example.om@EXAMPLE.OM (aes256-cts-hmac-sha1-96) 
4 08/04/22 15:06:24 http/lmio-wec.example.om@EXAMPLE.OM (aes256-cts-hmac-sha1-96)

9. Move the new keytab file and the configuration into place on LogMan.io collector

# mkdir /conf/kerberos
# cp /tmp/lmio-result.keytab /conf/kerberos/krb5.keytab
# rm /tmp/*.keytab
# chmod a+r /conf/kerberos/krb5.keytab

# cp /etc/krb5.conf /conf/kerberos/krb5.conf
# chmod a+r /conf/kerberos/krb5.conf

In /conf/kerberos/krb5.keytab, comment out the section includedir.

10. Test that the authentication with Active Directory is working successfully when using the keytab.

Run the following command on the LogMan.io collector.
If the configuration is correct a ticket-granting ticket (TGT) will be created and cached.
This command should be invoked with the same OS user that the LogMan.io service runs as.

```
# kinit -kt /etc/krb5.keytab http/lmio-wec.example.om@EXAMPLE.OM
```

11. Verify the ticket was obtained by running klist as the same user from the previous step

```
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: http/lmio-wec.example.om@EXAMPLE.OM

Valid starting     Expires            Service principal
08/04/22 15:09:29  08/05/22 01:09:29  krbtgt/EXAMPLE.OM@EXAMPLE.OM
renew until 08/05/22 15:09:29
```

12. We are done.

Now create a Group Policy Object to forward Windows Events from Windows Machine to Linux WEC (check "Group Policy configuration" part below)

The URL for WEF is:  
`http://<WEC_SERVER_HOSTNAME>:5985/wsman/SubscriptionManager/WEC,Refresh=<Refresh interval in seconds>`

HTTPS certificate authentication

WEC authentication is the alternative to Kerberos, using HTTPS certificates expects WEC server (LogMan.io collector) to provide HTTPS server on port tcp/5986. This authentication can be operated without Active Directory.

LogMan.io Collector WEC configuration options for HTTPS certificate authentication

Following options are used to setup HTTPS certificate authentication.

listen: 0.0.0.0 5986 ssl

cert:  # Specify path to the WEC server certificate
key:  # Specify path to the WEC server private key 
issuer_thumbprints:  # Specify issuer (CA) certificate SHA1 thumbprints separated by space (f. e. d6986fef2104f21ab0c7ccb279217abe29c0808a)
password:  # (optional) Specify a private key file password (default: none)
cafile:  # (optional) Specify a file to verify the peer (default: none)
capath:  # (optional) Specify a path to verify the peer (default: none)
ciphers:  # (optional) Specify custom SSL ciphers (default: none)
dh_params:  # (optional) Diffie–Hellman (D-H) key exchange (TLS) parameters (default: none)
verify_mode:  # (optional) Empty or one of CERT_NONE, CERT_OPTIONAL or CERT_REQUIRED

queries:  # The Windows Event queries separated by new lines, which determine which Windows Events should be loaded in subscriptions
output:  # Which output to send the incoming events to

TLS/SSL Certificate requirements:

The WEF certificate (client) MUST have the X509 v3 Extended Key Usage: TLS Web Client Authentication.
The WEC certificate (server) MUST have the X509 v3 Extended Key Usage: TLS Web Server Authentication.

Tip

You MAY use certificates provided by your CA, if applicable. Make sure that certificates are compliant with above criteria.

Tip

You can use XCA tool to generate CA, WEC server and WEF client certificates.

Tip

You can use also Windows certificate manager to generate respective certificates.

The URL for WEF is:
https://<WEC_SERVER_HOSTNAME>:5986/wsman/SubscriptionManager/WEC,Refresh=<Refresh interval in seconds>,IssuerCA=<Thumbprint of the issuing CA certificate>

The certificate must be read by NT_AUTHORITY\NetworkService user, which can be specified in the Certificate Manager / Security setting.

Group Policy configuration

The Windows Event Forwarding (WEF) must be configured on Windows machines that should ship logs to LogMan.io.

The Windows Event Forwarding configuration can be provided by:

Local Group Policy
Active Directory Group Policy

Prerequisities:

Make sure that WinRM is enabled, firewall is not limiting the network connection and the WEC server hostname is trusted at the source Windows machine. To enable WinRM, run as administrator the following command:

winrm qc -q

The WEC server needs to be added to Trusted Hosts, so that WinRM allows WEC to communicate:

winrm set winrm/config/client '@{TrustedHosts="<SERVER_HOSTNAME>"}'

When the setting is ready, configure the Windows Event Forwarding (more information and screenshots are available on https://docs.microsoft.com/cs-cz/advanced-threat-analytics/configure-event-collection ):

Local Group Policy

Local Group Policy configuration can be used for configuring individual Windows machines.

1. Open Local Group Policy Editor

Press Win+R and type: gpedit.msc

2. Navigate to:

Local Computer Policy\Computer Configuration\Administrative Templates\Windows Components\Event Forwarding

3. Configure Target Subscription Manager policy

4. Add a new Server entry

For Kerberos and Active Directory authentication (port tcp/5985):

Server=http://<WEC_HOSTNAME>:5985/wsman/SubscriptionManager/WEC,Refresh=<Refresh interval in seconds>

For HTTPS authentication (port tcp/5986):

Server=https://<WEC_HOSTNAME>:5986/wsman/SubscriptionManager/WEC,Refresh=<Refresh interval in seconds>,IssuerCA=<Thumbprint of the issuing CA certificate>

Recommended Refresh interval is 60 seconds.

Thumbprint of the certificate is a SHA-1, lowercase string., eg. d6986fef2104f21ab0c7ccb279217abe29c0808a.

If intermediate CA is present, then IssuerCA must point to the issuing intermediate CA, NOT to the Root CA.

5. Hit "Apply" to save changes

6. Run gpupdate /force at the command line.

7. Enable Security log (see below)

For more information, see: https://docs.microsoft.com/en-us/windows/win32/wec/setting-up-a-source-initiated-subscription

Active Directory Group Policy

Active Directory Group Policy configuration is suitable for configuring higher number of Windows machines, connected to the respective Domain controller.

1. Open the Group Policy Management Console and edit an object

From the Start screen, select Administrative Tools. A list of available management tools is shown, including Group Policy Management installed in the previous section.
To open the Group Policy Management Console (GPMC), choose Group Policy Management.

2. Create GPO

In the Group Policy Management console, select your custom organizational unit (OU), such as MyCustomOU. Right-select the OU and choose Create a GPO in this domain, and Link it here...:

Specify a name for the new GPO, such as My custom GPO, then select OK. You can optionally base this custom GPO on an existing GPO and set of policy options.

3. Configure GPO

The custom GPO is created and linked to your custom OU. To now configure the policy settings, right-select the custom GPO and choose Edit...:

The Group Policy Management Editor opens to let you customize the GPO:

4. Configure Event Forwarding Policy under Compouter Configuration section ( as done above in the "Local Group Policy" section of this documentation page)

5. Enable Security log (see below)

Enable Security log

WEF doesn't have by default access to the security log. In order to enable forwarding of the Security log, Network Service HAS TO be added to WEF.

Launch Group Policy Editor (type gpedit.msc).
Navigate to Computer Configuration > Administrative Templates > Windows Components > Event Log Service > Security
Edit "Configure Log Access"
Enable the policy
In Log Access enter:
O:BAG:SYD:(A;;0xf0005;;;SY)(A;;0x5;;;BA)(A;;0x1;;;S-1-5-32-573)(A;;0x1;;;S-1-5-20)
Apply by executing gpupdate /force

Test

Use the following command to create a testing event at the Windows machine:

eventcreate /Id 500 /D "This is a test message." /T ERROR /L System

The test log should arrive into LogMan.io immediately.

Troubleshooting

Windows Forwaring Event Log

The Eventlog-forwardingPlugin/Operational event channel logs relevant information of machines that are setup to forward logs into the collector. It also contains the information about possible issues with WEF subscription. Use "Event Viewer" application to investigate.

Collecting by WinRM

Agent-less remote control connection to a desired Windows machine and run of the collection command there as a separate process to collect its standard output.

Input specification: input:WinRM:

WinRM input connects to a remote Windows Server machine, where is calls a specified command. It then periodically checks for new output at stdout and stderr, so it behaves in a similar manner to input:SubProcess.

LogMan.io Collector WinRM configuration options

endpoint:   # Endpoint URL of the Windows Management API of the remote Windows machine (f. e. http://MyMachine:5985/wsman)
transport: ntlm  # Authentication type
server_cert_validation:  # Specify the certificate validation (default: ignore)
cert_pem:  # (optional) Specify path to the certificate (if using HTTPS)
cert_key_pem:  # (optional) Specify path to the private key
username:  # (optional) When using username authentication (like over ntlm), specify username in format <DOMAIN>\<USER>
password:  # Password of the authenticated user above
output:  # Which output to send the incoming events to

The following configuration clarifies the command that should be remotely called:

# Read 1000 system logs once per 2 seconds
command:  # Specify the command, that should be remotely called (f. e. wevtutil qe system /c:1000 /rd:true)
chilldown_period:  # How often in seconds should the remote command be called, if it is ended (default: 5)
duplicity_check:  # Specify if to check for duplicities based on time (true/false)
duplicity_reverse_order:  # Specify if to check for duplicities in reverse order (f. e. logs come in descending order)
last_value_storage:  # Persistent storage for the current last value in duplicity check (default: ./var/last_value_storage)

Collecting by the agent on the Windows machine

In this variant, LogMan.io collector runs as an agent on a desired Windows machine and collect Windows Events.

Input specification: input:WinEvent

Note: input:WinEvent only works at Windows-based machine.

This input periodically reads Windows Events from the specified event type.

LogMan.io Collector WinEvent configuration options

server:  # (optional) Specify source of the events (default: localhost, i. e. the entire local machine)
event_type:  # (optional) Specify the event type to be read (default: System)
buffer_size:  # (optional) Specify how many events should be read in one query (default: 1024)
event_block_size:  # (optional) Specify the amount of events after which an idle time will be executed for other operations to take place (default: 100)
event_idle_time:  # (optional) Specify the idle time in seconds mentioned above (default: 0.01)
last_value_storage:  # Persistent storage for the current last value (default: ./var/last_value_storage)
output:  # Which output to send the incoming events to

The event type can be specified for every Window Event log type, including:

Application for application logs
System for system logs
Security for security logs etc.