LogMan.io Collector Mirage¶

LogMan.io Collector has the ability to create mock logs and send them through the data pipeline, mainly for testing and demonstration purposes. The source that produces mock logs is called Mirage.

Mirage uses LogMan.io Collector Library as a repository for collected logs from various providers. The logs in this library are derived from real logs.

Mirage input configuration¶

# Configuration YAML file for inputs
[config]
path=./etc/lmio-collector.yaml

# Connection to the Library where Mirage logs are stored
[library]
providers=git+http://user:password@gitlab.example.com/lmio/lmio-collector-library.git

[web]
listen=0.0.0.0 8088

input:Mirage:MirageInput:
    path: /Mirage/<path>/
    eps: <number of logs sent per second>
    output: <output_id>

Throughput of logs¶

You can define the number of logs produced every second (EPS - events per second).

The configuration below will produce 20 logs every second:

input:Mirage:MirageInput:
  eps: 20

This configuration will also add deviation, so each second the amount will be between 10 and 40 logs:

input:Mirage:MirageInput:
  eps: 20
  deviation: 0.5

In order to create a more realistic log source and change EPS during the day, you can use scenarios.

input:Mirage:MirageInput:
  eps: dayshift
  deviation: 0.5

Available options are:

normal
gaussian
dayshift
nightshift
tiny
peak

Finally, you can create a custom scenario. You can set EPS every minute:

input:Mirage:MirageInput:
  eps:
    "10:00": 10
    "12:00": 20
    "15:10": 10
    "15:11": 12
    "16:00": 5
    "23:00": 0
  deviation: 0.5

Adding new log sources to the Library¶

Create a new repository for log collection or clone the existing one.
Create a new directory. Name the directory after the log source.
Create a new file for each log in the source directory. Mirage can use the same log multiple times when sending logs through the pipeline. (You don't need 100 separate log files to send 100 logs - Mirage will repeat the same logs.)

By default, when sending logs through the pipeline, Mirage chooses from your log files randomly and approximately evenly, but you can add weight to logs to change that.

Templating logs¶

To get more unique logs without having to create more log files, template your logs. You can choose which fields in the log will have variable values, then make a list of values you want to populate that field, and Mirage will randomly choose the values.

In your log, choose which field you would like to have variable values. Replace the field values with ${fieldname}, where fieldname is what you will call the field in your list.

"user.name":${user.name}, "id":${id}, "msg":${msg}
Make a file called values.yaml in your log source directory.

In the new file values.yaml, list possible values for each templated field. Match the field names in the values.yaml file to the field names in your log files.

values:
    user.name:
        - Albus Dumbledore
        - Harry Potter
        - Hermione Granger
    id:
        - 171
        - 182
        - 193
    msg:
        - Connection ended
        - Connection interrupted
        - Connection success
        - Connection failure

Datetime formats¶

Mirage can generate a current timestamp for each log. For that, you need to choose the format the timestamp will be in. To add a timestamp to your mock log, add ${datetime: <format>} to the text in the file.

Example

${datetime: %y-%m-%d %H:%M:%S} generates the timestamp 23-06-07 12:55:26

${datetime: %Y %b %d %H:%M:%S:%f} generates the timestamp 2023 Jun 07 12:55:26:002651

Datetime directives¶

Datetime directives are derived from the Python datetime module.

Directive	Meaning	Example
`%a`	Weekday as locale’s abbreviated name.	Sun, Mon, ..., Sat (en_US);
`%A`	Weekday as locale’s full name.	Sunday, Monday, ..., Saturday (en_US);
`%w`	Weekday as a decimal number, where 0 is Sunday and 6 is Saturday.	0, 1, ..., 6
`%d`	Day of the month as a zero-padded decimal number.	01, 02, ..., 31
`%b`	Month as locale’s abbreviated name.	Jan, Feb, ..., Dec (en_US);
`%B`	Month as locale’s full name.	January, February, ..., December (en_US);
`%m`	Month as a zero-padded decimal number.	01, 02, ..., 12
`%y`	Year without century as a zero-padded decimal number.	00, 01, ..., 99
`%Y`	Year with century as a decimal number.	0001, 0002, ..., 2013, 2014, ..., 9998, 9999
`%H`	Hour (24-hour clock) as a zero-padded decimal number.	00, 01, ..., 23
`%I`	Hour (12-hour clock) as a zero-padded decimal number.	01, 02, ..., 12
`%p`	Locale’s equivalent of either AM or PM.	AM, PM (en_US); am, pm (de_DE)
`%M`	Minute as a zero-padded decimal number.	00, 01, ..., 59
`%S`	Second as a zero-padded decimal number.	00, 01, ..., 59
`%f`	Microsecond as a decimal number, zero-padded to 6 digits.	000000, 000001, ..., 999999
`%z`	UTC offset in the form ±HHMM[SS[.ffffff]] (empty string if the object is naive).	(empty), +0000, -0400, +1030, +063415, -030712.345216
`%Z`	Time zone name (empty string if the object is naive).	(empty), UTC, GMT
`%j`	Day of the year as a zero-padded decimal number.	001, 002, ..., 366
`%U`	Week number of the year (Sunday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0.	00, 01, ..., 53
`%W`	Week number of the year (Monday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Monday are considered to be in week 0.	00, 01, ..., 53
`%c`	Locale’s appropriate date and time representation.	Tue Aug 16 21:30:00 1988 (en_US); Di 16 Aug 21:30:00 1988 (de_DE)
`%x`	Locale’s appropriate date representation.	08/16/88 (None); 08/16/1988 (en_US); 16.08.1988 (de_DE)
`%X`	Locale’s appropriate time representation.	21:30:00 (en_US); 21:30:00 (de_DE)
`%%`	A literal '%' character.	%

Log weight¶

If you want Mirage to select some logs more often and some less, you can give certain log files more weight. Weight means how much more or less a log file is selected, compared to others.

To change the weight, add a number at the beginning of the log file name. This number creates a ratio with the other log files. If a file does not begin with a number, Mirage considers it to be a 1.

Example

5-example1.log
2-example2.log
3-example3.log
example4.log

Mirage would send these logs in a 5:2:3:1 ratio.