How to continuously upload logs from the LogMan.io archive to S3 storage¶

The LogMan.io Receiver keeps an immutable copy of every raw log in its archive. The archive is your long-term store, but it does not keep data forever: by default, partitions are deleted from the cold phase after 18 months. If you need to retain raw logs longer, i.e. for compliance, forensics, or as an off-site copy, the simplest approach is to continuously upload the archive to S3-compatible object storage.

This guide describes a one-way, scheduled sync from the receiver cold phase to an S3 bucket using the AWS CLI. The same procedure works with AWS S3 and with S3-compatible providers such as Oracle Cloud Object Storage.

Why sync the archive to S3¶

The receiver archive is immutable, column-oriented, and append-only. Every received log is stored once and never modified. The archive is typically divided into lifecycle phases:

hot: recently ingested partitions, best placed on SSDs
warm: partitions after one day, best placed on HDDs
cold: the long-term copy, xz-compressed, best placed on slow HDDs or NAS

By default, a partition stays in the cold phase for 18 months and is then deleted. That is the retention you get out of the box.

Syncing the cold phase to object storage gives you three things:

Longer retention. The bucket becomes your archive beyond the 18-month default.
An off-site copy. The data lives in a different location than the cluster.
Cheap storage. Object storage with a STANDARD_IA or Glacier lifecycle is cheaper than keeping the same volume on cluster disks.

The archive design supports this directly. Partition names are globally unique, so partitions can be moved to shared or cloud storage and reassembled correctly without collisions.

How it works¶

The core is a single command:

aws s3 sync source_folder s3://your_bucket_name/destination_folder/

aws s3 sync is a one-way, additive transfer. It uploads files that are missing at the destination, differ in size, or are newer locally. Files that you delete locally are never removed from the bucket, as long as you do not add the --delete flag.

This is exactly the behavior you want for an archive backup: the bucket only ever grows, and an operator mistake on the local side cannot destroy the copy.

The sync runs on a schedule (for example every ten minutes from cron), so new partitions are uploaded soon after they close and move into the cold phase.

flowchart LR
  subgraph lmio [LogMan.io node]
    cold[lmio-receiver<br/>cold phase<br/>/data/hdd/lmio-receiver-1/cold]
  end

  subgraph cli [AWS CLI container]
    sync[aws s3 sync]
  end

  subgraph storage [Object storage]
    bucket[S3 bucket<br/>s3://bucket/tenant/]
  end

  cold -- "cron, every 10 min" --> sync
  sync --> bucket

Prerequisites¶

A LogMan.io deployment with a receiver archive and a defined cold phase directory
Superuser (root) access to the node where the receiver stores the cold phase
An S3-compatible account with a bucket, AWS S3 or any provider that supports the S3 API
The AWS CLI, preferably the official Docker image amazon/aws-cli

1. Create the bucket and the IAM user¶

Create a bucket for the archive. Recommended bucket settings:

Versioning: on: every re-upload of a changed file keeps the prior version, protecting against overwrites as well as deletions.
Block Public Access: on (default for new buckets).
Default encryption: SSE-S3 (on by default for new buckets).
Lifecycle rule: for archival content, transition to Glacier Instant Retrieval after 30 days, and expire noncurrent versions after 90 days so versioning does not grow without bound.

Create an IAM user with no console access, then attach this inline policy. Delete actions are deliberately absent, so the no-delete guarantee holds even if someone later adds --delete to the command by mistake.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::your_bucket_name"
    },
    {
      "Sid": "WriteObjects",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::your_bucket_name/*"
    }
  ]
}

Then create an access key: Security credentials → Create access key → "Application running outside AWS". The secret is shown only once.

2. Store credentials in the Vault¶

On a node that runs ASAB Remote Control, upload the IAM access key values from step 1. Use /opt/site/gov.sh curl so the call is authorized locally (see Using the Vault from the command line):

/opt/site/gov.sh curl -X PUT localhost:8891/vault/AWS_ACCESS_KEY_ID --data 'AKIA...'
/opt/site/gov.sh curl -X PUT localhost:8891/vault/AWS_SECRET_ACCESS_KEY --data 'wJalr...'

3. Deploy the AWS CLI¶

Add the AWS CLI service to model.yaml. Reference the access key and secret from the Vault, do not put them in the model as plain text.

/Site/model.yaml

services:
  aws-cli:
    instances:
      1:
        node: <node-1>
      2:
        node: <node-2>

    descriptor:
      environment:
        AWS_ACCESS_KEY_ID: "{{AWS_ACCESS_KEY_ID}}"
        AWS_SECRET_ACCESS_KEY: "{{AWS_SECRET_ACCESS_KEY}}"
        AWS_DEFAULT_REGION: eu-central-1

secrets:
  AWS_ACCESS_KEY_ID: {}
  AWS_SECRET_ACCESS_KEY: {}

Apply the model and wait for the aws-cli service to start. The Vault entries from step 2 must already exist; if a declared secret is missing when you apply, Maestro generates a random token and stores it in the Vault, not a valid AWS key.

4. Verify credentials¶

Verify the credentials before the first sync:

docker exec aws-cli-1 aws sts get-caller-identity
docker exec aws-cli-1 aws s3 ls s3://your_bucket_name/

5. Run the sync¶

The source is the cold phase directory of the receiver. The destination prefix should identify the tenant, so one bucket can serve the whole cluster:

docker exec aws-cli-1 aws s3 sync /data/hdd/lmio-receiver-<instance>/cold/ s3://your_bucket_name/<tenant>/

<instance> is typically 1 on a single-node installation, or the node number in a cluster.
<tenant> is the unique name of the tenant whose logs you are uploading.

Preview first, this lists what would transfer and moves nothing:

docker exec aws-cli-1 aws s3 sync /data/hdd/lmio-receiver-1/cold/ s3://your_bucket_name/tenant/ --dryrun

Useful flags:

Flag	Effect
`--dryrun`	Show planned operations only
`--exclude '.DS_Store' --exclude '*.tmp'`	Skip noise (repeatable)
`--storage-class STANDARD_IA`	Cheaper storage for archival content
`--only-show-errors`	Quiet output for cron
`--exact-timestamps`	Stricter change detection (downloads)

Never add --delete

Never add --delete, that is the one flag that would mirror local deletions into the bucket.

How change detection works¶

sync compares file size and last-modified time. A file is uploaded when it is missing at the destination, differs in size, or is newer locally. It does not compare checksums, so a modified file that keeps its exact byte size and an older mtime than the S3 object will be skipped. In practice this is rare; touch the file to force it if needed.

6. Schedule the sync¶

Run the sync every ten minutes from cron. Use the absolute path to docker (cron has a minimal PATH) and log the output:

*/10 * * * * /usr/bin/docker exec aws-cli-1 aws s3 sync /data/hdd/lmio-receiver-1/cold/ s3://your_bucket_name/tenant/ --only-show-errors >> /var/log/lmio-sync.log 2>&1

Add one line per LMIO Receiver instance. The sync is incremental: after the first run it transfers only new or changed partitions, so each run is cheap.

7. Verify¶

After the first run, check that objects are appearing in the bucket:

docker exec aws-cli-1 aws s3 ls --recursive s3://your_bucket_name/tenant/ | tail

Compare the object count with the number of partitions in the cold phase. The cold phase holds one directory per partition, so a successful sync means every partition directory is present at the destination.

Non-AWS S3 targets¶

The same command works against S3-compatible providers, for example Oracle Cloud Object Storage. Keep credentials in the Vault as in steps 2–3; extend the aws-cli service environment from step 3 with the region and checksum settings (these are not secrets, they must be set in the container, not as export on the host):

/Site/model.yaml

    descriptor:
      environment:
        AWS_ACCESS_KEY_ID: "{{AWS_ACCESS_KEY_ID}}"
        AWS_SECRET_ACCESS_KEY: "{{AWS_SECRET_ACCESS_KEY}}"
        AWS_DEFAULT_REGION: <region>
        AWS_REQUEST_CHECKSUM_CALCULATION: when_required
        AWS_RESPONSE_CHECKSUM_VALIDATION: when_required

Apply the change, then pass the provider endpoint on the sync command:

docker exec aws-cli-1 aws s3 sync /data/hdd/lmio-receiver-1/cold/ s3://bucket/tenant/ \
  --endpoint-url https://<namespace>.compat.objectstorage.<region>.oraclecloud.com \
  --region <region>

Two caveats for this path:

The two checksum variables in environment are required for AWS CLI 2.23.5+, which otherwise sends CRC checksums that OCI rejects with a misleading The Content-Length header is required error.
Credentials there come from Customer Secret Keys in the OCI console, not from IAM.

Conclusion¶

Syncing the LogMan.io archive to S3-compatible object storage is a single aws s3 sync command, a small IAM policy without delete permissions, and one cron line per receiver instance. The result is a cheap, durable, off-site copy of every raw log that outlives the receiver's native retention. The archive is immutable and partition names are globally unique, so the bucket is a faithful copy of the cold phase, and a local mistake can never propagate deletions into it.