Collecting events from Google Cloud PubSub

Info

This option is available from version v23.27 onwards.

TeskaLabs LogMan.io Collector can collect events from Google Cloud PubSub using a native asynchronous consumer.

Google Cloud PubSub Documentation

Google Cloud Pull Subscription Explanation

Prerequisites

In Pub Sub, the following information need to be gathered:

1.) The name of the project the messages are to be consumed from

How to create a topic in a project

2.) the subscription name created in the topic the messages are to be consumed from

How to create a PubSub subscription

3.) Service account file with a private key to authorize to the given topic and subscription

How to create a service account

LogMan.io Collector Input setup

Google Cloud PubSub Input

The input named input:GoogleCloudPubSub: needs to be provided in the LogMan.io Collector YAML configuration:

input:GoogleCloudPubSub:GoogleCloudPubSub:
  subscription_name: <NAME_OF_THE_SUBSCRIPTION_IN_THE_GIVEN_TOPIC>
  project_name: <NAME_OF_THE_PROJECT_TO_CONSUME_FROM>
  service_account_file: <PATH_TO_THE_SERVICE_ACCOUNT_FILE>
  output: <OUTPUT>

<NAME_OF_THE_SUBSCRIPTION_IN_THE_GIVEN_TOPIC>, <NAME_OF_THE_PROJECT_TO_CONSUME_FROM> and <PATH_TO_THE_SERVICE_ACCOUNT_FILE> must be provided from the Google Clould Pub Sub

The output is events as a byte stream with the following meta information: publish_time, message_id, project_name and subscription_name.

Commit

The commit/acknowledgement is done automatically after each individual bulk of messages is processed, so the same messages are not set by PubSub repeatedly. The default bulk is 5,000 messages and can be changed in the input configuration via max_messages option:

max_messages: 10000