Export Life Cycle

Each export has a state. Each state transition triggers an action.

Created

First, the export is created based on the input. Every export must be created based on a declaration. There are multiple ways to provide a declaration - as JSON or YAML file or as a reference to Library. An export ID is generated and the <export_id>.exp folder appears in the data storage. Then, the export is moved to the "extracting" state.

Extracting

Each export declaration contains a "datasource" item. This is a reference to /DataSources Library section. In the processing state, BS-Query reads the declaration of the data source from the Library and creates a datasource object - this is a specific connection to the database of choice. Each export object creates and uses one datasource object.

Query is a string variable from the export declaration saying which data to collect. Using document databases, each document is processed one by one, creating a stream, which lowers memory usage and enhances performance. Each document/record goes through transformation functions. Learn more about how to transform the exported data based on the schema. Then, it is stored in a format selected in the output section of the export declaration. Not all datasources support all output formats.

Compressing

The compressing step is optional, based on the export declaration.

Postprocessing

In this stage the export content is not edited anymore, it is shipped to selected targets, e-mail being the first of a choice. BS-Query uses ASAB Iris to send exports through e-mail.

Finished

There are two options. The export finished successfully and is ready for download or there was an exception in the export life-cycle and such export "finished with error". There are known errors that can be prevented by providing better input. Such errors are designed to provide enough information on UI. Export object that finished with an exception is stored together with the information about the error. Unknown errors are labeled with GENERAL_ERROR code.

Scheduled Exports

Scheduled exports do not follow the standard export life cycle. Instead, they end up in scheduled state when created. There are two types of scheduled exports.

One-off scheduled exports

These exports are planned for one specific moment in the future. When created, the future timestamp is stored in the schedule.json file. When this time comes, BS-Query creates the new export that inherits the declaration of the scheduled export BUT the schedule item. The original scheduled export is not needed anymore and is deleted.

Repeated scheduled exports

These exports contain a cron format string in the schedule option of the declaration. This cron format schedule is used to calculate the time of the next trigger when the new export inheriting the scheduled export's declaration is created in the same way as one-off exports. The scheduled export does not get deleted but calculates new future timestamp the cycle repeats.

Warning

Mind the scheduled export behavior when troubleshooting. A scheduled export has always at least two IDs. The one of the original scheduled export and the second one of the descendent running export. It can be even more complicated - when editing a scheduled export, it gets deleted and a new export with a new ID is created. Like this, scheduled export (as seen from the UI) has multiple IDs (being multiple export objects in BS-Query application).