Skip to content

ASAB Maestro descriptor

Descriptors are YAML files living in the Library. Each application consists of a group of descriptors /Site/<application name>/Descriptors/.

A descriptor provides detailed infomation about the service and/or technology. Descriptors serve as specific extensions of the model.

Note

Descriptors are provided by the authors of each application.

Structure of the descriptor

Example of /Site/ASAB Maestro/Descriptors/mongo.yaml:

define:
  type: rc/descriptor
  name: MongoDB document database
  url: https://github.com/mongodb/mongo

descriptor:
  image: library/mongo

  volumes:
    - "{{SLOW_STORAGE}}/{{INSTANCE_ID}}/data:/data/db"
    - "{{SITE}}/{{INSTANCE_ID}}/conf:/etc/mongo:ro"

  command: mongod --config /etc/mongo/mongod.conf --directoryperdb

  healthcheck:
    test: ["CMD-SHELL", 'echo "db.runCommand(\"ping\").ok" | mongosh 127.0.0.1:27017/rs0 --quiet']
    interval: 60s
    timeout: 10s
    retries: 5
    start_period: 30s

sherpas:
  init:
    image: library/mongo
    entrypoint: ["mongosh", "--nodb", "--file", "/script/mongo-init.js"]
    command: ["echo", "DONE"]
    volumes:
    - "{{SITE}}/{{INSTANCE_ID}}/script:/script:ro"
    - "{{SITE}}/{{INSTANCE_ID}}/conf:/etc/mongo:ro"
    depends_on: ["{{INSTANCE_ID}}"]
    environment:
      MONGO_HOSTNAMES: "{{MONGO_HOSTNAMES}}"

files:
  - "conf/mongod.conf": |
      net:
        bindIp: 0.0.0.0
        port: 27017
      replication:
        replSetName: rs0
  - "script/mongo-init.js"

Templating

Descriptors utilize Jinja2 templates that are expanded when a descriptor is applied.

Common parameters:

  • {{NODE_ID}}: Node identification / hostname of the host machine (node1).
  • {{SERVICE_ID}}: Service identification (i.e. mongo).
  • {{INSTANCE_ID}}: Instance identification (i.e. mongo-2).
  • {{INSTANCE_NO}}: Instance number (i.e, 2).
  • {{SITE}}: Directory with the site config on the host machine (i.e. /opt/site).
  • {{FAST_STORAGE}}: Directory with the fast storage on the host machine (i.e. /data/ssd).
  • {{SLOW_STORAGE}}: Directory with the slow storage on the host machine (i.e. /data/hdd).

Note

Other parameters can be specified within descriptors, in the model or provided by technologies.

Technologies

Not only that multiple Library files are composed into the final configuration. There are also techs playing their parts. Techs are part of ASAB Remote Control microservice and provide extra cluster configuration.

Some of them also introduce specific sections to the descriptors.

Learn more about Techs

Composability

Descriptors can be overridden in the deployment via specific configuration options or through model.

  • /Site/<application name>/Descriptors/__commons.yaml__ file of the Library is a common base for all descriptors of the application. It specifically contains entries for network mode, restart policy, logging and others.
  • The specific descriptor of the service (e.g. /Site/<application name>/Descriptors/nginx.yaml) is layered on the top of the content of the __commons__.yaml
  • Model can override the descriptor.

Merge algorithm

This composability is implemented through a merge algorithm. You'll find the same algorithm being used in multiple cases where chunks from various sources result into a functional site configuration.

Library layering

To get a full picture of the Library within ASAB Maestro, learn also about ASAB Library layering.

Sections

Section define

define:
  type: rc/descriptor
  name: <human-readable name>
  url: <URL with relevant info>

type must be rc/descriptor.

Items name and url provide information about the service and/or technology.

Section params

Specify parameters for templating of this and all other descriptors. Any parameter specified in this section can be used in the double courly brackets for Jinja2 templating.

define:
  type: rc/descriptor

params:
  MY_PARAMETER: "ABCDEFGH"

descriptor:
  environment: "{{MY_PARAMETER}}"

Section secrets

Similar to params, secrets can be also used as parameters for templating. However, their value is not specified in the descriptor but generated and stored in the Vault. You can customize the secret by specifying type and lenght. Default is "token" of 64 bytes.

define:
  type: rc/descriptor

secrets:
  MY_SECRET:
    type: token
    length: 32

descriptor:
  environment: "{{MY_SECRET}}"

Warning

Parts of the descriptor are used directly to prepare docker-compose.yaml. The section secrets can be specified in the docker-compose.yaml as well. However, this functionality of Docker Compose is omitted within ASAB Maestro and fully replaced by secrets section of the descriptor.

Section descriptor

The descriptor section is a template for a service section of the docker-compose.yaml.

Following transformations are done:

  • The Jinja2 variables are expanded.
  • The version from ../Versions/... is appended to image, if not present.
  • Specific techs performs custom transformations, these are typically marked by null.

Details about volumes

The service has following three storage location available for their persistent data:

  • {{SITE}}: the site directory (i.e. /opt/site/...)
  • {{SLOW_STORAGE}}: the slow storage (i.e. /data/hdd/...)
  • {{FAST_STORAGE}}: the fast storage (i.e. /data/ssd/...)

Each instance can create a sub-directory in any of above locations named by its instance_id.

Section files

This section specifies what files to be copied into the instance sub-directory of the site directory (i.e. /opt/site/...). Subsequently, this content can be made available to the instance container by relevant volumes entry.

List files using the following scheme:

files:
  - destination:
      source: file_name.txt

OR

files:
  - destination:
      content: |
        Multiline plain text
        that will be written into
        the destination path.

Destination

There are three possible destinations:

  • This service
  • Other service
  • ZooKeeper
1. This service

File path specified is relative to the destination in the site directory.

In example, this record in the descriptor...

files:
- script/mongo-init.js:
    source: some_source_dir/mongo-init.js

...will create /opt/site/mongo-1/script/mongo-init.js file if the INSTANCE ID of the mongo instance is mongo-1.

2. Other service

Use URL with service scheme to target the file into other service.

In example, this record in ANY descriptor of a service in model...

files:
- service://mongo/script/mongo-init.js:
    source: some_source_dir/mongo-init.js

...will create /opt/site/mongo-1/script/mongo-init.js file if mongo instance is present in the model too and its INSTANCE ID is mongo-1.

3. ZooKeeper

Use zk URL scheme to specify path in ZooKeeper where to upload the file. The file is in "managed" mode. It means that it is always updated according to the current state of the library.

files:
  - zk:///asab/library/settings/lmio-library-settings.json:
      source: asab-library/setup.json

In this example, a ZooKeeper node with path /asab/library/settings/lmio-library-settings.json will be created, or updated if it already exists.

Source

Source is a relative path to library folder assigned as /Site/<application>/Files/<service>/. E.g. for mongo service it referes to /Site/ASAB Maestro/Files/mongo/.

Source can be both file or folder. Folder path must end with trailing slash.

File source syntax abbreviations

If source is missing in the declaration it shares the same path with the destination.

This entry copies the /Site/ASAB Maestro/Files/mongo/script/mongo-init.js into /opt/site/mongo-1/script/mongo-init.js, assuming the instance identification is mongo-1:

files:
  - "script/mongo-init.js"

The similar entry with trailing / copies the whole directory from /Site/ASAB Maestro/Files/mongo/script/conf into /opt/site/mongo-1/conf/ directory.

files:
  - "conf/"

Files are not templated

Unlike descriptors and model, files stored in /Site/<application name>/Files/<service_id>/ directory are not templated. That means that curly brackets with parameters are not replaced by the respective values. If you need to use templating within the file, enter the file into the decriptor directly, using the multiline string operator ("|").

Content

Declare the content of the file directly in the descriptor. This is especially convenient for short files and/or files that require parameters provided by maestro.

Literal style using pipe (|) in the yaml file enables writing multiline strings (block scalars).

files:
  - "conf/mongod.conf":
      content: |
        net:
          bindIp: 0.0.0.0
          port: 27017
        replication:
          replSetName: rs0

The content keyword can be omitted for bravity.

files:
  - "conf/mongod.conf": |
      net:
        bindIp: 0.0.0.0
        port: 27017
      replication:
        replSetName: rs0

Section sherpas

The sherpas are auxiliary containers that are launched together with the main instance containers. Sherpa containers are expected to finish relativelly quickly and they are not restarted. Sherpas exited sucessfuly (with exit code being 0) are promptly deleted.

Example:

sherpas:
  init:
    image: library/mongo
    entrypoint: ["mongosh", "--nodb", "--file", "/script/mongo-init.js"]
    command: ["echo", "DONE"]
    volumes:
    - "{{SITE}}/{{INSTANCE_ID}}/script:/script:ro"
    - "{{SITE}}/{{INSTANCE_ID}}/conf:/etc/mongo:ro"
    depends_on: ["{{INSTANCE_ID}}"]
    environment:
      MONGO_HOSTNAMES: "{{MONGO_HOSTNAMES}}"

This defines an init sherpa container. The container name of the sherpa would be mongo-1-init, the INSTANCE_ID parameter remains mongo-1.

The content of the sherpa is a template for a relevant part of the docker-compose.yaml.

If sherpa does not specify the image, the service image including the version is used. Alternatively, we recommend to use docker.teskalabs.com/asab/asab-governator:stable as an image for a sherpa, since this image is always present and doesn't need to be downloaded.