Overview¶
This section of the documentation is intended for individuals planning to use Deployer to mange their own deployments of CIC projects. Before starting this section, make sure to go through the Getting Started section and install the Deployer CLI.
Terminology¶
Deployer makes use of the following concepts to manage remote deployments:
Project: | A CIC software project that will be managed. Support is planned for JupyterHub, Abaco, and Agave. |
---|---|
Instance: | A physical manifestation of a project, often times for a specific environment such as “dev”, “staging”, “production”, etc. |
Tenant: | A logical isolation of a (typically, subset) of resources within an instance. |
Action: | The devops action to take against the project such as “deploy”, “update”, etc. Represents a single, atomic action to take on a set of servers; typically implemented as one or more playbooks. Actions may invoke other actions as part of execution. |
Note that a given instance is only defined within the context of a project; for example, the prod
instance of the
JupyterHub
project and the prod
instance of the Abaco
project are different. In the same way, a tenant
is only defined within a given instance.
Configs¶
Operations teams put desired project configuration in YML files ending in the .configs
extension. The config file
to be used can be specified at the command line (-c
) or in the deployment.yml file. Otherwise, the first file with a
.configs
extension in the current working directory will be used.
Each config must specify a project and can be optionally assigned to an instance or tenant. These are specified by
defining a YML key in the format <project>.<instance>.<tenant>
and then defining the properties using valid
YML within that stanza. Configs can be complex YML objects.
Consider the following example config file:
---
jupyterhub:
jupyterhub_image: taccsciapps/jupyterhub:0.8.0
jupyter_user_image: taccsciapps/jupyteruser-ds:1.1.0
use_stock_nfs: false
jupyterhub_config_py_template: jupyterhub_config_0.8.py.j2
worker_tasks_file: designsafe.yml
jupyterhub.prod.designsafe:
use_stock_nfs: true
# prod didn't define images, etc., so it should inherit from the above.
# here's an example of a config leveraging a complex YML object:
volume_mounts:
- /corral-repl/tacc/NHERI/shared/{username}:/home/jupyter/MyData:rw
- /corral-repl/tacc/NHERI/published:/home/jupyter/NHERI-Published:ro
jupyterhub.staging.designsafe:
jupyterhub_image: taccsciapps/jupyterhub:0.8.1
jupyter_user_image: taccsciapps/jupyteruser-ds:1.2.0rc9
# these image properties ^^ override the global onces defined above, but just for Designsafe staging
cull_idle_notebook_timeout: 259200
notebook_memory_limit: 3G
notebook_offer_lab: True
In the first stanza, with key jupyterhub
, a set of configuration properties and values are specified. These
values will provide a default for all actions taken on the “jupyterhub” project if the property isn’t otherwise
specified. In the next two stanzas, properties values for the “designsafe” tenant within the “prod” and “staging”
instances, respectively, are defined. These values override the values defined in the “jupyterhub” stanza when
deployment actions are taken against the particular instances.
So, when running Deployer actions against the
prod instance of the DesignSafe tenant, the jupyterhub_image
would have value taccsciapps/jupyterhub:0.8.0
(inherited from the jupyterhub
stanza) and use_stock_nfs
would have value true
. For the staging instance,
jupyterhub_image
would have value taccsciapps/jupyterhub:0.8.1
and use_stock_nfs
would have value false
.
Servers¶
Servers are the remote hosts where projects will be deployed. Deployer will ultimately execute commands over SSH on these servers to manage the deployments on behalf of the operator. The servers an operator wishes to have Deployer operate on are described in one or more server files. Here is a small example of a potential servers file:
---
ssh_user: root
ssh_key: my_ssh.key
jupyterhub.prod.sd2e:
cloud: roundup
hosts:
- host_id: manager1
ssh_host: 129.114.97.247
private_ip: 129.114.97.247
roles:
- hub
- host_id: worker1
ssh_host: 129.114.97.248
private_ip: 129.114.97.248
roles:
- worker
- host_id: worker2
ssh_host: 129.114.97.249
private_ip: 129.114.97.249
roles:
- worker
jupyterhub.staging:
cloud: jetstream
hosts:
- host_id: worker2
ssh_host: 129.114.17.47
private_ip: 10.10.10.6
roles:
- worker
In the above example, we define two global properties, ssh_user
and ssh_key
, and two groups of servers defined
in the jupyterhub.prod.sd2e
and jupyterhub.staging
stanzas. Property values are organized hierarchically just like
configuration values, and more “local” values override more “global” values.
In general, server files should conform to the following requirements:
- Written in the YML format ending in the
.hosts
extension. - Each entry in the YML file should be either a global property to apply to all servers in the file (e.g.,
ssh_key
for the SSH key Deployer should use to access to the servers, as in the above example) or a stanza containing descriptions of one or more servers. - Each host must be assigned to a project and can be optionally assigned to an instance or tenant. The project, instance
and tenant are specified through the YML key in dot notation,
<project>.<instance>.<tenant>
- Each host must define
ssh_host
, the IP address or domain of the host andhost_id
, a unique name for the host. - The
ssh_key
andssh_user
properties are also required for all hosts, though they can be inherited. Thessh_key
parameter must be a relative path from the directory where Deployer is run to a key file that can be used to access the server via SSH, and thessh_user
parameters must be a string representing the user account to SSH as. - Each host can optionally provide a list of
roles
. Project scripts useroles
to distinguish which services should run on which servers. The roles attribute must be a list. - Each host can have a additional key:value pairs defined by the operators used for further filtering (e.g. “cloud: jetstream”)
The servers file to be used can be specified at the command line (-s) or in the deployment.yml file. Otherwise, the first file with a .hosts extension in the current working directory will be used.
Passwords¶
In Deployer, password files are used to specify sensitive properties such as database passwords or the contents of an SSL certificate. The passwords file to be used can be specified at the command line (-z) or in the deployment.yml file. Otherwise, the first file with a .passwords extension in the current working directory will be used. Password files are the same as config files in all regards; for example, each password is either a global proerty or (optionally) is assigned to a project, instance or tenant and password properties can be arbitrary YML objects.
extras_dir¶
Path relative to the current working directory to a directory containing extra files needed for configuring the actions. These files could include SSL certificates, the Jupyter volume_mounts config, the Abaco service.conf, etc. When possible, actions are strongly encouraged to generate these files using configuration strings and templates instead of requiring physical files in the extras_dir.