Service Provisioning Checlist

Purpose

This checklist is to ensure that all aspects of a new service are provisioned properly, completely, and in the correct order to prevent potential failures elsewhere in the system.

Steps

• Determine any potential impact to any other services; see things to look out for below
  • Is this service going to be running on app-01 or a different host?
  • Is it going to utilize SSO auth?
  • Is it going to need a database? Service files folder in /mnt/data/services on app-01?
  • Is it going to need any other secrets?
  • Does this service need to be monitored?
  • Does this service's data need to be backed up?
  • Exposed to the public internet?
  • Utilizing a mailserver or ntfy to send notifications?
• Determine the most feasible deployment method
  • Docker container
  • nixOS module (preferred for reproducibility and programmatic configuration)

Check on repology.org to verify if the nixOS module is up to date with upstream before choosing to use the nixOS module

• If the service uses a database
  • Typically, I create databases whose names are the same as the service name e.g. for forgejo, the database name is forgejo 
  • Create and store database secrets under Bitwarden Secrets Manager, using the following naming convention: webservices.<service name>.db_pass
  • Create a new branch in Projects/ansible from staging using the naming convention databases.<service name>
  • Add an entry under the appropriate database server's host_vars file. Run the appropriate playbook to auto-provision that database
  • If this service needs a folder in the filesystem, create that respective folder under /mnt/data/services and create a symlink to it from /srv such that the symlink's location is /srv/<service name>
  • If this service is being provisioned using Docker
    • Follow Docker service provisioning procedure to create a docker-compose file and appropriate config and .env files
  • If this service is being provisioned using a nixOS module
    • Add the database secrets to the SOPS config. This will be used for both database access and borgmatic backups
    • Add a SOPS secrets config block and pass to the module to prevent the secret being exposed in the store
      

Vikunja Copy-Paste Version

• Shutdown/disable needed monitoring services
• Remove/disable borgmatic database backup entry from nixOS to prevent borgmatic failure
• Teardown compose project/remove nixOS service config, push change to staging
  • If docker-compose project, move to ~/Containers/.retired-services
• Remove service's entry from traefik's acme.json file to prevent unwanted cert renewals
• If unneeded, clear all remaining files from the filesystem i.e. /srv/<servicename> and any relevant databases and secrets
• Push changes fromstaging to main 
• Re-enable monitoring systems as needed