Service Provisioning Checlist

Purpose

This checklist is to ensure that all aspects of a new service are provisioned properly, completely, and in the correct order to prevent potential failures elsewhere in the system.

Steps

• Determine any potential impact to any other services; see things to look out for below
  • Is this service going to be running on app-01 or a different host?
  • Is it going to utilize SSO auth?
  • Is it going to need a database? Service files folder in /mnt/data/services on app-01?
  • Is it going to need any other secrets?
  • Does this service need to be monitored?
  • Does this service's data need to be backed up?
  • Exposed to the public internet?
  • Send pings to HealthChecks?
  • Utilizing a mailserver or ntfy to send notifications?
• Determine the most feasible deployment method
  • Docker container
  • nixOS module (preferred for reproducibility and programmatic configuration)

Check on repology.org to verify if the nixOS module is up to date with upstream before choosing to use the nixOS module

• If the service uses a database
  • Typically, I create databases whose names are the same as the service name e.g. for forgejo, the database name is forgejo 
  • Create and store database secrets under Bitwarden Secrets Manager, using the following naming convention: webservices.<service name>.db_pass
  • Create a new branch in Projects/ansible from staging using the naming convention databases.<service name>
  • Add an entry under the appropriate database server's host_vars file. Run the appropriate playbook to auto-provision that database
• If this service needs a folder in the filesystem, create that respective folder under /mnt/data/services and create a symlink to it from /srv such that the symlink's location is /srv/<service name>
• If this service is being provisioned using Docker
  • Follow Docker service provisioning procedure to create a docker-compose file and appropriate config and .env files
• If this service is being provisioned using a nixOS module
  • Write wrapper module as needed 
  • Add the database secrets to the SOPS config. This will be used for both database access and borgmatic backups
  • Add a SOPS secrets config block to the host config running the service and pass to the module to prevent the secret being exposed in the store
• If this service needs to send pings to HealthChecks
  • Create a ping in the HealthChecks service
  • Configure the service to contact the specific endpoint provisioned with the check
• If this service needs to send notifications using ntfy
  • Create a channel if needed and set permissions
  • Create a token with the appropriate permissions, save it to SOPS if needed
  • Configure the service with the token
• If this service needs to send email notifications
  • Configure the email server and run a test using the SMTP relay credentials
• If this service has metrics that need to be scraped
  • Set a port that the grafana-alloy collector can poll
  • Add an alloy configuration under the service's module
• If this service needs to be monitored for uptime
  • Determine if this service has a specific health/up monitoring endpoint 
  • Add a check for uptime-kuma but do not engage yet
• Test service on a separate testing VM if needed
• When the service is ready to deploy, push to staging 
• Test that the nixOS config builds and starts successfully, or the docker-compose project starts normally
• Enable uptime-kuma service monitor

Vikunja Copy-Paste Version