Service Provisioning Checlist

Purpose

This checklist is to ensure that all aspects of a new service are provisioned properly, completely, and in the correct order to prevent potential failures elsewhere in the system.

Steps

• Determine any potential impact to any other services; see things to look out for below
  • Is this service going to be running on app-01 or a different host?
  • Is it going to utilize SSO auth?
  • Is it going to need a database? Service files folder in /mnt/data/services on app-01?
  • Is it going to need any other secrets?
  • Does this service need to be monitored?
  • Does this service's data need to be backed up?
  • Exposed to the public internet?
• Send pings to HealthChecks?
• Utilizing a mailserver or ntfy to send notifications?

Determine the most feasible deployment method

• Docker container
• nixOS module (preferred for reproducibility and programmatic configuration)

Check on repology.org to verify if the nixOS module is up to date with upstream before choosing to use the nixOS module

• If the service uses a database
  • Typically, I create databases whose names are the same as the service name e.g. for forgejo, the database name is forgejo 
  • Create and store database secrets under Bitwarden Secrets Manager, using the following naming convention: webservices.<service name>.db_pass
  • Create a new branch in Projects/ansible from staging using the naming convention databases.<service name>
  • Add an entry under the appropriate database server's host_vars file. Run the appropriate playbook to auto-provision that database

If this service needs a folder in the filesystem, create that respective folder under /mnt/data/services and create a symlink to it from /srv such that the symlink's location is /srv/<service name> If this service is being provisioned using Docker

• Follow Docker service provisioning procedure to create a docker-compose file and appropriate config and .env files

If this service is being provisioned using a nixOS module

• Write wrapper module as needed 

Add the database secrets to the SOPS config. This will be used for both database access and borgmatic backups Add a SOPS secrets config block to the host config running the service and pass to the module to prevent the secret being exposed in the store
If this service needs to send pings to HealthChecks

Create a ping in the HealthChecks service Configure the service to contact the specific endpoint provisioned with the check If this service needs to send notifications using ntfy

Create a channel if needed and set permissions Create a token with the appropriate permissions, save it to SOPS if needed Configure the service with the token If this service needs to send email notifications

Configure the email server and run a test using the SMTP relay credentials If this service has metrics that need to be scraped

Set a port that the grafana-alloy collector can poll Add an alloy configuration under the service's module If this service needs to be monitored for uptime

Determine if this service has a specific health/up monitoring endpoint  Add a check for uptime-kuma but do not engage yet Test service on a separate testing VM if needed When the service is ready to deploy, push to staging  Test that the nixOS config builds and starts successfully, or the docker-compose project starts normally Enable uptime-kuma service monitor

Vikunja Copy-Paste Version