Database Decommissioning Checlist

Purpose

This checklist is to ensure that all aspectssteps are taken to ensure that deleting a service database does not cause any disruption in other parts of a new service are provisioned properly, completely, and in the correct order to prevent potential failures elsewhere in the system.infrastructure.

Steps

• Determine any potential impact to any other services; see things to look out for below
  • IsAre any services dependent on this servicedatabase, going to be running on app-01directly or a different host?indirectly?
  • Is itthe goingdatabase tobacked utilizeup? SSO auth?
    • Is it going to needthere a database?specific Serviceborg filesrepo folder in /mnt/data/services on app-01?
• Is it going to need any other secrets?

Doesfor this service need to be monitored? Does this service's data need to be backed up? Exposed to the public internet? Send pings to HealthChecks? Utilizing a mailserver or ntfy to send notifications?database? Determine the most feasible deployment method

Docker container nixOS module (preferred for reproducibility and programmatic configuration)

CheckEnsure onthis repology.orgdatabase's toentry verifyis ifdeleted for borgmatic, otherwise the nixOSauto modulebackup isservice upwill toerror date with upstream before choosing to use the nixOS moduleout

• IfEnsure that the servicedatabase's usessecrets aare databasedeleted everywhere
  • Typically,Delete Idatabase createsecrets databasesfrom whoseBitwarden namesSecrets areManager after 1 year in the sameevent asthat theaccess serviceis nameneeded e.g.again
  for
• Delete forgejo,entries of the database namefrom isany forgejohost_vars

Create and store database secrets under Bitwarden Secrets Manager, using the following naming convention: webservices.<service name>.db_pass Create a new branchfiles in Projects/ansibleansible, to stop it from staging using the naming convention databases.<service name> Add an entry under the appropriate database server's host_vars file. Run the appropriate playbook to auto-provision that databasere-provisioning If this service needs a folder in the filesystem, create that respective folder under /mnt/data/services and create a symlink to it from /srv such that the symlink's location is /srv/<service name> If this service is being provisioned using Docker

Follow Docker service provisioning procedure to create a docker-compose file and appropriate config and .env files If this service is being provisioned using a nixOS module

Write wrapper module as needed  Add the database secrets to the SOPS config. This will be used for both database access and borgmatic backups Add a SOPS secrets config block to the host config running the service and pass to the module to prevent the secret being exposed in the store If this service needs to send pings to HealthChecks

Create a ping in the HealthChecks service Configure the service to contact the specific endpoint provisioned with the check If this service needs to send notifications using ntfy

Create a channel if needed and set permissions Create a token with the appropriate permissions, save it to SOPS if needed Configure the service with the token If this service needs to send email notifications

Configure the email server and run a test using the SMTP relay credentials If this service has metrics that need to be scraped

Set a port that the grafana-alloy collector can poll Add an alloy configuration under the service's module If this service needs to be monitored for uptime

Determine if this service has a specific health/up monitoring endpoint  Add a check for uptime-kuma but do not engage yet Test service on a separate testing VM if needed When the service is ready to deploy, push to staging  Test that the nixOS config builds and starts successfully, or the docker-compose project starts normally Enable uptime-kuma service monitor

Vikunja Copy-Paste Version