Skip to main content

Database Decommissioning Checlist

Purpose

This checklist is to ensure that all aspectssteps are taken to ensure that deleting a service database does not cause any disruption in other parts of a new service are provisioned properly, completely, and in the correct order to prevent potential failures elsewhere in the system.infrastructure.

Steps

  • Determine any potential impact to any other services; see things to look out for below
    • IsAre any services dependent on this servicedatabase, going to be running on app-01directly or a different host?indirectly?
    • Is itthe goingdatabase tobacked utilizeup? SSO auth?
      • Is it going to needthere a database?specific Serviceborg filesrepo folder in /mnt/data/services on app-01?
      • Is it going to need any other secrets?
      • Doesfor this service need to be monitored?
      • Does this service's data need to be backed up?
      • Exposed to the public internet?
      • Send pings to HealthChecks?
      • Utilizing a mailserver or ntfy to send notifications?database?
    • Determine the most feasible deployment method
      • Docker container
      • nixOS module (preferred for reproducibility and programmatic configuration)

    CheckEnsure onthis repology.orgdatabase's toentry verifyis ifdeleted for borgmatic, otherwise the nixOSauto modulebackup isservice upwill toerror date with upstream before choosing to use the nixOS moduleout

    • IfEnsure that the servicedatabase's usessecrets aare databasedeleted everywhere
      • Typically,Delete Idatabase createsecrets databasesfrom whoseBitwarden namesSecrets areManager after 1 year in the sameevent asthat theaccess serviceis nameneeded e.g.again
        • for
      • Delete forgejo,entries of the database namefrom isany forgejohost_vars
      • Create and store database secrets under Bitwarden Secrets Manager, using the following naming convention: webservices.<service name>.db_pass
      • Create a new branchfiles in Projects/ansibleansible, to stop it from staging using the naming convention databases.<service name>
      • Add an entry under the appropriate database server's host_vars file. Run the appropriate playbook to auto-provision that databasere-provisioning
    • If this service needs a folder in the filesystem, create that respective folder under /mnt/data/services and create a symlink to it from /srv such that the symlink's location is /srv/<service name>
    • If this service is being provisioned using Docker
      • Follow Docker service provisioning procedure to create a docker-compose file and appropriate config and .env files
    • If this service is being provisioned using a nixOS module
      • Write wrapper module as needed 
      • Add the database secrets to the SOPS config. This will be used for both database access and borgmatic backups
      • Add a SOPS secrets config block to the host config running the service and pass to the module to prevent the secret being exposed in the store
    • If this service needs to send pings to HealthChecks
      • Create a ping in the HealthChecks service
      • Configure the service to contact the specific endpoint provisioned with the check
    • If this service needs to send notifications using ntfy
      • Create a channel if needed and set permissions
      • Create a token with the appropriate permissions, save it to SOPS if needed
      • Configure the service with the token
    • If this service needs to send email notifications
      • Configure the email server and run a test using the SMTP relay credentials
    • If this service has metrics that need to be scraped
      • Set a port that the grafana-alloy collector can poll
      • Add an alloy configuration under the service's module
    • If this service needs to be monitored for uptime
      • Determine if this service has a specific health/up monitoring endpoint 
      • Add a check for uptime-kuma but do not engage yet
    • Test service on a separate testing VM if needed
    • When the service is ready to deploy, push to staging 
    • Test that the nixOS config builds and starts successfully, or the docker-compose project starts normally
    • Enable uptime-kuma service monitor

    Vikunja Copy-Paste Version

Back to top