Mankomail

Upgrades ​

Upgrading Mankomail means replacing the application image and restarting it. The database schema follows on its own: migrations run when the application starts. This page describes how that works, the procedure, and how to come back to the previous version.

How migrations work ​

  • They run at boot. When an application process starts, it applies the migrations its image ships and the database does not have yet, then starts serving. There is no separate migration command.
  • Under a lock. Migrations run under a PostgreSQL advisory lock. Several processes can start at the same time: one migrates, the others wait, then find nothing left to do.
  • One transaction per migration. A migration that fails is rolled back entirely, the ones before it stay applied, and the process stops with migration <file> failed in its error output.
  • Forward only. There are no "down" migrations. A database migrated by a newer version cannot be brought back to an older schema, except by restoring a backup.
  • Immutable. The database records a checksum of each applied migration. If an image ships a migration whose content differs from the one already applied, the process refuses to start (was modified after being applied).
  • No time limit. Migrations are not affected by DATABASE_STATEMENT_TIMEOUT_MS: a long migration on a large database is not cancelled.

You can follow them in the logs: one applying migration line per migration, then database schema is up to date.

DATABASE_RUN_MIGRATIONS=false turns migrations off for a process: it logs migrations are disabled and starts without migrating. Its /readyz then answers 503 with the list of pending migrations, until another process has applied them.

Before you upgrade ​

  1. Take a backup, and keep it until the new version has proven itself:

    bash
    ./scripts/backup.sh

    See Backups. The manifest records the schema version: it is the backup you restore if you roll back.

  2. Note the version you run now, from /healthz:

    bash
    curl -s localhost:3000/healthz

  3. Keep a way back to the current image: the commit or tag you built from, or the reference of the published image.

Upgrade an instance built from source ​

  1. Fetch the new version of the source code and switch to it, for example a tag:

    bash
    git fetch --tags
git checkout <version>

  2. Set APP_VERSION=<version> in .env. It becomes the version shown by /healthz and in every log line, and the one recorded in backup manifests.

  3. Build the image and recreate the containers:

    bash
    docker compose -f compose.reference.yaml up -d --build

  4. Check the upgrade.

Read the changes of compose.reference.yaml and .env.example between the two versions: a new version may add a variable or change a service. If you keep your own copy of the Compose file, report these changes into it.

Upgrade with a published image ​

If you run a published image instead of building your own:

  1. Set APP_IMAGE in .env to the reference of the new image, and APP_VERSION to its version.

  2. Pull the image and recreate the application container:

    bash
    docker compose -f compose.reference.yaml pull app
docker compose -f compose.reference.yaml up -d app

  3. Check the upgrade.

Check the upgrade ​

bash
docker compose -f compose.reference.yaml logs app | grep -E 'applying migration|schema is up to date|configuration:'
curl -s localhost:3000/healthz
curl -s localhost:3000/readyz
  • The logs show the migrations applied, then database schema is up to date.
  • /healthz returns the new version.
  • /readyz answers 200 with "status":"ready".
  • No configuration: warning appeared: a variable renamed or removed by the new version would show up here.

If the process stops on a failed migration, the container restarts and fails again in a loop. Read the error in the logs, then either fix its cause or roll back.

Several application processes ​

When you run separate api and worker processes, or several replicas:

  • Start one process of the new version first and wait for its /readyz to answer 200: the migrations are then applied. Then replace the other processes.
  • Processes that start together are safe anyway: the lock lets only one of them migrate.
  • To control precisely which process migrates, set DATABASE_RUN_MIGRATIONS=false on all the others.
  • Do not keep processes of two versions running side by side longer than the replacement takes.

Work in progress during an upgrade ​

Stopping a process does not lose work, because all the state lives in the database.

  • On SIGTERM, the process stops accepting requests, stops its scheduler, then lets the background jobs it holds finish for up to 30 seconds. Jobs still running after that are abandoned.
  • An abandoned job keeps its lease until it expires (60 seconds), then a process that runs background work (all or worker) hands it back to the queue, and it runs again.
  • Runs that wait for a delay, an approval or a signal are stored in the database. They are not affected by the restart.

Docker gives a container 10 seconds to stop by default before killing it. To let the 30-second drain finish, raise the grace period of the application service in your Compose file:

yaml
services:
  app:
    stop_grace_period: 40s

Roll back ​

Because migrations only go forward, rolling back means restoring the backup taken before the upgrade, with the previous image:

  1. Switch back to the previous version: git checkout <previous version> and rebuild, or set APP_IMAGE back to the previous image. Set APP_VERSION back too.

  2. Restore the backup taken before the upgrade:

    bash
    ./scripts/restore.sh backups/<backup taken before the upgrade>

    The script restarts the application with the image now configured.

  3. Check /healthz and /readyz.

Everything created between the backup and the rollback is lost: runs, workflow edits, tables, settings. Mailboxes catch up with their providers on their own.

WARNING

Do not start an older image on a database already migrated by a newer version. The older image applies only the migrations it knows and starts, but its code does not match the newer schema.

Upgrade PostgreSQL ​

The reference Compose file pins the major version of PostgreSQL (POSTGRES_IMAGE, postgres:16 by default). Pulling a new image of the same major version is a normal minor upgrade. A major version change cannot be done by changing the image alone: the data files of one major version cannot be read by another. Go through a dump and a restore.

Before you start, read the notes of the PostgreSQL image for the new major version: the location of the data directory inside the container may change from one major version to another, and the volume of the Compose file must match it.

  1. Stop the application, so that nothing changes after the backup: docker compose -f compose.reference.yaml stop app. Then take a backup with ./scripts/backup.sh.
  2. Stop the stack: docker compose -f compose.reference.yaml down (without -v).
  3. Remove only the PostgreSQL volume. Its name is the Compose project name followed by _postgres-data; list the volumes with docker volume ls to find it, then docker volume rm <name>.
  4. Set POSTGRES_IMAGE to the new major version in .env.
  5. Restore the backup with ./scripts/restore.sh. It starts the new PostgreSQL on an empty volume, then restores the dump into it.

Rehearse this procedure on a copy first, as described in Test a restore without touching production.