598f74c26b
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
50 lines
2.2 KiB
Bash
50 lines
2.2 KiB
Bash
#!/bin/bash
|
|
|
|
# Stalwart upgrade verifier.
|
|
# ---------------------------------------------------------------------------
|
|
# Stalwart publishes Kubernetes-style probes on its admin port (8080):
|
|
# GET /healthz/live — the process is alive and not deadlocked
|
|
# GET /healthz/ready — dependencies initialised, config loaded, accepting traffic
|
|
# https://stalw.art/docs/http/overview/
|
|
#
|
|
# Readiness is the one that matters after a version move: it only answers 200
|
|
# once the storage backend is open and the config has loaded, which is exactly
|
|
# the window where a schema change would otherwise go unnoticed. Liveness alone
|
|
# would pass on a process that is up but unable to serve.
|
|
#
|
|
# Deliberately weaker than the Nextcloud verifier: Stalwart's probes confirm the
|
|
# server is serving, but do not report a version, so this asserts readiness
|
|
# rather than "running exactly $expected". Stated plainly instead of implied —
|
|
# the ladder is only ever as strong as the check underneath it, and pretending
|
|
# otherwise is how a half-migrated app advances a rung.
|
|
|
|
# stalwart_upgrade_verify <app> <expected-tag> <deadline-epoch>
|
|
# 0 only when /healthz/ready answers 200 and keeps answering it.
|
|
stalwart_upgrade_verify() {
|
|
local app="$1" expected="$2" deadline="$3"
|
|
local stable=0 stable_needed=3 last=""
|
|
|
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
|
local code
|
|
code="$(runFileOp docker exec stalwart-service curl -fsS -o /dev/null -w '%{http_code}' \
|
|
--max-time 5 http://localhost:8080/healthz/ready 2>/dev/null | tr -d '\r')"
|
|
last="healthz/ready=${code:-none}"
|
|
|
|
if [ "$code" = "200" ]; then
|
|
stable=$((stable + 1))
|
|
# Ready must HOLD: a server that flaps ready/not-ready is mid-restart,
|
|
# and one lucky 200 is not evidence the upgrade settled.
|
|
if (( stable >= stable_needed )); then
|
|
isSuccessful "Stalwart is ready (readiness probe stable) after moving to $expected."
|
|
return 0
|
|
fi
|
|
else
|
|
stable=0
|
|
fi
|
|
sleep 5
|
|
done
|
|
|
|
isError "Stalwart did not report ready for $expected before the deadline.${last:+ Last probe: $last}"
|
|
return 1
|
|
}
|