598f74c26b
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
87 lines
3.9 KiB
Bash
87 lines
3.9 KiB
Bash
#!/bin/bash
|
|
|
|
# Upgrade verification — "did that rung actually land?"
|
|
# ---------------------------------------------------------------------------
|
|
# A stepped upgrade is only as safe as its verification. Stepping 31 -> 32 -> 33
|
|
# is arithmetic; knowing that 32 FINISHED before touching 33 is the whole ball
|
|
# game, because the dangerous state is invisible from the outside: Nextcloud
|
|
# runs its migration on boot and can sit in maintenance mode, or fail halfway,
|
|
# while Docker cheerfully reports the container healthy. Advance a rung then and
|
|
# you have skipped a migration on live data.
|
|
#
|
|
# So "the container is up" is explicitly NOT accepted as proof for an app that
|
|
# declares a verifier. Contract:
|
|
#
|
|
# <app>_upgrade_verify <app> <expected-tag> <deadline-epoch> -> 0 = verified
|
|
#
|
|
# It must poll until the deadline and return 0 ONLY when it can positively
|
|
# confirm the app is serving at the expected version with no migration
|
|
# outstanding. Any other outcome — unhealthy, indeterminate, timed out — must
|
|
# return non-zero. Uncertainty is a failure here, not a maybe: the engine
|
|
# aborts and restores rather than guessing, which is the only honest reading
|
|
# when the alternative risks someone's data.
|
|
#
|
|
# Apps with no verifier fall back to updaterVerifyGeneric, which is deliberately
|
|
# conservative and is NOT sufficient for a stepped upgrade — the engine refuses
|
|
# to ladder an app that has not declared a real one.
|
|
|
|
# Container name for an app's primary service, matching the compose convention.
|
|
_updaterPrimaryContainer() { printf '%s-service' "${1//_/-}"; }
|
|
|
|
# Generic health: running, not restarting, healthcheck (if any) reporting
|
|
# healthy, and STILL true after a settle period — a crash-loop looks perfect
|
|
# in the instant between restarts. Good enough for a single in-place update,
|
|
# never good enough to justify climbing another rung.
|
|
updaterVerifyGeneric() {
|
|
local app="$1" deadline="${3:-$(( $(date +%s) + 120 ))}"
|
|
local c; c="$(_updaterPrimaryContainer "$app")"
|
|
local stable_needed=3 stable=0
|
|
|
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
|
local state health restarts
|
|
state="$(dockerCommandRun "docker inspect --format '{{.State.Status}}' $c" 2>/dev/null | tr -d '\r')"
|
|
health="$(dockerCommandRun "docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' $c" 2>/dev/null | tr -d '\r')"
|
|
restarts="$(dockerCommandRun "docker inspect --format '{{.RestartCount}}' $c" 2>/dev/null | tr -d '\r')"
|
|
|
|
if [ "$state" = "running" ] && { [ "$health" = "healthy" ] || [ "$health" = "none" ]; }; then
|
|
stable=$((stable + 1))
|
|
[ -z "${_uv_restarts:-}" ] && _uv_restarts="$restarts"
|
|
# A restart during the settle window means it is looping, not up.
|
|
[ "$restarts" != "$_uv_restarts" ] && stable=0 && _uv_restarts="$restarts"
|
|
(( stable >= stable_needed )) && { unset _uv_restarts; return 0; }
|
|
else
|
|
stable=0
|
|
fi
|
|
sleep 5
|
|
done
|
|
unset _uv_restarts
|
|
return 1
|
|
}
|
|
|
|
# Does this app ship a real verifier? The stepped engine requires one.
|
|
updaterHasVerifier() {
|
|
local app="$1"
|
|
declare -F "${app}_upgrade_verify" >/dev/null 2>&1
|
|
}
|
|
|
|
# updaterVerifyUpgrade <app> <expected-tag> [timeout-secs]
|
|
# Dispatches to the app's verifier, falling back to the generic check. Returns
|
|
# 0 only on positive confirmation.
|
|
updaterVerifyUpgrade() {
|
|
local app="$1" expected="$2" timeout="${3:-600}"
|
|
local deadline=$(( $(date +%s) + timeout ))
|
|
|
|
if updaterHasVerifier "$app"; then
|
|
isNotice "Verifying $app is serving $expected (up to ${timeout}s)…"
|
|
if "${app}_upgrade_verify" "$app" "$expected" "$deadline"; then
|
|
isSuccessful "$app verified at $expected."
|
|
return 0
|
|
fi
|
|
isError "$app did NOT verify at $expected — treating as failed."
|
|
return 1
|
|
fi
|
|
|
|
isNotice "$app has no upgrade verifier; using the generic health check."
|
|
updaterVerifyGeneric "$app" "$expected" "$deadline"
|
|
}
|