598f74c26b
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
55 lines
2.4 KiB
Bash
55 lines
2.4 KiB
Bash
#!/bin/bash
|
|
|
|
# Mastodon upgrade verifier.
|
|
# ---------------------------------------------------------------------------
|
|
# Mastodon runs Rails migrations on boot and expects releases to be applied in
|
|
# order. The web process answers /health once Puma is listening — which happens
|
|
# BEFORE migrations necessarily finish — so /health alone is not proof. The
|
|
# authoritative check is that no migration is still pending, which
|
|
# `rails db:migrate:status` reports (any row marked "down" means outstanding).
|
|
#
|
|
# Version is confirmed from the instance API rather than the image tag, so we
|
|
# are reading what the app says about itself, not what we asked for.
|
|
|
|
# mastodon_upgrade_verify <app> <expected-tag> <deadline-epoch>
|
|
# 0 only when the app serves, reports the expected version, and has no
|
|
# migration outstanding.
|
|
mastodon_upgrade_verify() {
|
|
local app="$1" expected="$2" deadline="$3"
|
|
# "v4.6" -> "4.6"; the API reports "4.6.5", so this is a prefix comparison.
|
|
local want; want="$(printf '%s' "$expected" | sed 's/^v//')"
|
|
|
|
local last=""
|
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
|
# 1. Serving at all?
|
|
local health
|
|
health="$(runFileOp docker exec mastodon-service curl -fsS --max-time 5 \
|
|
http://localhost:3000/health 2>/dev/null | tr -d '\r\n')"
|
|
|
|
if [ -n "$health" ]; then
|
|
# 2. Migrations finished? Any "down" row means Rails still has work.
|
|
local pending
|
|
pending="$(runFileOp docker exec mastodon-service bin/rails db:migrate:status 2>/dev/null \
|
|
| awk '$1 == "down" { n++ } END { print n+0 }')"
|
|
|
|
# 3. Which version does it actually report?
|
|
local ver
|
|
ver="$(runFileOp docker exec mastodon-service curl -fsS --max-time 5 \
|
|
http://localhost:3000/api/v1/instance 2>/dev/null \
|
|
| grep -oE '"version":"[^"]+"' | head -1 | cut -d'"' -f4)"
|
|
|
|
last="health=$health pendingMigrations=${pending:-?} version=${ver:-?}"
|
|
|
|
if [ "${pending:-1}" = "0" ] && [ -n "$ver" ] && [[ "$ver" == "$want"* ]]; then
|
|
isSuccessful "Mastodon reports $ver with no pending migrations."
|
|
return 0
|
|
fi
|
|
isNotice "Mastodon not ready yet: $last"
|
|
fi
|
|
sleep 10
|
|
done
|
|
|
|
isError "Mastodon did not reach a verified state for $expected before the deadline.${last:+ Last status: $last}"
|
|
return 1
|
|
}
|