feat(updater): step apps to the next version automatically, one rung a day

Two halves: the ladder could not climb the commonest versioning scheme,
and nothing ever climbed it on its own.

The ladder stepped by bumping a tag's LAST numeric component, so
v1.158.0 went v1.158.1, v1.158.2, … and never arrived at v1.159.0. It
then failed closed, refusing to build a path. Synapse publishes
v1.159.0 and no v1.158.1 at all, so Matrix could not be laddered by the
button either — three-part semver minor bumps were simply unreachable.
updaterNextRung now considers a bump of every component, keeps the
candidates that exist upstream and takes the smallest: the next release
by definition, whether it lands in the patch position or crosses into a
new major. Shape discipline is unchanged, so 31-fpm-alpine still never
becomes 31-apache, and each rung is still probed, so none can be
skipped. updaterTagBumpAt moves here from the scan, its natural home,
which also breaks a source cycle.

updaterUpgradeAuto then climbs at most ONE rung per app per calendar
day, inside the install window, for apps set to auto. One rung because a
ladder run unattended can be several migrations deep before anyone
looks, and "restore the snapshot from a minute ago" stops comforting
once four have stacked; one a day so there is time to notice. It crosses
a major if that is genuinely the next release — refusing would strand an
app on the last version of its line forever — but one step at a time,
never as a leap. Two stamps: the target rung (a failure is not retried
until something newer ships) and the day.

Every rung goes through updaterUpgradeApp unchanged, so GATE 1 still
refuses any app without a real verifier, and the per-rung contract is
identical to the button: snapshot fail-closed, set version, pull, up,
verify, restore that rung and stop on any failure. History now records
the trigger instead of hardcoding "manual", including on the rollback
paths. CFG_UPDATER_LADDER_AUTO gates the whole thing separately from
CFG_UPDATER_AUTO, because "keep my apps patched" and "move my apps
between versions unattended" are different appetites for risk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
librelad 2026-08-19 04:55:51 +01:00
parent 166acb9b7c
commit 64344bc5dc
6 changed files with 124 additions and 13 deletions

View File

@ -6,6 +6,7 @@ CFG_UPDATER_SCAN_INTERVAL=30 # App Scan Interval -
CFG_UPDATER_REGISTRY_INTERVAL=360 # Registry Check Interval - Minutes between registry lookups for new image builds. 0 = only on Check now.
CFG_HOTFIX_AUTO=security-breakage # Hotfix Auto-Apply - Which signed hotfix severities apply automatically on the update check [security-breakage|all|off]
CFG_UPDATER_AUTO=true # Automatic App Updates - Master switch for per-app automatic updates; off makes every app manual [true:On|false:Off]
CFG_UPDATER_LADDER_AUTO=true # Automatic Version Steps - Whether automatic updates may also move an app to the NEXT published version, not just rebuild the one it is on. One step per app per day, each snapshotted and verified before the next is considered, so an app is never more than one version from a state that worked. Off keeps version moves to the Upgrade button. [true:On|false:Off]
CFG_UPDATER_STALE_DAYS=365 # Unmaintained Warning After - Days without an upstream rebuild before an app is flagged as possibly unmaintained. 0 disables.
CFG_UPDATER_UPGRADE_PRUNE=true # Clean Up After Version Upgrades - After a stepped upgrade, delete the intermediate images it stepped through [true:On|false:Off]
CFG_UPDATER_WINDOW=06:00-08:00 # Automatic Update Window - When automatic updates may install, in host local time (HH:MM-HH:MM, or 'always')

View File

@ -162,3 +162,87 @@ updaterApplyAuto()
fi
return 0
}
# ---------------------------------------------------------------------------
# Automatic version LADDERING — at most one rung, at most once a day.
#
# updaterApplyAuto above moves an app WITHIN its version line (a rebuild of the
# tag it already tracks). This moves it BETWEEN lines, and it is the deliberately
# cautious half of that: it never plans a climb, only ever the single next
# published release, and it does that at most once per calendar day per app.
#
# Why one rung and not a ladder: a ladder run unattended can be several
# migrations deep before anyone looks, and "restore the snapshot from sixty
# seconds ago" stops being a comfort once four of them have stacked. One rung a
# day means the app is never more than a single version from a state that
# verified, and there is a day in which to notice. It crosses a major boundary
# if that is genuinely the next release, because refusing would strand an app
# on the last version of a line forever — but it gets there one step at a time,
# never as a leap.
#
# Every rung still goes through updaterUpgradeApp, so the contract is unchanged
# from the button: snapshot (fail-closed) -> set version -> pull -> up -> verify,
# and on any failure that rung is restored and the climb stops.
#
# Two stamps, both one-shot in different ways:
# <app>.rung the target last attempted — a rung that failed is not retried
# until a NEWER one is published (mirrors the .digest stamp).
# <app>.rungday the day a rung was last attempted — the once-a-day bound.
_updaterAutoRungStamp() { echo "$(_updaterAutoDir)/$1.rung"; }
_updaterAutoRungDay() { echo "$(_updaterAutoDir)/$1.rungday"; }
updaterUpgradeAuto()
{
[[ "${CFG_UPDATER_AUTO:-true}" == "true" ]] || return 0
# Its own switch as well as the master one: "keep my apps patched" and "move
# my apps between versions on their own" are different appetites for risk,
# and someone should be able to want the first without the second.
[[ "${CFG_UPDATER_LADDER_AUTO:-true}" == "true" ]] || return 0
command -v jq >/dev/null 2>&1 || return 0
local upd; upd="$(_updaterAutoGenDir)/updates.json"
[[ -f "$upd" ]] || return 0
updaterInWindow || return 0
local _f
for _f in cli_updater_ladder cli_updater_verify cli_updater_upgrade; do
declare -F updaterNextRung >/dev/null 2>&1 && break
source "$install_scripts_dir/cli/commands/updater/${_f}.sh" 2>/dev/null
done
declare -F updaterNextRung >/dev/null 2>&1 || return 0
local auto_dir; auto_dir="$(_updaterAutoDir)"
[[ -d "$auto_dir" ]] || runFileOp mkdir -p "$auto_dir" 2>/dev/null
local today; today="$(date +%F)"
local app channel image repo next stamp dayf enq=0
while IFS=$'\t' read -r app channel image; do
[[ -n "$app" && -n "$channel" && -n "$image" ]] || continue
[[ "$(updaterAppPolicy "$app")" == "auto" ]] || continue
# An in-flight updater task for this app: leave it alone rather than
# stacking a version move on top of an update that is still running.
updaterAutoTaskPending "$app" && continue
dayf="$(_updaterAutoRungDay "$app")"
[[ -f "$dayf" && "$(cat "$dayf" 2>/dev/null)" == "$today" ]] && continue
repo="$(updaterRepoTag "$image")"; repo="${repo%:*}"
next="$(updaterNextRung "$channel" "$repo")"
[[ -n "$next" ]] || continue
stamp="$(_updaterAutoRungStamp "$app")"
[[ -f "$stamp" && "$(cat "$stamp" 2>/dev/null)" == "$next" ]] && continue
# Stamp BEFORE enqueueing, same reasoning as the digest stamp: a crash
# between the two costs one skipped upgrade, never a loop.
printf '%s' "$next" | runFileWrite "$stamp" 2>/dev/null || true
printf '%s' "$today" | runFileWrite "$dayf" 2>/dev/null || true
cliTaskRun "libreportal updater upgrade $app $next --auto" "updater_upgrade" "$app" "--detach"
enq=$((enq + 1))
isNotice "$app: queued the next version step $channel$next (snapshotted and verified; one step only)."
done < <(jq -r '.apps[]? | select((.type // "") == "versioned")
| "\(.name)\t\(.channel // "")\t\(.current_image // "")"' "$upd" 2>/dev/null)
(( enq > 0 )) && isSuccessful "Queued $enq automatic version step(s) — one rung each, and no more today."
return 0
}

View File

@ -77,6 +77,10 @@ cliHandleUpdaterCommands()
source "$install_scripts_dir/cli/commands/updater/cli_updater_auto.sh" 2>/dev/null
fi
declare -F updaterApplyAuto >/dev/null 2>&1 && updaterApplyAuto
# App versions: step apps set to auto onto the next published
# release, one rung and once a day. Runs after the in-line updates
# so an app that just took a rebuild is already skipped as pending.
declare -F updaterUpgradeAuto >/dev/null 2>&1 && updaterUpgradeAuto
;;
"apply"|"now")
@ -107,8 +111,15 @@ cliHandleUpdaterCommands()
# and refused with "no safe path to --detach" — it failed safe, but
# blaming the version for a misplaced flag is a poor way to say
# "that flag isn't supported here".
# --auto marks a climb the updater scheduled for itself, so History
# can say so. It is the ONLY difference from the button: same
# engine, same snapshot-and-verify per rung, same rollback.
case "$upgrade_mode" in
--auto) export UPDATER_UPGRADE_TRIGGER=auto; upgrade_mode="" ;;
esac
case "$upgrade_target" in
--dry-run) upgrade_mode="--dry-run"; upgrade_target="" ;;
--auto) export UPDATER_UPGRADE_TRIGGER=auto; upgrade_target="" ;;
--*) isError "Unknown option '$upgrade_target'. Usage: libreportal updater upgrade <app> [version] [--dry-run]"; return 1 ;;
esac
for _f in cli_updater_ladder cli_updater_verify cli_updater_upgrade; do

View File

@ -127,14 +127,14 @@ updaterUpgradeApp() {
isNotice "Snapshotting $app before $rung"
if ! backupAppStart "$app" >/dev/null 2>&1; then
isError "Pre-step snapshot failed — stopping with $app on $from."
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "aborted-no-snapshot" "" "" "" "manual"
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "aborted-no-snapshot" "" "" "" "${UPDATER_UPGRADE_TRIGGER:-manual}"
return 1
fi
# 2. Move the version.
if ! updaterSetAnchorVersion "$app" "$rung"; then
isError "Could not set $app to $rung — stopping, nothing changed."
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "aborted-set-version" "" "" "" "manual"
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "aborted-set-version" "" "" "" "${UPDATER_UPGRADE_TRIGGER:-manual}"
return 1
fi
@ -152,7 +152,7 @@ updaterUpgradeApp() {
return 1
fi
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "ok" "" "" "" "manual"
updaterRecordHistory "$app" "upgrade" "$from" "$rung" "ok" "" "" "" "${UPDATER_UPGRADE_TRIGGER:-manual}"
isSuccessful "$app is verified on $rung."
from="$rung"; done_n=$((done_n + 1))
done
@ -209,12 +209,12 @@ _updaterUpgradeRollbackStep() {
updaterSetAnchorVersion "$app" "$from" || isError "Could not put $app's version back to $from — check its compose file."
if restoreAppStart "$app" latest "" >/dev/null 2>&1; then
dockerComposeUp "$app" >/dev/null 2>&1 || true
updaterRecordHistory "$app" "upgrade" "$from" "$failed" "rolled-back" "" "" "" "manual"
updaterRecordHistory "$app" "upgrade" "$from" "$failed" "rolled-back" "" "" "" "${UPDATER_UPGRADE_TRIGGER:-manual}"
isSuccessful "$app restored to $from from its pre-step snapshot."
isNotice "The ladder stopped here. Read $failed's release notes before trying again."
return 0
fi
updaterRecordHistory "$app" "upgrade" "$from" "$failed" "rollback-failed" "" "" "" "manual"
updaterRecordHistory "$app" "upgrade" "$from" "$failed" "rollback-failed" "" "" "" "${UPDATER_UPGRADE_TRIGGER:-manual}"
isError "Could not restore $app automatically. Its data snapshot is intact — restore it from the Backups page."
return 1
}

View File

@ -1029,6 +1029,8 @@ declare -gA LP_FN_MAP=(
[updaterAppPolicy]="cli/commands/updater/cli_updater_auto.sh"
[_updaterAutoDir]="cli/commands/updater/cli_updater_auto.sh"
[_updaterAutoGenDir]="cli/commands/updater/cli_updater_auto.sh"
[_updaterAutoRungDay]="cli/commands/updater/cli_updater_auto.sh"
[_updaterAutoRungStamp]="cli/commands/updater/cli_updater_auto.sh"
[_updaterAutoStamp]="cli/commands/updater/cli_updater_auto.sh"
[updaterAutoTaskPending]="cli/commands/updater/cli_updater_auto.sh"
[updaterClassifyTag]="webui/data/generators/updater/webui_updater_scan.sh"
@ -1044,6 +1046,7 @@ declare -gA LP_FN_MAP=(
[updaterNewerVersionByList]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterNewerVersionByProbe]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterNewerVersionTag]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterNextRung]="cli/commands/updater/cli_updater_ladder.sh"
[_updaterPrimaryContainer]="cli/commands/updater/cli_updater_verify.sh"
[updaterPrimaryImage]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterRecordHistory]="cli/commands/updater/cli_updater_commands.sh"
@ -1054,7 +1057,7 @@ declare -gA LP_FN_MAP=(
[updaterRollbackApp]="cli/commands/updater/cli_updater_commands.sh"
[updaterSetAnchorRef]="cli/commands/updater/cli_updater_commands.sh"
[updaterSetAnchorVersion]="cli/commands/updater/cli_updater_upgrade.sh"
[updaterTagBumpAt]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterTagBumpAt]="cli/commands/updater/cli_updater_ladder.sh"
[updaterTagExists]="cli/commands/updater/cli_updater_ladder.sh"
[updaterTagGreater]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterTagIncrement]="cli/commands/updater/cli_updater_ladder.sh"
@ -1064,6 +1067,7 @@ declare -gA LP_FN_MAP=(
[updaterTagShape]="webui/data/generators/updater/webui_updater_scan.sh"
[updaterTagSortKey]="cli/commands/updater/cli_updater_ladder.sh"
[updaterUpgradeApp]="cli/commands/updater/cli_updater_upgrade.sh"
[updaterUpgradeAuto]="cli/commands/updater/cli_updater_auto.sh"
[_updaterUpgradeGenDir]="cli/commands/updater/cli_updater_upgrade.sh"
[_updaterUpgradePruneImages]="cli/commands/updater/cli_updater_upgrade.sh"
[_updaterUpgradeRollbackStep]="cli/commands/updater/cli_updater_upgrade.sh"
@ -2180,6 +2184,8 @@ declare -gA LP_FN_ROOT=(
[updaterAppPolicy]="scripts"
[_updaterAutoDir]="scripts"
[_updaterAutoGenDir]="scripts"
[_updaterAutoRungDay]="scripts"
[_updaterAutoRungStamp]="scripts"
[_updaterAutoStamp]="scripts"
[updaterAutoTaskPending]="scripts"
[updaterClassifyTag]="scripts"
@ -2195,6 +2201,7 @@ declare -gA LP_FN_ROOT=(
[updaterNewerVersionByList]="scripts"
[updaterNewerVersionByProbe]="scripts"
[updaterNewerVersionTag]="scripts"
[updaterNextRung]="scripts"
[_updaterPrimaryContainer]="scripts"
[updaterPrimaryImage]="scripts"
[updaterRecordHistory]="scripts"
@ -2215,6 +2222,7 @@ declare -gA LP_FN_ROOT=(
[updaterTagShape]="scripts"
[updaterTagSortKey]="scripts"
[updaterUpgradeApp]="scripts"
[updaterUpgradeAuto]="scripts"
[_updaterUpgradeGenDir]="scripts"
[_updaterUpgradePruneImages]="scripts"
[_updaterUpgradeRollbackStep]="scripts"
@ -3367,6 +3375,8 @@ updaterApplyAuto() { unset -f updaterApplyAuto; __lpAutoload "${install_scripts_
updaterAppPolicy() { unset -f updaterAppPolicy; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; updaterAppPolicy "$@"; }
_updaterAutoDir() { unset -f _updaterAutoDir; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; _updaterAutoDir "$@"; }
_updaterAutoGenDir() { unset -f _updaterAutoGenDir; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; _updaterAutoGenDir "$@"; }
_updaterAutoRungDay() { unset -f _updaterAutoRungDay; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; _updaterAutoRungDay "$@"; }
_updaterAutoRungStamp() { unset -f _updaterAutoRungStamp; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; _updaterAutoRungStamp "$@"; }
_updaterAutoStamp() { unset -f _updaterAutoStamp; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; _updaterAutoStamp "$@"; }
updaterAutoTaskPending() { unset -f updaterAutoTaskPending; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; updaterAutoTaskPending "$@"; }
updaterClassifyTag() { unset -f updaterClassifyTag; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterClassifyTag "$@"; }
@ -3382,6 +3392,7 @@ updaterLastUpdateFrom() { unset -f updaterLastUpdateFrom; __lpAutoload "${instal
updaterNewerVersionByList() { unset -f updaterNewerVersionByList; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterNewerVersionByList "$@"; }
updaterNewerVersionByProbe() { unset -f updaterNewerVersionByProbe; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterNewerVersionByProbe "$@"; }
updaterNewerVersionTag() { unset -f updaterNewerVersionTag; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterNewerVersionTag "$@"; }
updaterNextRung() { unset -f updaterNextRung; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_ladder.sh"; updaterNextRung "$@"; }
_updaterPrimaryContainer() { unset -f _updaterPrimaryContainer; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_verify.sh"; _updaterPrimaryContainer "$@"; }
updaterPrimaryImage() { unset -f updaterPrimaryImage; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterPrimaryImage "$@"; }
updaterRecordHistory() { unset -f updaterRecordHistory; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_commands.sh"; updaterRecordHistory "$@"; }
@ -3392,7 +3403,7 @@ updaterRepoTag() { unset -f updaterRepoTag; __lpAutoload "${install_scripts_dir}
updaterRollbackApp() { unset -f updaterRollbackApp; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_commands.sh"; updaterRollbackApp "$@"; }
updaterSetAnchorRef() { unset -f updaterSetAnchorRef; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_commands.sh"; updaterSetAnchorRef "$@"; }
updaterSetAnchorVersion() { unset -f updaterSetAnchorVersion; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_upgrade.sh"; updaterSetAnchorVersion "$@"; }
updaterTagBumpAt() { unset -f updaterTagBumpAt; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterTagBumpAt "$@"; }
updaterTagBumpAt() { unset -f updaterTagBumpAt; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_ladder.sh"; updaterTagBumpAt "$@"; }
updaterTagExists() { unset -f updaterTagExists; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_ladder.sh"; updaterTagExists "$@"; }
updaterTagGreater() { unset -f updaterTagGreater; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterTagGreater "$@"; }
updaterTagIncrement() { unset -f updaterTagIncrement; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_ladder.sh"; updaterTagIncrement "$@"; }
@ -3402,6 +3413,7 @@ updaterTagOf() { unset -f updaterTagOf; __lpAutoload "${install_scripts_dir}webu
updaterTagShape() { unset -f updaterTagShape; __lpAutoload "${install_scripts_dir}webui/data/generators/updater/webui_updater_scan.sh"; updaterTagShape "$@"; }
updaterTagSortKey() { unset -f updaterTagSortKey; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_ladder.sh"; updaterTagSortKey "$@"; }
updaterUpgradeApp() { unset -f updaterUpgradeApp; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_upgrade.sh"; updaterUpgradeApp "$@"; }
updaterUpgradeAuto() { unset -f updaterUpgradeAuto; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_auto.sh"; updaterUpgradeAuto "$@"; }
_updaterUpgradeGenDir() { unset -f _updaterUpgradeGenDir; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_upgrade.sh"; _updaterUpgradeGenDir "$@"; }
_updaterUpgradePruneImages() { unset -f _updaterUpgradePruneImages; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_upgrade.sh"; _updaterUpgradePruneImages "$@"; }
_updaterUpgradeRollbackStep() { unset -f _updaterUpgradeRollbackStep; __lpAutoload "${install_scripts_dir}cli/commands/updater/cli_updater_upgrade.sh"; _updaterUpgradeRollbackStep "$@"; }

View File

@ -242,15 +242,18 @@ updaterNewerVersionByList() {
# turn one app's scan into a crawl.
updaterNewerVersionByProbe() {
local cur="$1" repo="$2"
# updaterTagExists lives in the ladder, and a cross-file function is not
# reliably already loaded here — same situation updaterAppPolicy handles
# below, so use the same remedy rather than assuming. Without the explicit
# source a missing function would make this whole fallback silently do
# nothing, which is precisely the failure mode it exists to remove.
if ! declare -F updaterTagExists >/dev/null 2>&1; then
# updaterTagExists and updaterTagBumpAt both live in the ladder, and a
# cross-file function is not reliably already loaded here — same situation
# updaterAppPolicy handles below, so use the same remedy rather than
# assuming. Without the explicit source a missing function would make this
# whole fallback silently do nothing, which is precisely the failure mode it
# exists to remove. Check BOTH: they arrive together, but a guard that only
# names one is a guard that stops being true the day the other moves.
if ! declare -F updaterTagExists >/dev/null 2>&1 || ! declare -F updaterTagBumpAt >/dev/null 2>&1; then
[ -f "$install_scripts_dir/cli/commands/updater/cli_updater_ladder.sh" ] \
&& source "$install_scripts_dir/cli/commands/updater/cli_updater_ladder.sh" 2>/dev/null
declare -F updaterTagExists >/dev/null 2>&1 || return 0
declare -F updaterTagBumpAt >/dev/null 2>&1 || return 0
fi
local shape; shape="$(updaterTagShape "$cur")"
local ncomp; ncomp="$(printf '%s' "$cur" | grep -oE '[0-9]+' | wc -l | tr -d ' ')"