64ff5f508b
GATE 1 refuses to ladder an app that cannot prove a rung landed, and only mastodon, nextcloud and stalwart could. None of those are installed here, so the stepped upgrade — button or automatic — was unreachable for every app on the box. Three fixes. _updaterPrimaryContainer assumed the container is "<app>-service". It is a convention, not a rule: matrix names its anchor service matrix-synapse and stoat names its api (container stoat-api). The verifier therefore inspected a container that does not exist, saw no state, and could only time out — on exactly the stateful apps that most need verifying. It now reads the anchor service's container_name from the compose, buffering per service block because container_name may sit either side of the image line. Added updaterVerifyHttpVersion: poll the app over its PUBLISHED port from the host, pull the version from a JSON field or a response header, and require agreement three polls running. Probed from the host rather than `docker exec … curl` because half these images ship no curl at all (mattermost is one), so exec-based probing is a coin flip on the vendor's base image. Version comparison matches only the components both sides state, since tags and self-reported builds rarely share precision: v1.158.0 vs 1.158.0, 11.9 vs 11.9.1, 8.7.0 vs 8.7 all agree; 11.9 vs 11.10 does not. Each app hook is then three facts. Verified live: all three confirm at the version they are actually on, and all three REFUSE a version they are not — which is the property that makes stepping them safe. updaterUpgradeAuto now skips apps with no verifier instead of queueing a task that GATE 1 will reject, which would otherwise mean a failure notification every day for an app that was never eligible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
256 lines
13 KiB
Bash
256 lines
13 KiB
Bash
#!/bin/bash
|
|
|
|
# Automatic app updates — policy resolution + the auto-apply enqueuer.
|
|
# ---------------------------------------------------------------------------
|
|
# The decision half of the updater: `webui_updater_scan.sh` answers "is a new
|
|
# build available?", `updaterApplyApp` answers "how do I install it safely?",
|
|
# and this file answers "should I install it without being asked?".
|
|
#
|
|
# Policy is per app, with a fleet-wide master switch and a daily install window:
|
|
# CFG_<APP>_UPDATE_TYPE = auto | manual (per app, default auto)
|
|
# CFG_UPDATER_AUTO = true | false (master switch, default true)
|
|
# CFG_UPDATER_WINDOW = HH:MM-HH:MM (host local time, default 06:00-08:00
|
|
# — right after the 05:00 backups;
|
|
# 'always' = any time)
|
|
# Precedence mirrors the established backup-strategy shape (per-app override on
|
|
# top of a global default): the master switch can only ever make things MORE
|
|
# manual, so "turn auto-updates off" is one flip, not 33 config edits.
|
|
#
|
|
# What it deliberately does NOT do:
|
|
# * It never applies anything inline. Like artifactApplyAuto, it only enqueues
|
|
# the ordinary `updater_apply` task, so an automatic update is the exact
|
|
# same code path (snapshot -> pull -> up -> auto-rollback on failure), with
|
|
# the same task log, the same History entry, and the same Roll back button
|
|
# as a hand-pressed Update. The only difference is who pressed it, which is
|
|
# recorded as the history entry's `trigger`.
|
|
# * It never retries a build that already failed. Each auto-attempt stamps the
|
|
# target digest under generated/auto/<app>; the same digest is skipped
|
|
# forever after. Without this a broken upstream build would be re-attempted
|
|
# (and rolled back) on every scan, snapshotting the app each time. A newer
|
|
# build changes the digest and is attempted normally; the Update button
|
|
# stays available for a manual retry of the skipped one.
|
|
|
|
_updaterAutoGenDir() { echo "${containers_dir%/}/libreportal/frontend/data/updater/generated"; }
|
|
_updaterAutoDir() { echo "$(_updaterAutoGenDir)/auto"; }
|
|
_updaterAutoStamp() { echo "$(_updaterAutoDir)/$1.digest"; } # $1=app
|
|
|
|
# Is the automatic-update window open right now? CFG_UPDATER_WINDOW is
|
|
# HH:MM-HH:MM in the HOST's local time (the same clock cron and the daemon run
|
|
# on — deliberately NOT CFG_TIMEZONE, which only sets the containers' TZ), or
|
|
# 'always'. start > end crosses midnight (22:00-02:00); start == end means the
|
|
# window never closes. Scans are not gated by this — only the enqueue is, so
|
|
# detection stays current all day and pending updates simply wait.
|
|
# Malformed values FAIL CLOSED (hold updates): the user typed something trying
|
|
# to restrict when updates land, and ignoring it would violate that intent —
|
|
# the Updates page shows the raw window value, so a typo is visible, and the
|
|
# WebUI validator rejects bad values before they ever land in the config.
|
|
updaterInWindow()
|
|
{
|
|
local win="${CFG_UPDATER_WINDOW:-always}"
|
|
[[ "$win" == "always" ]] && return 0
|
|
[[ "$win" =~ ^([01][0-9]|2[0-3]):([0-5][0-9])-([01][0-9]|2[0-3]):([0-5][0-9])$ ]] || return 1
|
|
local now s e
|
|
now=$(( 10#$(date +%H) * 60 + 10#$(date +%M) ))
|
|
s=$(( 10#${BASH_REMATCH[1]} * 60 + 10#${BASH_REMATCH[2]} ))
|
|
e=$(( 10#${BASH_REMATCH[3]} * 60 + 10#${BASH_REMATCH[4]} ))
|
|
if (( s < e )); then
|
|
(( now >= s && now < e ))
|
|
else
|
|
(( now >= s || now < e )) # wraps midnight; s == e -> whole day
|
|
fi
|
|
}
|
|
|
|
# Effective update policy for one app -> "auto" | "manual".
|
|
# 1. master switch off -> manual (everything, no exceptions)
|
|
# 2. per-app CFG_<APP>_UPDATE_TYPE
|
|
# 3. unset / unrecognised -> auto (the documented default)
|
|
# App configs are sourced globally (sourceScanFiles app_configs), so the per-app
|
|
# key is read by indirect expansion exactly like backupResolveStrategy does.
|
|
updaterAppPolicy()
|
|
{
|
|
local app="$1"
|
|
[[ -n "$app" ]] || { echo manual; return 0; }
|
|
[[ "${CFG_UPDATER_AUTO:-true}" == "true" ]] || { echo manual; return 0; }
|
|
local key="CFG_${app^^}_UPDATE_TYPE"
|
|
case "$(printf '%s' "${!key-}" | tr 'A-Z' 'a-z')" in
|
|
manual|off|false) echo manual ;;
|
|
*) echo auto ;;
|
|
esac
|
|
}
|
|
|
|
# True when an updater task for this app is already queued or in flight, so a
|
|
# scan that lands while the previous update is still running doesn't stack a
|
|
# second one. Cheap: one jq over the task files, and only for apps that got
|
|
# this far (update available + policy auto).
|
|
updaterAutoTaskPending()
|
|
{
|
|
local app="$1" dir="${containers_dir%/}/libreportal/frontend/data/tasks"
|
|
command -v jq >/dev/null 2>&1 || return 1
|
|
local files=( "$dir"/task_*.json )
|
|
[[ -e "${files[0]}" ]] || return 1
|
|
local n
|
|
n="$(jq -rs --arg a "$app" '
|
|
[ .[] | select((.app // "") == $a
|
|
and ((.type // "") | startswith("updater_"))
|
|
and ((.status // "") == "queued" or (.status // "") == "pending"
|
|
or (.status // "") == "running")) ] | length
|
|
' "${files[@]}" 2>/dev/null)"
|
|
[[ "${n:-0}" -gt 0 ]]
|
|
}
|
|
|
|
# updaterApplyAuto — enqueue an `updater apply` task for every app that has an
|
|
# update available and is set to auto. Called from `updater check` right after
|
|
# the scan that produced updates.json, so it always acts on fresh data.
|
|
updaterApplyAuto()
|
|
{
|
|
if [[ "${CFG_UPDATER_AUTO:-true}" != "true" ]]; then
|
|
isNotice "Automatic app updates are off (CFG_UPDATER_AUTO=false)."
|
|
return 0
|
|
fi
|
|
if ! command -v jq >/dev/null 2>&1; then
|
|
isNotice "Automatic app updates need jq to read the scan results — skipping (updates are still listed in the WebUI)."
|
|
return 0
|
|
fi
|
|
local upd; upd="$(_updaterAutoGenDir)/updates.json"
|
|
[[ -f "$upd" ]] || return 0
|
|
|
|
# The window gates the ENQUEUE only — the scan that got us here already ran.
|
|
# Quiet unless something is actually waiting, so the daemon's half-hourly
|
|
# no-op scans don't fill the log with window notices.
|
|
if ! updaterInWindow; then
|
|
local n_waiting
|
|
n_waiting="$(jq -r '[.apps[]? | select(.update_available == true)] | length' "$upd" 2>/dev/null)"
|
|
[[ "${n_waiting:-0}" -gt 0 ]] && \
|
|
isNotice "$n_waiting update(s) are waiting for the automatic update window (${CFG_UPDATER_WINDOW:-always})."
|
|
return 0
|
|
fi
|
|
|
|
local auto_dir; auto_dir="$(_updaterAutoDir)"
|
|
[[ -d "$auto_dir" ]] || runFileOp mkdir -p "$auto_dir" 2>/dev/null
|
|
|
|
local app dig stamp enqueued=0 skipped=0
|
|
while IFS=$'\t' read -r app dig; do
|
|
[[ -n "$app" ]] || continue
|
|
[[ "$(updaterAppPolicy "$app")" == "auto" ]] || continue
|
|
|
|
# An update we already queued may still be waiting its turn — that is
|
|
# in-flight, not stuck, so it must be tested BEFORE the attempted-digest
|
|
# stamp (which the enqueue already wrote) or it would be miscounted as a
|
|
# failure below.
|
|
updaterAutoTaskPending "$app" && continue
|
|
|
|
# Already auto-attempted this exact build? (failed last time — see header)
|
|
stamp="$(_updaterAutoStamp "$app")"
|
|
if [[ -n "$dig" && -f "$stamp" ]] && [[ "$(cat "$stamp" 2>/dev/null)" == "$dig" ]]; then
|
|
skipped=$((skipped + 1))
|
|
continue
|
|
fi
|
|
|
|
# Stamp BEFORE enqueueing: if the update fails and rolls back, the stamp
|
|
# is what stops the next scan from trying the same build again. A crash
|
|
# between stamp and enqueue costs one skipped auto-update, never a loop.
|
|
printf '%s' "$dig" | runFileWrite "$stamp" 2>/dev/null || true
|
|
cliTaskRun "libreportal updater apply $app auto" "updater_apply" "$app" "--detach"
|
|
enqueued=$((enqueued + 1))
|
|
done < <(jq -r '.apps[]? | select(.update_available == true)
|
|
| "\(.name)\t\(.available_digest // "")"' "$upd" 2>/dev/null)
|
|
|
|
if (( enqueued > 0 )); then
|
|
isSuccessful "Queued $enqueued automatic app update(s) — each is snapshotted before it is applied."
|
|
elif (( skipped > 0 )); then
|
|
isNotice "$skipped app(s) have an update that a previous automatic attempt could not apply — update them from the WebUI to see why."
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Automatic version LADDERING — at most one rung, at most once a day.
|
|
#
|
|
# updaterApplyAuto above moves an app WITHIN its version line (a rebuild of the
|
|
# tag it already tracks). This moves it BETWEEN lines, and it is the deliberately
|
|
# cautious half of that: it never plans a climb, only ever the single next
|
|
# published release, and it does that at most once per calendar day per app.
|
|
#
|
|
# Why one rung and not a ladder: a ladder run unattended can be several
|
|
# migrations deep before anyone looks, and "restore the snapshot from sixty
|
|
# seconds ago" stops being a comfort once four of them have stacked. One rung a
|
|
# day means the app is never more than a single version from a state that
|
|
# verified, and there is a day in which to notice. It crosses a major boundary
|
|
# if that is genuinely the next release, because refusing would strand an app
|
|
# on the last version of a line forever — but it gets there one step at a time,
|
|
# never as a leap.
|
|
#
|
|
# Every rung still goes through updaterUpgradeApp, so the contract is unchanged
|
|
# from the button: snapshot (fail-closed) -> set version -> pull -> up -> verify,
|
|
# and on any failure that rung is restored and the climb stops.
|
|
#
|
|
# Two stamps, both one-shot in different ways:
|
|
# <app>.rung the target last attempted — a rung that failed is not retried
|
|
# until a NEWER one is published (mirrors the .digest stamp).
|
|
# <app>.rungday the day a rung was last attempted — the once-a-day bound.
|
|
_updaterAutoRungStamp() { echo "$(_updaterAutoDir)/$1.rung"; }
|
|
_updaterAutoRungDay() { echo "$(_updaterAutoDir)/$1.rungday"; }
|
|
|
|
updaterUpgradeAuto()
|
|
{
|
|
[[ "${CFG_UPDATER_AUTO:-true}" == "true" ]] || return 0
|
|
# Its own switch as well as the master one: "keep my apps patched" and "move
|
|
# my apps between versions on their own" are different appetites for risk,
|
|
# and someone should be able to want the first without the second.
|
|
[[ "${CFG_UPDATER_LADDER_AUTO:-true}" == "true" ]] || return 0
|
|
command -v jq >/dev/null 2>&1 || return 0
|
|
|
|
local upd; upd="$(_updaterAutoGenDir)/updates.json"
|
|
[[ -f "$upd" ]] || return 0
|
|
updaterInWindow || return 0
|
|
|
|
local _f
|
|
for _f in cli_updater_ladder cli_updater_verify cli_updater_upgrade; do
|
|
declare -F updaterNextRung >/dev/null 2>&1 && break
|
|
source "$install_scripts_dir/cli/commands/updater/${_f}.sh" 2>/dev/null
|
|
done
|
|
declare -F updaterNextRung >/dev/null 2>&1 || return 0
|
|
|
|
local auto_dir; auto_dir="$(_updaterAutoDir)"
|
|
[[ -d "$auto_dir" ]] || runFileOp mkdir -p "$auto_dir" 2>/dev/null
|
|
|
|
local today; today="$(date +%F)"
|
|
local app channel image repo next stamp dayf enq=0
|
|
while IFS=$'\t' read -r app channel image; do
|
|
[[ -n "$app" && -n "$channel" && -n "$image" ]] || continue
|
|
[[ "$(updaterAppPolicy "$app")" == "auto" ]] || continue
|
|
# An in-flight updater task for this app: leave it alone rather than
|
|
# stacking a version move on top of an update that is still running.
|
|
updaterAutoTaskPending "$app" && continue
|
|
|
|
dayf="$(_updaterAutoRungDay "$app")"
|
|
[[ -f "$dayf" && "$(cat "$dayf" 2>/dev/null)" == "$today" ]] && continue
|
|
|
|
# No verifier, no automatic climb. GATE 1 in the engine would refuse
|
|
# this anyway, but refusing HERE means we never enqueue a task that
|
|
# exists only to fail: the user would get a failure notification every
|
|
# single day for an app that was never eligible.
|
|
declare -F updaterHasVerifier >/dev/null 2>&1 && \
|
|
{ updaterHasVerifier "$app" || continue; }
|
|
|
|
repo="$(updaterRepoTag "$image")"; repo="${repo%:*}"
|
|
next="$(updaterNextRung "$channel" "$repo")"
|
|
[[ -n "$next" ]] || continue
|
|
|
|
stamp="$(_updaterAutoRungStamp "$app")"
|
|
[[ -f "$stamp" && "$(cat "$stamp" 2>/dev/null)" == "$next" ]] && continue
|
|
|
|
# Stamp BEFORE enqueueing, same reasoning as the digest stamp: a crash
|
|
# between the two costs one skipped upgrade, never a loop.
|
|
printf '%s' "$next" | runFileWrite "$stamp" 2>/dev/null || true
|
|
printf '%s' "$today" | runFileWrite "$dayf" 2>/dev/null || true
|
|
cliTaskRun "libreportal updater upgrade $app $next --auto" "updater_upgrade" "$app" "--detach"
|
|
enq=$((enq + 1))
|
|
isNotice "$app: queued the next version step $channel → $next (snapshotted and verified; one step only)."
|
|
done < <(jq -r '.apps[]? | select((.type // "") == "versioned")
|
|
| "\(.name)\t\(.channel // "")\t\(.current_image // "")"' "$upd" 2>/dev/null)
|
|
|
|
(( enq > 0 )) && isSuccessful "Queued $enq automatic version step(s) — one rung each, and no more today."
|
|
return 0
|
|
}
|