The registry helpers rejected any repo containing a dotted host segment,
which caught 'docker.io/authelia/authelia' — Docker Hub spelled out in
full. Those apps were silently skipped by tag enumeration and version
laddering. Strip the docker.io/ and index.docker.io/ prefixes before the
host check; genuinely third-party registries (ghcr.io, quay.io, lscr.io)
are still correctly skipped.
Found by auditing every app's anchor image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The live Nextcloud 31→34 climb left 4.4 GB of images behind — one per
rung, each ~1.5 GB, all still present after it finished. On a small VPS
that is the difference between working and full.
`system reclaim` cannot help: it collects DANGLING images, and every rung
is a real tag, so all of them stay tagged and stay on disk. (Rolling apps
never hit this — moving a floating tag orphans the old image, which
reclaim then collects. It is specific to laddering.)
After a SUCCESSFUL climb only, remove the images stepped through, keeping
the immediately-previous version so a roll-back needs no download.
CFG_UPDATER_UPGRADE_PRUNE=false keeps everything. Never runs on failure,
where the older images are exactly what recovery may need.
Tested: a 3-rung climb removes 31 and 32 and keeps 33; a single-step
climb removes nothing (its previous version IS the rollback target); the
config switch disables it.
Found by looking at the box after the first real ladder run — the feature
worked, and then quietly cost 4.4 GB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`updater upgrade <app> --detach` parsed "--detach" as the target version
and refused with "no safe path from 31-fpm-alpine to --detach". It failed
safe, but blaming the version for a misplaced flag is a poor way to say
the flag is not supported here. Unknown options now say so.
Found during the first live ladder run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ties the ladder and the verifiers together behind a new verb:
libreportal updater upgrade <app> [version] [--dry-run]
Per rung, and every part is load-bearing:
snapshot (fail-closed) -> set version -> pull -> up -> VERIFY -> next
On failure anywhere: restore THIS rung's snapshot, put the version back,
stop, and leave the app on the last version it actually verified at. The
ladder never continues past a doubt.
A snapshot PER RUNG rather than one at the start, because upstream
migrations are usually one-way — Nextcloud 32's schema cannot be undone
by putting the 31 image back. The recovery guarantee is "restore the
snapshot from sixty seconds ago", which only holds if every rung has one.
Two gates before anything moves. An app with no <app>_upgrade_verify is
refused outright: the generic health check cannot see a half-finished
migration, so laddering on it would be a guess wearing a safety label.
And a ladder that cannot be computed end to end refuses rather than
attempting a partial climb.
`updater upgrade` is a separate verb from `apply` on purpose: apply moves
you WITHIN a release line (and may be automatic), upgrade moves you
BETWEEN lines and is always a deliberate act. Dry runs execute inline so
the plan is instant to read.
updaterSetAnchorVersion rewrites the image tag AND its version sentinel
together — updating only the image would leave the sentinel advertising
the old version, and the next config regeneration would silently revert
the app.
Tested with stubs against the real code paths: the no-verifier gate holds
and changes nothing; a dry run has zero side effects; the happy path
snapshots at each current version before moving; a verify failure on rung
2 of 3 stops with the app on rung 1, restored, and never touches rung 3;
a failed snapshot moves no version and pulls nothing; a container that
will not start is rolled back.
NOT yet exercised on a live install — no app here needs a ladder. The
first real run should be a dry run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Foundation for stepped upgrades. Answers one question only — WHICH
versions, in WHICH order — with no side effects, so it can be tested
exhaustively. Applying the rungs is a separate job.
Nextcloud refuses to skip a major ("Updates between multiple major
versions and downgrades are unsupported") and will not start; databases
behave the same way about their data directory. For those apps 31 -> 34
is three upgrades, each with a migration that must finish before the
next begins.
Built by PROBING each candidate rung, not by enumerating tags — because
enumeration is provably unsafe here. Docker Hub pages at 100 ordered by
recency, and the first real-registry run proved the danger: it produced
v4.2 -> v4.4 -> v4.5 -> v4.6 for mastodon, silently skipping v4.3, which
exists (HTTP 200) but had fallen off the newest-100 listing. Skipping a
rung is the precise failure this file exists to prevent, so the ladder is
now built by incrementing and probing: v4.2 -> v4.3 -> v4.4 -> v4.5 ->
v4.6, 4 steps.
Guarantees: same shape only (never 31-fpm-alpine onto 31-apache),
strictly ascending, never a downgrade, rolling tags refused outright, and
a version upstream never published is stepped over only because the probe
said so. If a continuous path to the target cannot be constructed it
returns 1 and prints nothing — refusing to guess, because a wrong ladder
means a skipped migration.
20 unit tests, including the exact listing-truncation case above and the
numeric ordering that would otherwise drive an app backwards (0.9 vs
0.10). Real registry: nextcloud 3 steps, mastodon 4, stalwart current.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes that make the auto-updater a trustworthy background system:
* CFG_UPDATER_WINDOW (default 06:00-08:00 host time, right after the
05:00 backup cron; HH:MM-HH:MM wraps midnight, 'always' = any time).
Gates only the enqueue — scans keep running all day, so the Updates
page stays current and pending updates visibly wait for the window.
Malformed values fail closed and are rejected by the WebUI validator.
* "Check now" actually checks: an explicit `updater check` sets
UPDATER_REGISTRY_FORCE=1. The flag existed but nothing ever set it,
so the button silently reused the 6h digest cache and could not find
a build the user knew had shipped. Force also overrides interval 0,
which now means "manual-only" as documented in the roadmap.
* Registry stamp moved from /tmp to <system>/logs: the task processor
runs under PrivateTmp, so daemon and CLI each kept a separate 6h
clock and the daemon's reset on every service restart.
* A failed automatic attempt is no longer invisible: the scan emits
auto_attempted_digest (the one-shot no-retry stamp), and when it
matches the available build the UI stops promising an install that
will never come — per-app detail explains, the fleet row gets an
"auto failed" chip, and the Overview board counts it as needing you.
Also corrects the CFG_TIMEZONE label: it sets the containers' TZ only;
scheduled tasks follow the host clock (timedatectl), and the old
"Timezone for scheduled tasks" wording promised a knob that never
existed. The window + auto_window display state plainly WHEN updates
land, answering "how does the user know when the next update happens".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First real end-to-end auto-update on a live install failed like this:
Automatically updating trivy (a recovery snapshot is taken first)
Snapshotting trivy before update…
Pulling new image(s) for …
Update of failed — rolling back…
Could not roll back automatically
The app name went empty after the snapshot. Cause: bash is dynamically
scoped, so a callee assigning an undeclared variable writes the CALLER's
local of that name — and a `while read app` loop leaves it EMPTY at EOF.
webuiBackupAppStatus's dashboard generator runs at the end of every backup
and did exactly that to updaterApplyApp's `app`.
Nothing was damaged: the pull ran against an empty name, failed before
touching the image, and the rollback was a no-op on a nonexistent app.
Fixed both ends. The generator (and three gluetun loops with the same
latent leak) now declare `local app`. updaterApplyApp/updaterRollbackApp
hold the name in `_upd_app` so they no longer depend on every callee's
hygiene, and updaterApplyAll stops leaking its own loop var.
This is exactly the untested path the roadmap flagged: "apply/revert not
yet exercised end-to-end on a live install with a pending update."
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the decision half of the app updater. Detection (P2) and the
snapshot-first apply/revert (P3) were already real, but nothing ever
pressed the button — every update waited for a click.
CFG_<APP>_UPDATE_TYPE=auto|manual per app, default auto (33 templates)
CFG_UPDATER_AUTO=true|false master switch, default true
updaterAppPolicy resolves the two the way backupResolveStrategy already
resolves backup strategy: the global switch can only make things more
manual. updaterApplyAuto runs at the end of `updater check` and enqueues
the ordinary updater_apply task for each auto app that has an update —
never applies inline, so an automatic update is the same code path, task
log, History entry and Roll back button as a manual one.
Safety: each attempt stamps its target digest under generated/auto/, so a
build that fails is rolled back and then left alone rather than retried on
every scan; in-flight updater tasks are skipped so scans can't stack.
Tracked end to end: updates.json carries each app's resolved update_type,
History entries carry trigger=manual|auto. The WebUI says whether updates
install themselves, chips only the apps that opted out, labels automatic
history, and — since an auto app's pending update needs no decision — keeps
it off the Overview board's "Needs action" view.
Also fixes artifactApplyAuto enqueueing without --detach: it runs inside
the single-threaded task processor's own poll, so following the new task in
the foreground waits for a task that cannot start until it returns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make the Update / Roll-back buttons tell the truth, closing the "new code on
old data" hole a floating tag creates.
updaterApplyApp:
- Anchor-correct: capture before/after from updaterPrimaryImage (the
<slug>-service image), not `grep -m1 image:` — fixes ollama et al.
- Records EXACT build refs in history from->to: repo:tag@sha256:<digest>
(via updaterRefDigest), so history is meaningful even when the tag doesn't
move (a rebuilt `latest`).
- Un-pins any digest a prior rollback pinned before pulling, so Update tracks
the channel again instead of freezing on the rolled-back build.
updaterRollbackApp:
- Before recreating, re-pins the anchor image to the pre-update build's digest
(from history's last update/ok `from`) via updaterSetAnchorRef, so `up` runs
the OLD code — not the current channel head. This is the fix for restoring a
data snapshot but recreating on a newer image.
New helpers (cli_updater_commands.sh): updaterRefDigest (local RepoDigest),
updaterSetAnchorRef (rewrite the anchor image line by service name, preserving
indent + the version sentinel; correct for companion-first apps like ollama),
updaterLastUpdateFrom (roll-back target from history).
Verified the helpers on nextcloud + ollama: pin adds @sha256 to the right
anchor only, sidecars untouched, sentinel preserved, unpin restores, YAML valid.
Caught and fixed a `local a=$1 b=...$a...` same-statement expansion bug that
would have silently no-op'd the rollback pin. End-to-end apply/revert not
exercised live here (no installed app has a pending update on this box).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
webuiRegistryCatalogScan (run by updater check, same atomic keep-prior
pattern as webuiArtifactScan) writes apps/generated/registry_catalog.json:
the type:"app"/kind:"bundle" rows of the signed index annotated with
defined/installed, browse metadata from the envelope meta, and icons
mirrored into core/icons/apps/registry/ ONLY when their bytes match the
sha256 pin in the signed index — the browser stays same-origin; a tampered
or oversized icon is skipped, never served.
webuiArtifactScan now selects type=="hotfix" so app rows never render as
pseudo-hotfixes in the Improvements tab, and counts+logs artifacts of
unrecognized type instead of surfacing them (the §8.1 forward-compat
firewall on the scan path).
Harness vs a locally served registry: 14/14 (catalog row + meta + flags,
icon pin verify + tamper skip, hotfix-only stream, unknown-type skip+log,
unreachable-registry keeps prior files).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replace the click-to-scan-only flow with a self-throttled auto-scan that
rides the existing task-processor idle poll (the same shape as the
network-drift check — no new daemon, unit, or endpoint):
- 'libreportal updater check auto' gates on the age of the generated
updates.json vs CFG_UPDATER_SCAN_INTERVAL (minutes, default 30,
0 disables); a fresh file makes the 60s tick a single stat() + return.
Manual checks and post-update rescans reset the clock for free, and a
missing file means the first scan runs ~a minute after install.
- Eligible signed hotfixes keep flowing through artifactApplyAuto, which
only enqueues ordinary tasks — mutations stay on the task path.
- Open updater surfaces (standalone /updater and the fleet Overview's
headless UpdaterPage) follow along with a 60s static-JSON re-read that
repaints only when a generated_at stamp changed; timer released via
dispose() on unmount, ticks skipped while hidden.
- Empty states now say the first scan happens automatically; Check now
stays as the immediate manual override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- CFG_HOTFIX_AUTO (security-breakage|all|off, default security-breakage) seeded in
general_terminal; reaches existing installs via the add-only config reconciler.
- webui_artifact_scan.sh (webuiArtifactScan): fetch+verify the signed index, write
artifacts_available.json ATOMICALLY (build in temp → jq-validate → one write;
keep the prior file on any failure — never emits broken JSON). Annotates each
artifact with applied (a per-id record exists) + applicable (target installed).
- artifactApplyAuto + `libreportal artifact apply-auto`: enqueue apply tasks for
the eligible signed hotfixes — only when the index is VERIFIED-signed, only
auto==true + in the severity policy + applicable + not already applied. Each
apply is its own task (visible in the log + History), never applied inline.
- `updater check` now also refreshes the index (webuiArtifactScan) and runs
artifactApplyAuto — one front door, no second phone-home.
Unit-tested 13/13: policy filtering (security-breakage / off / all), auto:false
exclusion, already-applied skip, non-installed-app skip, unsigned-index fail-closed,
and the scan transform's signed/applied/applicable fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The mutating side of the unified distribution primitive (spec §8.3). Hotfixes
can now be applied and reverted, first-party, through the task system.
New scripts/cli/commands/artifact/cli_artifact_apply.sh:
- artifactApply <id>: resolve+gate (applies_when / min_lp / max_lp /
max_footprint / publishers-map role) → fetch+verify payload (sha256 pinned by
the signed index + minisig) → dry-precheck ALL ops (all-or-nothing) → best-
effort snapshot → apply each op recording a precise inverse → bring app up →
auto-rollback (replay undo LIFO, snapshot fallback) → applied-record + History.
- artifactRevert <id>: replay the applied-record's undo log (LIFO).
- Bounded, CLOSED op vocabulary (no run-script/exec, ever): set-config-key,
set-compose-image, patch-file-if-checksum-matches, set-data-file. An
unsupported op rejects the whole artifact at precheck (fail-closed).
- Write-target firewall: scope:app → containers/<app>/ only; scope:system →
configs/ only; the install tree (our code) is off-limits to hotfixes (fork 1).
Drift guards (expect_current / checksum) skip cleanly rather than clobber.
- Two-tier trust: index minisig-verified vs the footprint key (lpFetchIndex)
covers the envelope; payload sha256-pinned + minisig-verified; publishers-map
role gate (a non-official publisher can't claim official). Community per-
artifact-key sigs are gated off until that tier is enabled.
cli_artifact_commands.sh: apply/revert via the task system (artifact_apply /
artifact_revert types — no allowlist needed), + read-only `applied` list.
cli_updater_commands.sh:
- FIX verified safety bug: updaterApplyApp/RollbackApp called `libreportal backup
app "$app"` and `... restore latest`, which parse the app name as the ACTION,
hit the dispatcher's `*)` default (exits 0) — so updates ran with NO snapshot
and rollback was a silent no-op. Call backupAppStart / restoreAppStart directly.
- FIX updaterRecordHistory jq-silent-skip: was `command -v jq || return 0`
(silently dropped the audit entry). Now fail-closed with a brace-agnostic
bash-native prepend fallback; extended with artifact_id/serial/undo_id.
fetch.sh: add _lpJsonEsc (shared JSON-escape for the jq-free fallbacks).
Regenerated source arrays + lazy-load manifest for the new file/functions.
Unit-tested 31/31: every op apply+precheck+undo round-trip, the path-allowlist
firewall (incl. .. traversal + install-tree + cross-app rejection), all-or-
nothing abort, unsupported-op rejection, and the History bash-native fallback
(records + preserves prior entries without jq). A full signed-apply e2e needs
minisign + the signing key (Phase 5 make_hotfix.sh).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- scripts/webui/data/generators/updater/webui_updater_scan.sh (webuiUpdaterScan):
writes frontend/data/updater/generated/{updates,cves,history}.json from the
installed-apps DB (current image per app from compose). Available-version +
CVE-scanner are clearly-marked pluggable hooks; always emits valid JSON.
- scripts/cli/commands/updater/{cli_updater_commands.sh,cli_updater_header.sh}:
auto-dispatched as 'libreportal updater <sub>' (check/apply/apply-all/rollback).
apply does disaster-recovery FIRST — snapshots the app via the backup engine,
then pulls + recreates (real dockerComposeUp/compose-pull helpers), records
history, and auto-rolls-back on failure. Standard LIBREPORTAL_TASK_EXEC
enqueue/exec split so WebUI + CLI share locking + audit trail.
New .sh files: the array/function-manifest regen self-heals on deploy; the
check path also sources its generator on demand to cover the gap.
NOTE: host-side bash — written to the repo's conventions but not runnable in
this env; this is the surface to test (the WebUI feature is lp-shot-verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
