LibrePortal/docs/roadmap/app-version-updater-and-cve.md
librelad 23cef6f427 docs(roadmap): record the notification channel in the updater refinements
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 21:17:35 +01:00

174 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design)
**Status:** Largely built — P0P3 (2026-07-17), the Trivy/CVE half (2026-07-16), and the auto-update policy of §6 (2026-08-11). Fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing.
> **Revised 2026-07-17 — config-first identity + version types + anchor fix.** Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the `#LIBREPORTAL|…|…` tagging system:
> 1. **`CFG_<APP>_VERSION` is adopted, not rejected.** The tagging system (`scripts/config/tags/processors/tags_processor_app_config_values.sh`) already derives compose values from `CFG_<APP>_<KEY>` config — so a version config makes the config *the* source and the image tag *derived*, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8.
> 2. **Two version *types*, not one model.** *Versioned* apps track a real semver-ish tag (nextcloud `31`, mastodon `v4.2.0`); *rolling* apps track a floating channel (`latest`/`stable`, ~32 of 50 lines). Version numbers are only comparable for the first kind; the **digest is the universal fingerprint** that makes "update available?" answerable for *both*. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2.
> 3. **The per-app version anchor is the `<slug>-service` image, not the first `image:` line.** Every app defines a `<slug>-service` (33/33); a few list a companion first (ollama: `ollama-webui`/open-webui before `ollama-service`), so `grep -m1 image:` mis-picks. Fixed live 2026-07-17 (`updaterPrimaryImage`). See revised §1.5.
>
> The digest-pinning idea (below) is unchanged in mechanism but **demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."**
---
## 0. Where we actually are (so we don't redesign what exists)
Most of the updater is **already built and working**. What's missing is narrow:
| Piece | Status |
|---|---|
| `updater apply` — snapshot → `compose pull``up`, auto-rollback on failure | ✅ real (`cli_updater_commands.sh`) |
| `history.json` (fail-closed from→to audit trail) | ✅ real |
| Signed hotfix channel (fetch/verify/apply/undo, `CFG_HOTFIX_AUTO`) | ✅ real |
| Auto-scan scheduling (`updater check auto`, 30-min throttle) | ✅ real |
| Auto-**apply** policy (`CFG_<APP>_UPDATE_TYPE`, default auto) | ✅ real (2026-08-11, §6) |
| WebUI — fleet Updates tab, per-app Updates tab, Security/Recovery/History renderers, Update/Roll-back buttons wired to tasks | ✅ real, waiting on data |
| **"Update available" detection** | ❌ stub — `update_available` is hard-coded `false` (`webui_updater_scan.sh:55-58` is a marked hook) |
| **CVE data** (`cves.json`) | ❌ stub — written empty once, then never rewritten (the `[ ! -f ]` guard at `:82-89` must go) |
| Recovery fields (`dr_ready`, `last_snapshot*`) | ❌ never emitted; frontend infers from History |
So this is **not** a new tab or a new subsystem. The surfaces the feature needs already exist — the fleet **Updates** tab and the **per-app Updates tab** (the "version tab per app" — it's already there, beside Backups). The job is to make the two generators tell the truth, and to fix one real correctness hole described next.
## 1. The core problem: `latest` is a channel, not a version
Across `containers/*/docker-compose.yml` there are 49 active `image:` lines: **21 on `:latest`, 11 untagged (implicit latest), 17 on pinned or floating-stable tags**. Zero are variable-driven. Only nextcloud (`31-fpm-alpine`) and mastodon (`v4.2.0`) are meaningfully pinned among primary apps.
Floating tags break both halves of what we want:
- **Detection is textually impossible.** "Is `vaultwarden/server:latest` newer than `vaultwarden/server:latest`?" — the compose file can't answer; the string never changes.
- **Revert is currently a lie.** `updaterRollbackApp` restores the *data* snapshot and re-runs `compose up` — but the local image is already the *new* latest, so you get **new code on old data**. History records `from → to`, and for a floating tag both are the same string.
### The one idea
> **Keep the floating tag as the *channel*; pin the *digest* in the live compose.**
> `image: vaultwarden/server:latest@sha256:ab12…`
Compose supports `tag@digest` natively. The tag stays as the human-readable statement of *what we track* ("latest", "stable", "31-fpm-alpine"); the digest pins *exactly which build runs*. Then:
- **Detect:** resolve the channel tag at the registry (`docker manifest inspect`, no pull) and compare digests → a truthful `update_available`, even for `latest`.
- **Update:** snapshot → rewrite the pin to the new digest → pull → up. Nothing moves unless the user (or a future policy) says so.
- **Revert:** restore the data snapshot **and** rewrite the pin back to the previous digest. Now rollback means what the button says.
Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only `latest`, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a *pinned* tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the **live** copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's `set-compose-image` op is a working, drift-guarded, undoable sed for precisely this line).
### Multi-service apps — the anchor, not the first line
10 of 33 apps are multi-service, and they break three *different* naive assumptions (this is why "which image is the app's version?" can't be answered positionally):
- **Nextcloud (4 services): most images aren't the app.** `nextcloud:31-fpm-alpine` is the app; `mariadb:11` + `redis:alpine` + `nginx:alpine` are infra sidecars. The version the user cares about lives on *one* of four lines.
- **ollama (2 services): the first image isn't the app.** File order is `ghcr.io/open-webui/open-webui:main` **then** `ollama/ollama:latest`. `grep -m1 image:` reports open-webui as the version of an app *named ollama* — a live mislabel (fixed 2026-07-17).
- **jitsi (4 services): one app, one lock-step version, four images.** `jitsi/{web,prosody,jicofo,jvb}:stable` share one release channel and must move together.
Design consequences:
- **The version anchor = the app's `<slug>-service` image** (underscores → hyphens: `libreportal_catalog``libreportal-catalog-service`). Every app defines one (33/33), and it correctly resolves ollama to `ollama-service`. This is the primary for `updates.json`'s top-level fields, so the existing UI works unchanged. Helper: `updaterPrimaryImage` (`webui_updater_scan.sh`).
- **Track every `image:` line, keyed by service** — `updates.json` gains a `services[]` array. An app "has an update" if *any* tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services").
- **Lock-step sets → one channel drives several image lines.** A single `CFG_<APP>_VERSION` (see §2) can feed every `jitsi/*` sentinel, so the four move as a unit.
- **Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions.** They get pinned + CVE-scanned (you *do* want redis CVEs), but they're not "the app version" and are candidates for **fleet-shared** pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable.
## 2. Version identity — three facts, three sources, two types
The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing **three different questions** into one word, "version." They have different answers and you need all three:
| Question | Source | Who sets it | Nextcloud e.g. |
|---|---|---|---|
| **What version do I *want*?** | `CFG_<APP>_VERSION` → drives the anchor image's tag via the tagging system | **user** (defaults to the template's current tag) | `31` or `latest` |
| **What's *exactly* running?** | the resolved digest, recorded next to it | machine (on install / apply) | `sha256:ab12…` |
| **What's *available*?** | registry lookup on that channel tag | machine (scan) | `32` has shipped |
`CFG_<APP>_VERSION` is **the** source of truth, not a second one: the compose image tag is *derived* from it through the `#LIBREPORTAL|<APP>_VERSION_TAG|<default>` sentinel on the anchor's `image:` line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The **digest is a fingerprint, not the identity** — it disambiguates "`latest` yesterday vs today" and makes revert exact, but it's recorded *alongside* the config channel, never the thing the user edits. This is what lets a user **install/pin a specific version** (set the config, apply) while `latest` stays the default for hands-off freshness.
### Two version *types* (this is the "specific types" idea)
Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata):
- **Versioned** — anchor tag is a real version (`nextcloud:31`, `mastodon:v4.2.0`, `navidrome` via its OCI label `0.62.0`). UX: show the number, offer a version picker, "31 → 32". `update_available` when a newer tag or a newer build of the pinned tag exists.
- **Rolling** — anchor tag is a floating channel (`vaultwarden/server:latest`, `gitea:latest`, `jitsi/*:stable`; ~32 of 50 lines). There is no upstream number to compare, so UX shows the **channel + build freshness**: "latest · new build available (2 days newer)". `update_available` is purely a digest change.
The **digest comparison is the one uniform engine underneath both types** — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a *display enrichment* on top, as good as each upstream allows, never the detection mechanism.
### Human-readable version (display enrichment)
Digests are unreadable, so for display resolve in order: **`CFG_<APP>_VERSION` (if set) → OCI label `org.opencontainers.image.version` (`docker image inspect`; verified present on e.g. navidrome `0.62.0`, trivy `0.72.0`, absent on nginx/alpine) → meaningful tag (not `latest`/untagged) → short digest (`ab12cde`)**. Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it).
### `updates.json` additions
(existing fields keep their meaning) per app `type` (`versioned|rolling`), `channel` (the anchor tag), `pinned_digest`, `available_digest`, `version` (display, resolved as above), `available_version`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }`, plus the recovery fields the frontend already renders (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) from existing backup data. **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible.
## 3. Detection mechanics
- `docker manifest inspect <repo:tag>` (rootless, via `dockerCommandRunInstallUser`) returns the registry digest without pulling; compare against the pinned digest. Local `RepoDigests` and the registry manifest-list digest are the same identity, so the comparison is exact.
- **Throttle separately from the scan.** The 30-min `updater check auto` stat-gate stays cheap; registry lookups get their own stamp + knob, mirroring the established `/tmp` stamp idiom: `CFG_UPDATER_REGISTRY_INTERVAL` (minutes, **default 360**, `0` = manual-only). ~30 Docker Hub images once per 6 h sits comfortably under anonymous rate limits; add small jitter so fleets don't thundering-herd.
- **Failure = stale, not error.** A registry timeout keeps the previous verdict and stamps `last_checked`; never flip an app to "unscanned" because the network blinked (per-registry short-circuit after the first timeout in a run).
- **Reconcile existing installs:** on first scan, any installed app whose compose line is unpinned gets pinned to its *currently running* image's digest (`RepoDigests`) — zero behaviour change, purely recording reality. New installs pin right after the first pull. Apps using `build:` instead of `image:` are out of scope (scan already ignores them).
## 4. Apply, revert, and the per-app surface
`updaterApplyApp` becomes: resolve new digest (from `updates.json`) → **snapshot (unchanged, fail-closed)** → rewrite pin(s) via the factored-out first-class helper shared with the artifact op's sed → `compose pull``up` → history `ok` with digest-refs → rescan. On failure: auto-rollback now also **rewrites the pins back** before restoring the snapshot — that closes the "new code on old data" hole.
`updaterRollbackApp` (the user-facing Roll back button): read the last `update/ok` history entry's `from` refs → rewrite pins → restore data snapshot → `up` → history `rolled-back`. This is the "allow users to revert back" requirement, and it only works because of the pins.
**UI:** no new navigation. The per-app **Updates** tab gains a Version section (already scaffolded in `renderAppDetail`): channel · running build · available build · Update / Roll back. The fleet tab's rows finally show real `current → available`. Everything routes through the existing `updater_apply` / `updater_rollback` tasks — mutations stay task-only, no new API endpoints.
## 5. CVE scanning — decision
Three candidate shapes were on the table:
| Shape | Verdict |
|---|---|
| **A. Scanner as a catalog app** (trivy in the App Center, recommended alongside CrowdSec) | ✅ **Chosen** (2026-07-14). Installing/removing the scanner is the opt-in switch, visible and revertible like every other app; fits "everything is an app"; sits next to CrowdSec as the second security recommendation at first install. |
| **B. Scanner binary on the host** | ❌ Grows the host footprint we've deliberately kept lean (rootless + de-sudo); another thing to install/update outside the app model. |
| **C. Invisible ephemeral container** — the scan task does `docker run --rm` per scan, no catalog presence | ❌ Zero idle cost, but the scanner is invisible: no App Center card, no obvious opt-in/uninstall, and users can't see what has socket access. Visibility won. |
So: **trivy ships as a normal catalog app** (`containers/trivy/`), marked **recommended** and listed in the first-install recommended apps directly under CrowdSec. The updater's CVE step is gated on the app being installed — no trivy app, no scan, honest empty state pointing at the App Center ("Install the security scanner to enable CVE reports").
**Scanner: Trivy** (over grype — both fine, trivy has the larger ecosystem, single pinned OCI image, clean JSON, Apache-2.0). Details:
- The app runs `trivy server` (its long-running mode): one modest container that owns the vulnerability DB (~600 MB volume) and keeps it fresh. The updater scan executes scans **through the app's container** (`docker exec <trivy> trivy image --format json <image>`) — no per-scan container spawn, no host binary, and the DB downloads once instead of per run.
- Image access via the rootless docker socket mounted read-only into the trivy container (same daemon that owns the images) — the same trusted position CrowdSec-style security tooling already occupies, but here it's a visible, uninstallable app.
- **Privacy posture (worth stating in user-facing copy):** nothing about your images or apps ever leaves the box — Trivy matches locally against a downloaded DB; the only outbound call is the DB fetch from ghcr. The opt-in is installing the app (recommended, pre-ticked at first install like the other recommendations — skipping it = air-gap-friendly); `CFG_UPDATER_CVE_INTERVAL` (minutes, **default 1440** — daily; scans are the expensive step, results don't change hourly) lives in `configs/webui/webui_updater` next to the existing knobs.
- Output maps straight onto the schema the frontend already renders: per app `cves[] { id, severity, package, fixed_in, url }` + global `totals`. Dedupe per image (shared base layers repeat findings), scan each distinct image once per run. **Drop the `[ ! -f ]` guard** so re-scans overwrite `cves.json`.
- **Not alarmist** (house rule): the Security tab lists everything, but the per-app chip/badge only fires for **critical/high with a fix available** — "your box has 400 unfixable medium CVEs" red badges are noise, not signal.
- **Tie-in with updates:** a CVE whose `fixed_in` is satisfied by the available build marks that update as a *security update* — the Security filter chip and severity sort already exist in the UI, they just start meaning something.
## 6. Auto-update policy — **BUILT 2026-08-11** (per-app, not severity-split)
Shipped a shape the original sketch didn't consider: the policy is **per app**, not one fleet-wide severity rule.
- **`CFG_<APP>_UPDATE_TYPE=auto|manual`, default `auto`** — in all 33 app config templates (not `libreportal`, whose own updates are a different subsystem). Reaches existing installs through the normal template reconcile, and shows up on the app's Configure page as *Updates* (field mapping `UPDATE_TYPE`, category general).
- **`CFG_UPDATER_AUTO=true|false`, default `true`** — the master switch in `configs/webui/webui_updater`. It can only ever make things *more* manual, so "stop auto-updating everything" is one flip, not 33 edits. Precedence deliberately mirrors `backupResolveStrategy` (per-app override on top of a global default); `updaterAppPolicy` is the single resolver, used by both the enqueuer and the scan.
- **`updaterApplyAuto`** (`cli/commands/updater/cli_updater_auto.sh`), called at the end of `updater check`, enqueues the ordinary `updater_apply` task for every update-available app whose policy is auto — never applies inline, exactly like `artifactApplyAuto`. So an automatic update is byte-for-byte the manual path: snapshot → pull → up → auto-rollback on failure, one task log, one History entry, same Roll back button.
- **No retry storms.** Each attempt stamps the target digest under `generated/auto/<app>.digest`; the same digest is never auto-attempted twice. A broken upstream build is tried once, rolled back, then left alone (the Update button still offers a manual retry) until a *newer* build changes the digest. In-flight tasks are also skipped, so a scan landing mid-update can't stack a second one.
- **Tracked, not silent.** `updates.json` carries each app's resolved `update_type`, and History entries carry `trigger: manual|auto`. The WebUI reads both: the fleet list says whether updates install themselves, rows that opted out carry a `manual` chip, and automatic History entries are labelled. The Overview board treats an auto app's pending update as a *quiet* line, not a "needs action" one — it doesn't need a decision.
Why not the sketched `off|security|all` severity split: severity is a property of the *CVE data*, which is only as complete as Trivy's per-image scan, whereas "do I let this app move on its own?" is a property of the *app* — nextcloud and a DNS blocker are not the same risk. Per-app also composes with the CVE work instead of competing with it: a future `security` value can be added per app without changing the master switch or the enqueuer's shape.
**Soaked 2026-08-11:** the first live end-to-end auto-update (trivy 0.72.0→0.73.0) ran on a real install. Its first attempt exposed and fixed a latent dynamic-scoping bug (a callee's undeclared `while read app` loop blanked the updater's `app` local mid-update — pre-existing, had also corrupted a manual update's History on 08-10); the retry completed cleanly with `trigger:auto` and exact digest refs.
**Refinements (2026-08-11), after the "hands-off background" review:**
- **Install window** — `CFG_UPDATER_WINDOW` (HH:MM-HH:MM host local time, wraps midnight, `always`; default **06:00-08:00**, right after the 05:00 backup cron). Gates only the *enqueue* in `updaterApplyAuto`; scans keep their all-day cadence so detection stays current and pending updates visibly wait. Malformed values fail closed (hold updates) and are rejected by the WebUI validator; the raw value is surfaced in `updates.json` (`auto_window`) so the UI states when installs land. Ordering rationale: backups nightly at 05:00 → updates 06:00-08:00, and every update still takes its own pre-update snapshot regardless (fail-closed), so the window is about *quiet hours*, not safety.
- **Check now forces a live registry lookup** — `updater check` (non-`auto`) sets `UPDATER_REGISTRY_FORCE=1`; previously the flag existed but nothing set it, so the button silently reused the 6h digest cache. Force also overrides `CFG_UPDATER_REGISTRY_INTERVAL=0`, making 0 mean "manual-only" as originally intended.
- **Registry stamp out of /tmp** — the task-processor unit runs with systemd `PrivateTmp`, so the /tmp stamp existed once per namespace (daemon vs CLI, daemon's resetting every restart). Now at `<system>/logs/updater_registry_checked`.
- **Failed-auto honesty** — the scan emits each app's `auto_attempted_digest` (the one-shot no-retry stamp). When it matches `available_digest`, the UI stops claiming "installs automatically": the per-app detail says the build failed and won't retry, the fleet row gets an `auto failed` chip, and the Overview board counts it as *waiting for you* (warn) instead of quietly-installing (ok).
- **Failure notifications (2026-08-11)** — `libreportal notify` + `configs/general/general_notifications`: email (rides the Mail settings), ntfy, Gotify, Discord, Slack, Telegram, Pushover. One hook in the task processor reports every terminal task; the notify command applies the policy (`CFG_NOTIFY_EVENTS=failures|all|off`, cancelled never notifies). A failed auto-update now reaches the user with "already rolled back, won't retry" copy without the WebUI being open. Verified end-to-end through the live daemon against a mock endpoint.
- **`CFG_TIMEZONE` label corrected** — it only sets containers' TZ; scheduled tasks follow the host clock. The old "System Timezone - Timezone for scheduled tasks" label promised a knob that never existed.
## 7. Build phases (each independently shippable)
0. **P0 — Anchor fix.***done 2026-07-17.* `updaterPrimaryImage` resolves each app's version image from its `<slug>-service`, not the first `image:` line — fixes the ollama mislabel. Used by both the version and CVE loops.
1. **P1 — Config-first pin foundation.***done 2026-07-17.* (b) `CFG_<APP>_VERSION` drives the anchor tag via a `#LIBREPORTAL|<APP>_VERSION_TAG|<tag>` sentinel on every app's anchor image line (32 apps, opt-in, default = today's tag = no-op). (c) Digest recorded/reconciled by the scanner itself (no install hook needed — it inspects the running image each scan). `type` classified from the anchor tag. Shared image-line digest-rewrite = `updaterSetAnchorRef` (anchor-aware; artifact-op sharing deferred).
2. **P2 — Real detection + display.***done 2026-07-17.* Registry digest compare (`docker buildx imagetools inspect` = the RepoDigest identity) in `webuiUpdaterScan` behind `CFG_UPDATER_REGISTRY_INTERVAL` (throttled, reuse-between-windows); `updates.json` carries `type`/`channel`/`services[]`/digests + resolved display `version` (config → OCI label → tag → short digest). Fleet + Updates tabs show truthful data. **Note:** digest-compare fully detects *rolling* new-builds; *versioned* newer-version enumeration (tag listing) is a P5 add. Recovery fields still inferred from History (deferred).
3. **P3 — Pinned apply/revert.***done 2026-07-17.* `updaterApplyApp` records exact `repo:tag@sha256:…` from→to and un-pins before pulling; `updaterRollbackApp` re-pins the anchor to the pre-update build's digest before `up` (closes "new code on old data"). History carries digest refs. *(Helpers unit-tested; end-to-end apply/revert not yet exercised on a live install with a pending update.)*
4. **P4 — CVE scanner.** The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; `cves.json` for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4.
5. **P5 — Polish.** Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows.
6. **P6 — Auto-update policy.***done 2026-08-11.* Per-app `CFG_<APP>_UPDATE_TYPE` (default auto) under the `CFG_UPDATER_AUTO` master switch; `updaterApplyAuto` enqueues the normal apply task, one attempt per target digest, policy + trigger surfaced in `updates.json`/`history.json` and the WebUI. See the rewritten §6. *(Landed ahead of P3's live soak at the maintainer's call — the snapshot-first apply and the one-shot digest stamp are what make that acceptable.)*
## 8. Rejected alternatives (for the record)
- **~~`CFG_<APP>_VERSION` config vars~~ — ADOPTED 2026-07-17 (see §2).** Original objection: "second source of truth vs the compose file, collides with the `_TAG` namespace, doesn't solve floating `latest`." Reassessed: the tagging system makes the config *the* source (compose tag derived from it, like every other config value), and `CFG_<APP>_VERSION``<APP>_VERSION_TAG` fits the naming convention with no collision. The only surviving point — *doesn't detect floating `latest` alone* — is real and is exactly why the **digest fingerprint** rides alongside it; config = channel, digest = build.
- **Pure "compare version numbers" (no digest)** — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, *two types*).
- **Pin semver tags across all templates** — permanent curation burden, inconsistent upstreams, still blind to republished tags.
- **Watchtower-style auto-pull of latest** — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back." Note §6's auto default is *not* this: it is the same deliberate, snapshotted, revertible apply the button runs, scheduled — visible in Tasks and History, per-app switchable, and never retried blindly.
- **Docker Scout / hosted scanners** — requires accounts / sends data off-box; against the ethos.
- **A new top-level "Versions" area** — unnecessary; the fleet Updates tab + per-app Updates tab already are that surface, they just need real data.