LibrePortal/docs/roadmap/app-version-updater-and-cve.md

LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design)

Status: Largely built — P0–P3 (2026-07-17), the Trivy/CVE half (2026-07-16), and the auto-update policy of §6 (2026-08-11). Fills the two deliberately-unwired hooks in webui_updater_scan.sh. · Audience: us, future-self · Scope: real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind /apps/overview/updates · Origin: "the updates system is currently not set up" conversation 2026-07-14. Companion to updates-and-distribution.md — that doc owns the signed artifact/hotfix pipe; this one owns the generators it always treated as pre-existing.

Revised 2026-07-17 — config-first identity + version types + anchor fix. Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the #LIBREPORTAL|…|… tagging system:

1. CFG_<APP>_VERSION is adopted, not rejected. The tagging system (scripts/config/tags/processors/tags_processor_app_config_values.sh) already derives compose values from CFG_<APP>_<KEY> config — so a version config makes the config the source and the image tag derived, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8.
2. Two version types, not one model. Versioned apps track a real semver-ish tag (nextcloud 31, mastodon v4.2.0); rolling apps track a floating channel (latest/stable, ~32 of 50 lines). Version numbers are only comparable for the first kind; the digest is the universal fingerprint that makes "update available?" answerable for both. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2.
3. The per-app version anchor is the <slug>-service image, not the first image: line. Every app defines a <slug>-service (33/33); a few list a companion first (ollama: ollama-webui/open-webui before ollama-service), so grep -m1 image: mis-picks. Fixed live 2026-07-17 (updaterPrimaryImage). See revised §1.5.

The digest-pinning idea (below) is unchanged in mechanism but demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."

---

0. Where we actually are (so we don't redesign what exists)

Most of the updater is already built and working. What's missing is narrow:

Piece	Status
updater apply — snapshot → compose pull → up, auto-rollback on failure	✅ real (cli_updater_commands.sh)
history.json (fail-closed from→to audit trail)	✅ real
Signed hotfix channel (fetch/verify/apply/undo, CFG_HOTFIX_AUTO)	✅ real
Auto-scan scheduling (updater check auto, 30-min throttle)	✅ real
Auto-apply policy (CFG_<APP>_UPDATE_TYPE, default auto)	✅ real (2026-08-11, §6)
WebUI — fleet Updates tab, per-app Updates tab, Security/Recovery/History renderers, Update/Roll-back buttons wired to tasks	✅ real, waiting on data
"Update available" detection	❌ stub — update_available is hard-coded false (webui_updater_scan.sh:55-58 is a marked hook)
CVE data (cves.json)	❌ stub — written empty once, then never rewritten (the [ ! -f ] guard at :82-89 must go)
Recovery fields (dr_ready, last_snapshot*)	❌ never emitted; frontend infers from History

So this is not a new tab or a new subsystem. The surfaces the feature needs already exist — the fleet Updates tab and the per-app Updates tab (the "version tab per app" — it's already there, beside Backups). The job is to make the two generators tell the truth, and to fix one real correctness hole described next.

1. The core problem: latest is a channel, not a version

Across containers/*/docker-compose.yml there are 49 active image: lines: 21 on :latest, 11 untagged (implicit latest), 17 on pinned or floating-stable tags. Zero are variable-driven. Only nextcloud (31-fpm-alpine) and mastodon (v4.2.0) are meaningfully pinned among primary apps.

Floating tags break both halves of what we want:

• Detection is textually impossible. "Is vaultwarden/server:latest newer than vaultwarden/server:latest?" — the compose file can't answer; the string never changes.
• Revert is currently a lie. updaterRollbackApp restores the data snapshot and re-runs compose up — but the local image is already the new latest, so you get new code on old data. History records from → to, and for a floating tag both are the same string.

The one idea

Keep the floating tag as the channel; pin the digest in the live compose. image: vaultwarden/server:latest@sha256:ab12…

Compose supports tag@digest natively. The tag stays as the human-readable statement of what we track ("latest", "stable", "31-fpm-alpine"); the digest pins exactly which build runs. Then:

• Detect: resolve the channel tag at the registry (docker manifest inspect, no pull) and compare digests → a truthful update_available, even for latest.
• Update: snapshot → rewrite the pin to the new digest → pull → up. Nothing moves unless the user (or a future policy) says so.
• Revert: restore the data snapshot and rewrite the pin back to the previous digest. Now rollback means what the button says.

Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only latest, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a pinned tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the live copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's set-compose-image op is a working, drift-guarded, undoable sed for precisely this line).

Multi-service apps — the anchor, not the first line

10 of 33 apps are multi-service, and they break three different naive assumptions (this is why "which image is the app's version?" can't be answered positionally):

• Nextcloud (4 services): most images aren't the app. nextcloud:31-fpm-alpine is the app; mariadb:11 + redis:alpine + nginx:alpine are infra sidecars. The version the user cares about lives on one of four lines.
• ollama (2 services): the first image isn't the app. File order is ghcr.io/open-webui/open-webui:main then ollama/ollama:latest. grep -m1 image: reports open-webui as the version of an app named ollama — a live mislabel (fixed 2026-07-17).
• jitsi (4 services): one app, one lock-step version, four images. jitsi/{web,prosody,jicofo,jvb}:stable share one release channel and must move together.

Design consequences:

• The version anchor = the app's <slug>-service image (underscores → hyphens: libreportal_catalog → libreportal-catalog-service). Every app defines one (33/33), and it correctly resolves ollama to ollama-service. This is the primary for updates.json's top-level fields, so the existing UI works unchanged. Helper: updaterPrimaryImage (webui_updater_scan.sh).
• Track every image: line, keyed by service — updates.json gains a services[] array. An app "has an update" if any tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services").
• Lock-step sets → one channel drives several image lines. A single CFG_<APP>_VERSION (see §2) can feed every jitsi/* sentinel, so the four move as a unit.
• Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions. They get pinned + CVE-scanned (you do want redis CVEs), but they're not "the app version" and are candidates for fleet-shared pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable.

2. Version identity — three facts, three sources, two types

The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing three different questions into one word, "version." They have different answers and you need all three:

Question	Source	Who sets it	Nextcloud e.g.
What version do I want?	CFG_<APP>_VERSION → drives the anchor image's tag via the tagging system	user (defaults to the template's current tag)	31 or latest
What's exactly running?	the resolved digest, recorded next to it	machine (on install / apply)	sha256:ab12…
What's available?	registry lookup on that channel tag	machine (scan)	32 has shipped

CFG_<APP>_VERSION is the source of truth, not a second one: the compose image tag is derived from it through the #LIBREPORTAL|<APP>_VERSION_TAG|<default> sentinel on the anchor's image: line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The digest is a fingerprint, not the identity — it disambiguates "latest yesterday vs today" and makes revert exact, but it's recorded alongside the config channel, never the thing the user edits. This is what lets a user install/pin a specific version (set the config, apply) while latest stays the default for hands-off freshness.

Two version types (this is the "specific types" idea)

Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata):

• Versioned — anchor tag is a real version (nextcloud:31, mastodon:v4.2.0, navidrome via its OCI label 0.62.0). UX: show the number, offer a version picker, "31 → 32". update_available when a newer tag or a newer build of the pinned tag exists.
• Rolling — anchor tag is a floating channel (vaultwarden/server:latest, gitea:latest, jitsi/*:stable; ~32 of 50 lines). There is no upstream number to compare, so UX shows the channel + build freshness: "latest · new build available (2 days newer)". update_available is purely a digest change.

The digest comparison is the one uniform engine underneath both types — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a display enrichment on top, as good as each upstream allows, never the detection mechanism.

Human-readable version (display enrichment)

Digests are unreadable, so for display resolve in order: CFG_<APP>_VERSION (if set) → OCI label org.opencontainers.image.version (docker image inspect; verified present on e.g. navidrome 0.62.0, trivy 0.72.0, absent on nginx/alpine) → meaningful tag (not latest/untagged) → short digest (ab12cde). Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it).

updates.json additions

(existing fields keep their meaning) per app type (versioned|rolling), channel (the anchor tag), pinned_digest, available_digest, version (display, resolved as above), available_version, services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }, plus the recovery fields the frontend already renders (dr_ready, last_snapshot, last_snapshot_version, last_snapshot_at) from existing backup data. History entries record full pinned refs (repo:tag@sha256:…) in from/to, which is what makes revert-from-history possible.

3. Detection mechanics

• docker manifest inspect <repo:tag> (rootless, via dockerCommandRunInstallUser) returns the registry digest without pulling; compare against the pinned digest. Local RepoDigests and the registry manifest-list digest are the same identity, so the comparison is exact.
• Throttle separately from the scan. The 30-min updater check auto stat-gate stays cheap; registry lookups get their own stamp + knob, mirroring the established /tmp stamp idiom: CFG_UPDATER_REGISTRY_INTERVAL (minutes, default 360, 0 = manual-only). ~30 Docker Hub images once per 6 h sits comfortably under anonymous rate limits; add small jitter so fleets don't thundering-herd.
• Failure = stale, not error. A registry timeout keeps the previous verdict and stamps last_checked; never flip an app to "unscanned" because the network blinked (per-registry short-circuit after the first timeout in a run).
• Reconcile existing installs: on first scan, any installed app whose compose line is unpinned gets pinned to its currently running image's digest (RepoDigests) — zero behaviour change, purely recording reality. New installs pin right after the first pull. Apps using build: instead of image: are out of scope (scan already ignores them).

4. Apply, revert, and the per-app surface

updaterApplyApp becomes: resolve new digest (from updates.json) → snapshot (unchanged, fail-closed) → rewrite pin(s) via the factored-out first-class helper shared with the artifact op's sed → compose pull → up → history ok with digest-refs → rescan. On failure: auto-rollback now also rewrites the pins back before restoring the snapshot — that closes the "new code on old data" hole.

updaterRollbackApp (the user-facing Roll back button): read the last update/ok history entry's from refs → rewrite pins → restore data snapshot → up → history rolled-back. This is the "allow users to revert back" requirement, and it only works because of the pins.

UI: no new navigation. The per-app Updates tab gains a Version section (already scaffolded in renderAppDetail): channel · running build · available build · Update / Roll back. The fleet tab's rows finally show real current → available. Everything routes through the existing updater_apply / updater_rollback tasks — mutations stay task-only, no new API endpoints.

5. CVE scanning — decision

Three candidate shapes were on the table:

Shape	Verdict
A. Scanner as a catalog app (trivy in the App Center, recommended alongside CrowdSec)	✅ Chosen (2026-07-14). Installing/removing the scanner is the opt-in switch, visible and revertible like every other app; fits "everything is an app"; sits next to CrowdSec as the second security recommendation at first install.
B. Scanner binary on the host	❌ Grows the host footprint we've deliberately kept lean (rootless + de-sudo); another thing to install/update outside the app model.
C. Invisible ephemeral container — the scan task does docker run --rm per scan, no catalog presence	❌ Zero idle cost, but the scanner is invisible: no App Center card, no obvious opt-in/uninstall, and users can't see what has socket access. Visibility won.

So: trivy ships as a normal catalog app (containers/trivy/), marked recommended and listed in the first-install recommended apps directly under CrowdSec. The updater's CVE step is gated on the app being installed — no trivy app, no scan, honest empty state pointing at the App Center ("Install the security scanner to enable CVE reports").

Scanner: Trivy (over grype — both fine, trivy has the larger ecosystem, single pinned OCI image, clean JSON, Apache-2.0). Details:

• The app runs trivy server (its long-running mode): one modest container that owns the vulnerability DB (~600 MB volume) and keeps it fresh. The updater scan executes scans through the app's container (docker exec <trivy> trivy image --format json <image>) — no per-scan container spawn, no host binary, and the DB downloads once instead of per run.
• Image access via the rootless docker socket mounted read-only into the trivy container (same daemon that owns the images) — the same trusted position CrowdSec-style security tooling already occupies, but here it's a visible, uninstallable app.
• Privacy posture (worth stating in user-facing copy): nothing about your images or apps ever leaves the box — Trivy matches locally against a downloaded DB; the only outbound call is the DB fetch from ghcr. The opt-in is installing the app (recommended, pre-ticked at first install like the other recommendations — skipping it = air-gap-friendly); CFG_UPDATER_CVE_INTERVAL (minutes, default 1440 — daily; scans are the expensive step, results don't change hourly) lives in configs/webui/webui_updater next to the existing knobs.
• Output maps straight onto the schema the frontend already renders: per app cves[] { id, severity, package, fixed_in, url } + global totals. Dedupe per image (shared base layers repeat findings), scan each distinct image once per run. Drop the [ ! -f ] guard so re-scans overwrite cves.json.
• Not alarmist (house rule): the Security tab lists everything, but the per-app chip/badge only fires for critical/high with a fix available — "your box has 400 unfixable medium CVEs" red badges are noise, not signal.
• Tie-in with updates: a CVE whose fixed_in is satisfied by the available build marks that update as a security update — the Security filter chip and severity sort already exist in the UI, they just start meaning something.

6. Auto-update policy — BUILT 2026-08-11 (per-app, not severity-split)

Shipped a shape the original sketch didn't consider: the policy is per app, not one fleet-wide severity rule.

• CFG_<APP>_UPDATE_TYPE=auto|manual, default auto — in all 33 app config templates (not libreportal, whose own updates are a different subsystem). Reaches existing installs through the normal template reconcile, and shows up on the app's Configure page as Updates (field mapping UPDATE_TYPE, category general).
• CFG_UPDATER_AUTO=true|false, default true — the master switch in configs/webui/webui_updater. It can only ever make things more manual, so "stop auto-updating everything" is one flip, not 33 edits. Precedence deliberately mirrors backupResolveStrategy (per-app override on top of a global default); updaterAppPolicy is the single resolver, used by both the enqueuer and the scan.
• updaterApplyAuto (cli/commands/updater/cli_updater_auto.sh), called at the end of updater check, enqueues the ordinary updater_apply task for every update-available app whose policy is auto — never applies inline, exactly like artifactApplyAuto. So an automatic update is byte-for-byte the manual path: snapshot → pull → up → auto-rollback on failure, one task log, one History entry, same Roll back button.
• No retry storms. Each attempt stamps the target digest under generated/auto/<app>.digest; the same digest is never auto-attempted twice. A broken upstream build is tried once, rolled back, then left alone (the Update button still offers a manual retry) until a newer build changes the digest. In-flight tasks are also skipped, so a scan landing mid-update can't stack a second one.
• Tracked, not silent. updates.json carries each app's resolved update_type, and History entries carry trigger: manual|auto. The WebUI reads both: the fleet list says whether updates install themselves, rows that opted out carry a manual chip, and automatic History entries are labelled. The Overview board treats an auto app's pending update as a quiet line, not a "needs action" one — it doesn't need a decision.

Why not the sketched off|security|all severity split: severity is a property of the CVE data, which is only as complete as Trivy's per-image scan, whereas "do I let this app move on its own?" is a property of the app — nextcloud and a DNS blocker are not the same risk. Per-app also composes with the CVE work instead of competing with it: a future security value can be added per app without changing the master switch or the enqueuer's shape.

Soaked 2026-08-11: the first live end-to-end auto-update (trivy 0.72.0→0.73.0) ran on a real install. Its first attempt exposed and fixed a latent dynamic-scoping bug (a callee's undeclared while read app loop blanked the updater's app local mid-update — pre-existing, had also corrupted a manual update's History on 08-10); the retry completed cleanly with trigger:auto and exact digest refs.

Refinements (2026-08-11), after the "hands-off background" review:

• Install window — CFG_UPDATER_WINDOW (HH:MM-HH:MM host local time, wraps midnight, always; default 06:00-08:00, right after the 05:00 backup cron). Gates only the enqueue in updaterApplyAuto; scans keep their all-day cadence so detection stays current and pending updates visibly wait. Malformed values fail closed (hold updates) and are rejected by the WebUI validator; the raw value is surfaced in updates.json (auto_window) so the UI states when installs land. Ordering rationale: backups nightly at 05:00 → updates 06:00-08:00, and every update still takes its own pre-update snapshot regardless (fail-closed), so the window is about quiet hours, not safety.
• Check now forces a live registry lookup — updater check (non-auto) sets UPDATER_REGISTRY_FORCE=1; previously the flag existed but nothing set it, so the button silently reused the 6h digest cache. Force also overrides CFG_UPDATER_REGISTRY_INTERVAL=0, making 0 mean "manual-only" as originally intended.
• Registry stamp out of /tmp — the task-processor unit runs with systemd PrivateTmp, so the /tmp stamp existed once per namespace (daemon vs CLI, daemon's resetting every restart). Now at <system>/logs/updater_registry_checked.
• Failed-auto honesty — the scan emits each app's auto_attempted_digest (the one-shot no-retry stamp). When it matches available_digest, the UI stops claiming "installs automatically": the per-app detail says the build failed and won't retry, the fleet row gets an auto failed chip, and the Overview board counts it as waiting for you (warn) instead of quietly-installing (ok).
• Failure notifications (2026-08-11) — libreportal notify + configs/general/general_notifications: email (rides the Mail settings), ntfy, Gotify, Discord, Slack, Telegram, Pushover. One hook in the task processor reports every terminal task; the notify command applies the policy (CFG_NOTIFY_EVENTS=failures|all|off, cancelled never notifies). A failed auto-update now reaches the user with "already rolled back, won't retry" copy without the WebUI being open. Verified end-to-end through the live daemon against a mock endpoint.
• CFG_TIMEZONE label corrected — it only sets containers' TZ; scheduled tasks follow the host clock. The old "System Timezone - Timezone for scheduled tasks" label promised a knob that never existed.

7. Build phases (each independently shippable)

0. P0 — Anchor fix. ✅ done 2026-07-17. updaterPrimaryImage resolves each app's version image from its <slug>-service, not the first image: line — fixes the ollama mislabel. Used by both the version and CVE loops.
1. P1 — Config-first pin foundation. ✅ done 2026-07-17. (b) CFG_<APP>_VERSION drives the anchor tag via a #LIBREPORTAL|<APP>_VERSION_TAG|<tag> sentinel on every app's anchor image line (32 apps, opt-in, default = today's tag = no-op). (c) Digest recorded/reconciled by the scanner itself (no install hook needed — it inspects the running image each scan). type classified from the anchor tag. Shared image-line digest-rewrite = updaterSetAnchorRef (anchor-aware; artifact-op sharing deferred).
2. P2 — Real detection + display. ✅ done 2026-07-17. Registry digest compare (docker buildx imagetools inspect = the RepoDigest identity) in webuiUpdaterScan behind CFG_UPDATER_REGISTRY_INTERVAL (throttled, reuse-between-windows); updates.json carries type/channel/services[]/digests + resolved display version (config → OCI label → tag → short digest). Fleet + Updates tabs show truthful data. Note: digest-compare fully detects rolling new-builds; versioned newer-version enumeration (tag listing) is a P5 add. Recovery fields still inferred from History (deferred).
3. P3 — Pinned apply/revert. ✅ done 2026-07-17. updaterApplyApp records exact repo:tag@sha256:… from→to and un-pins before pulling; updaterRollbackApp re-pins the anchor to the pre-update build's digest before up (closes "new code on old data"). History carries digest refs. (Helpers unit-tested; end-to-end apply/revert not yet exercised on a live install with a pending update.)
4. P4 — CVE scanner. The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; cves.json for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4.
5. P5 — Polish. Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows.
6. P6 — Auto-update policy. ✅ done 2026-08-11. Per-app CFG_<APP>_UPDATE_TYPE (default auto) under the CFG_UPDATER_AUTO master switch; updaterApplyAuto enqueues the normal apply task, one attempt per target digest, policy + trigger surfaced in updates.json/history.json and the WebUI. See the rewritten §6. (Landed ahead of P3's live soak at the maintainer's call — the snapshot-first apply and the one-shot digest stamp are what make that acceptable.)

8. Rejected alternatives (for the record)

• CFG_<APP>_VERSION config vars — ADOPTED 2026-07-17 (see §2). Original objection: "second source of truth vs the compose file, collides with the _TAG namespace, doesn't solve floating latest." Reassessed: the tagging system makes the config the source (compose tag derived from it, like every other config value), and CFG_<APP>_VERSION → <APP>_VERSION_TAG fits the naming convention with no collision. The only surviving point — doesn't detect floating latest alone — is real and is exactly why the digest fingerprint rides alongside it; config = channel, digest = build.
• Pure "compare version numbers" (no digest) — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, two types).
• Pin semver tags across all templates — permanent curation burden, inconsistent upstreams, still blind to republished tags.
• Watchtower-style auto-pull of latest — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back." Note §6's auto default is not this: it is the same deliberate, snapshotted, revertible apply the button runs, scheduled — visible in Tasks and History, per-app switchable, and never retried blindly.
• Docker Scout / hosted scanners — requires accounts / sends data off-box; against the ethos.
• A new top-level "Versions" area — unnecessary; the fleet Updates tab + per-app Updates tab already are that surface, they just need real data.