LibrePortal/docs/roadmap/app-version-updater-and-cve.md

LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design)

Status: Design — not built (except the Trivy/CVE half, shipped 2026-07-16). Fills the two deliberately-unwired hooks in webui_updater_scan.sh. · Audience: us, future-self · Scope: real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind /apps/overview/updates · Origin: "the updates system is currently not set up" conversation 2026-07-14. Companion to updates-and-distribution.md — that doc owns the signed artifact/hotfix pipe; this one owns the generators it always treated as pre-existing.

Revised 2026-07-17 — config-first identity + version types + anchor fix. Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the #LIBREPORTAL|…|… tagging system:

1. CFG_<APP>_VERSION is adopted, not rejected. The tagging system (scripts/config/tags/processors/tags_processor_app_config_values.sh) already derives compose values from CFG_<APP>_<KEY> config — so a version config makes the config the source and the image tag derived, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8.
2. Two version types, not one model. Versioned apps track a real semver-ish tag (nextcloud 31, mastodon v4.2.0); rolling apps track a floating channel (latest/stable, ~32 of 50 lines). Version numbers are only comparable for the first kind; the digest is the universal fingerprint that makes "update available?" answerable for both. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2.
3. The per-app version anchor is the <slug>-service image, not the first image: line. Every app defines a <slug>-service (33/33); a few list a companion first (ollama: ollama-webui/open-webui before ollama-service), so grep -m1 image: mis-picks. Fixed live 2026-07-17 (updaterPrimaryImage). See revised §1.5.

The digest-pinning idea (below) is unchanged in mechanism but demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."

---

0. Where we actually are (so we don't redesign what exists)

Most of the updater is already built and working. What's missing is narrow:

Piece	Status
updater apply — snapshot → compose pull → up, auto-rollback on failure	✅ real (cli_updater_commands.sh)
history.json (fail-closed from→to audit trail)	✅ real
Signed hotfix channel (fetch/verify/apply/undo, CFG_HOTFIX_AUTO)	✅ real
Auto-scan scheduling (updater check auto, 30-min throttle)	✅ real
WebUI — fleet Updates tab, per-app Updates tab, Security/Recovery/History renderers, Update/Roll-back buttons wired to tasks	✅ real, waiting on data
"Update available" detection	❌ stub — update_available is hard-coded false (webui_updater_scan.sh:55-58 is a marked hook)
CVE data (cves.json)	❌ stub — written empty once, then never rewritten (the [ ! -f ] guard at :82-89 must go)
Recovery fields (dr_ready, last_snapshot*)	❌ never emitted; frontend infers from History

So this is not a new tab or a new subsystem. The surfaces the feature needs already exist — the fleet Updates tab and the per-app Updates tab (the "version tab per app" — it's already there, beside Backups). The job is to make the two generators tell the truth, and to fix one real correctness hole described next.

1. The core problem: latest is a channel, not a version

Across containers/*/docker-compose.yml there are 49 active image: lines: 21 on :latest, 11 untagged (implicit latest), 17 on pinned or floating-stable tags. Zero are variable-driven. Only nextcloud (31-fpm-alpine) and mastodon (v4.2.0) are meaningfully pinned among primary apps.

Floating tags break both halves of what we want:

• Detection is textually impossible. "Is vaultwarden/server:latest newer than vaultwarden/server:latest?" — the compose file can't answer; the string never changes.
• Revert is currently a lie. updaterRollbackApp restores the data snapshot and re-runs compose up — but the local image is already the new latest, so you get new code on old data. History records from → to, and for a floating tag both are the same string.

The one idea

Keep the floating tag as the channel; pin the digest in the live compose. image: vaultwarden/server:latest@sha256:ab12…

Compose supports tag@digest natively. The tag stays as the human-readable statement of what we track ("latest", "stable", "31-fpm-alpine"); the digest pins exactly which build runs. Then:

• Detect: resolve the channel tag at the registry (docker manifest inspect, no pull) and compare digests → a truthful update_available, even for latest.
• Update: snapshot → rewrite the pin to the new digest → pull → up. Nothing moves unless the user (or a future policy) says so.
• Revert: restore the data snapshot and rewrite the pin back to the previous digest. Now rollback means what the button says.

Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only latest, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a pinned tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the live copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's set-compose-image op is a working, drift-guarded, undoable sed for precisely this line).

Multi-service apps — the anchor, not the first line

10 of 33 apps are multi-service, and they break three different naive assumptions (this is why "which image is the app's version?" can't be answered positionally):

• Nextcloud (4 services): most images aren't the app. nextcloud:31-fpm-alpine is the app; mariadb:11 + redis:alpine + nginx:alpine are infra sidecars. The version the user cares about lives on one of four lines.
• ollama (2 services): the first image isn't the app. File order is ghcr.io/open-webui/open-webui:main then ollama/ollama:latest. grep -m1 image: reports open-webui as the version of an app named ollama — a live mislabel (fixed 2026-07-17).
• jitsi (4 services): one app, one lock-step version, four images. jitsi/{web,prosody,jicofo,jvb}:stable share one release channel and must move together.

Design consequences:

• The version anchor = the app's <slug>-service image (underscores → hyphens: libreportal_catalog → libreportal-catalog-service). Every app defines one (33/33), and it correctly resolves ollama to ollama-service. This is the primary for updates.json's top-level fields, so the existing UI works unchanged. Helper: updaterPrimaryImage (webui_updater_scan.sh).
• Track every image: line, keyed by service — updates.json gains a services[] array. An app "has an update" if any tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services").
• Lock-step sets → one channel drives several image lines. A single CFG_<APP>_VERSION (see §2) can feed every jitsi/* sentinel, so the four move as a unit.
• Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions. They get pinned + CVE-scanned (you do want redis CVEs), but they're not "the app version" and are candidates for fleet-shared pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable.

2. Version identity — three facts, three sources, two types

The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing three different questions into one word, "version." They have different answers and you need all three:

Question	Source	Who sets it	Nextcloud e.g.
What version do I want?	CFG_<APP>_VERSION → drives the anchor image's tag via the tagging system	user (defaults to the template's current tag)	31 or latest
What's exactly running?	the resolved digest, recorded next to it	machine (on install / apply)	sha256:ab12…
What's available?	registry lookup on that channel tag	machine (scan)	32 has shipped

CFG_<APP>_VERSION is the source of truth, not a second one: the compose image tag is derived from it through the #LIBREPORTAL|<APP>_VERSION_TAG|<default> sentinel on the anchor's image: line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The digest is a fingerprint, not the identity — it disambiguates "latest yesterday vs today" and makes revert exact, but it's recorded alongside the config channel, never the thing the user edits. This is what lets a user install/pin a specific version (set the config, apply) while latest stays the default for hands-off freshness.

Two version types (this is the "specific types" idea)

Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata):

• Versioned — anchor tag is a real version (nextcloud:31, mastodon:v4.2.0, navidrome via its OCI label 0.62.0). UX: show the number, offer a version picker, "31 → 32". update_available when a newer tag or a newer build of the pinned tag exists.
• Rolling — anchor tag is a floating channel (vaultwarden/server:latest, gitea:latest, jitsi/*:stable; ~32 of 50 lines). There is no upstream number to compare, so UX shows the channel + build freshness: "latest · new build available (2 days newer)". update_available is purely a digest change.

The digest comparison is the one uniform engine underneath both types — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a display enrichment on top, as good as each upstream allows, never the detection mechanism.

Human-readable version (display enrichment)

Digests are unreadable, so for display resolve in order: CFG_<APP>_VERSION (if set) → OCI label org.opencontainers.image.version (docker image inspect; verified present on e.g. navidrome 0.62.0, trivy 0.72.0, absent on nginx/alpine) → meaningful tag (not latest/untagged) → short digest (ab12cde). Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it).

updates.json additions

(existing fields keep their meaning) per app type (versioned|rolling), channel (the anchor tag), pinned_digest, available_digest, version (display, resolved as above), available_version, services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }, plus the recovery fields the frontend already renders (dr_ready, last_snapshot, last_snapshot_version, last_snapshot_at) from existing backup data. History entries record full pinned refs (repo:tag@sha256:…) in from/to, which is what makes revert-from-history possible.

3. Detection mechanics

• docker manifest inspect <repo:tag> (rootless, via dockerCommandRunInstallUser) returns the registry digest without pulling; compare against the pinned digest. Local RepoDigests and the registry manifest-list digest are the same identity, so the comparison is exact.
• Throttle separately from the scan. The 30-min updater check auto stat-gate stays cheap; registry lookups get their own stamp + knob, mirroring the established /tmp stamp idiom: CFG_UPDATER_REGISTRY_INTERVAL (minutes, default 360, 0 = manual-only). ~30 Docker Hub images once per 6 h sits comfortably under anonymous rate limits; add small jitter so fleets don't thundering-herd.
• Failure = stale, not error. A registry timeout keeps the previous verdict and stamps last_checked; never flip an app to "unscanned" because the network blinked (per-registry short-circuit after the first timeout in a run).
• Reconcile existing installs: on first scan, any installed app whose compose line is unpinned gets pinned to its currently running image's digest (RepoDigests) — zero behaviour change, purely recording reality. New installs pin right after the first pull. Apps using build: instead of image: are out of scope (scan already ignores them).

4. Apply, revert, and the per-app surface

updaterApplyApp becomes: resolve new digest (from updates.json) → snapshot (unchanged, fail-closed) → rewrite pin(s) via the factored-out first-class helper shared with the artifact op's sed → compose pull → up → history ok with digest-refs → rescan. On failure: auto-rollback now also rewrites the pins back before restoring the snapshot — that closes the "new code on old data" hole.

updaterRollbackApp (the user-facing Roll back button): read the last update/ok history entry's from refs → rewrite pins → restore data snapshot → up → history rolled-back. This is the "allow users to revert back" requirement, and it only works because of the pins.

UI: no new navigation. The per-app Updates tab gains a Version section (already scaffolded in renderAppDetail): channel · running build · available build · Update / Roll back. The fleet tab's rows finally show real current → available. Everything routes through the existing updater_apply / updater_rollback tasks — mutations stay task-only, no new API endpoints.

5. CVE scanning — decision

Three candidate shapes were on the table:

Shape	Verdict
A. Scanner as a catalog app (trivy in the App Center, recommended alongside CrowdSec)	✅ Chosen (2026-07-14). Installing/removing the scanner is the opt-in switch, visible and revertible like every other app; fits "everything is an app"; sits next to CrowdSec as the second security recommendation at first install.
B. Scanner binary on the host	❌ Grows the host footprint we've deliberately kept lean (rootless + de-sudo); another thing to install/update outside the app model.
C. Invisible ephemeral container — the scan task does docker run --rm per scan, no catalog presence	❌ Zero idle cost, but the scanner is invisible: no App Center card, no obvious opt-in/uninstall, and users can't see what has socket access. Visibility won.

So: trivy ships as a normal catalog app (containers/trivy/), marked recommended and listed in the first-install recommended apps directly under CrowdSec. The updater's CVE step is gated on the app being installed — no trivy app, no scan, honest empty state pointing at the App Center ("Install the security scanner to enable CVE reports").

Scanner: Trivy (over grype — both fine, trivy has the larger ecosystem, single pinned OCI image, clean JSON, Apache-2.0). Details:

• The app runs trivy server (its long-running mode): one modest container that owns the vulnerability DB (~600 MB volume) and keeps it fresh. The updater scan executes scans through the app's container (docker exec <trivy> trivy image --format json <image>) — no per-scan container spawn, no host binary, and the DB downloads once instead of per run.
• Image access via the rootless docker socket mounted read-only into the trivy container (same daemon that owns the images) — the same trusted position CrowdSec-style security tooling already occupies, but here it's a visible, uninstallable app.
• Privacy posture (worth stating in user-facing copy): nothing about your images or apps ever leaves the box — Trivy matches locally against a downloaded DB; the only outbound call is the DB fetch from ghcr. The opt-in is installing the app (recommended, pre-ticked at first install like the other recommendations — skipping it = air-gap-friendly); CFG_UPDATER_CVE_INTERVAL (minutes, default 1440 — daily; scans are the expensive step, results don't change hourly) lives in configs/webui/webui_updater next to the existing knobs.
• Output maps straight onto the schema the frontend already renders: per app cves[] { id, severity, package, fixed_in, url } + global totals. Dedupe per image (shared base layers repeat findings), scan each distinct image once per run. Drop the [ ! -f ] guard so re-scans overwrite cves.json.
• Not alarmist (house rule): the Security tab lists everything, but the per-app chip/badge only fires for critical/high with a fix available — "your box has 400 unfixable medium CVEs" red badges are noise, not signal.
• Tie-in with updates: a CVE whose fixed_in is satisfied by the available build marks that update as a security update — the Security filter chip and severity sort already exist in the UI, they just start meaning something.

6. Auto-update policy (deliberately later)

Once detection + pinned apply are trustworthy, add CFG_UPDATER_AUTO=off|security|all (default off): security auto-enqueues updater_apply only for security updates (mirrors CFG_HOTFIX_AUTO's severity-split precedent, and like artifactApplyAuto it only enqueues tasks). Not part of the initial build — auto-updating before the revert story is proven live would be backwards.

7. Build phases (each independently shippable)

0. P0 — Anchor fix. ✅ done 2026-07-17. updaterPrimaryImage resolves each app's version image from its <slug>-service, not the first image: line — fixes the ollama mislabel. Used by both the version and CVE loops.
1. P1 — Config-first pin foundation. ✅ done 2026-07-17. (b) CFG_<APP>_VERSION drives the anchor tag via a #LIBREPORTAL|<APP>_VERSION_TAG|<tag> sentinel on every app's anchor image line (32 apps, opt-in, default = today's tag = no-op). (c) Digest recorded/reconciled by the scanner itself (no install hook needed — it inspects the running image each scan). type classified from the anchor tag. Shared image-line digest-rewrite = updaterSetAnchorRef (anchor-aware; artifact-op sharing deferred).
2. P2 — Real detection + display. ✅ done 2026-07-17. Registry digest compare (docker buildx imagetools inspect = the RepoDigest identity) in webuiUpdaterScan behind CFG_UPDATER_REGISTRY_INTERVAL (throttled, reuse-between-windows); updates.json carries type/channel/services[]/digests + resolved display version (config → OCI label → tag → short digest). Fleet + Updates tabs show truthful data. Note: digest-compare fully detects rolling new-builds; versioned newer-version enumeration (tag listing) is a P5 add. Recovery fields still inferred from History (deferred).
3. P3 — Pinned apply/revert. ✅ done 2026-07-17. updaterApplyApp records exact repo:tag@sha256:… from→to and un-pins before pulling; updaterRollbackApp re-pins the anchor to the pre-update build's digest before up (closes "new code on old data"). History carries digest refs. (Helpers unit-tested; end-to-end apply/revert not yet exercised on a live install with a pending update.)
4. P4 — CVE scanner. The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; cves.json for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4.
5. P5 — Polish. Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows.
6. P6 — Auto-update policy (§6), only after P3 has soaked on a real install.

8. Rejected alternatives (for the record)

• CFG_<APP>_VERSION config vars — ADOPTED 2026-07-17 (see §2). Original objection: "second source of truth vs the compose file, collides with the _TAG namespace, doesn't solve floating latest." Reassessed: the tagging system makes the config the source (compose tag derived from it, like every other config value), and CFG_<APP>_VERSION → <APP>_VERSION_TAG fits the naming convention with no collision. The only surviving point — doesn't detect floating latest alone — is real and is exactly why the digest fingerprint rides alongside it; config = channel, digest = build.
• Pure "compare version numbers" (no digest) — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, two types).
• Pin semver tags across all templates — permanent curation burden, inconsistent upstreams, still blind to republished tags.
• Watchtower-style auto-pull of latest — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back."
• Docker Scout / hosted scanners — requires accounts / sends data off-box; against the ethos.
• A new top-level "Versions" area — unnecessary; the fleet Updates tab + per-app Updates tab already are that surface, they just need real data.