LibrePortal/docs/roadmap/app-version-updater-and-cve.md

# LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design)

**Status:** Design — **not built** (except the Trivy/CVE half, shipped 2026-07-16). Fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing.

> **Revised 2026-07-17 — config-first identity + version types + anchor fix.** Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the `#LIBREPORTAL|…|…` tagging system:
> 1. **`CFG_<APP>_VERSION` is adopted, not rejected.** The tagging system (`scripts/config/tags/processors/tags_processor_app_config_values.sh`) already derives compose values from `CFG_<APP>_<KEY>` config — so a version config makes the config *the* source and the image tag *derived*, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8.
> 2. **Two version *types*, not one model.** *Versioned* apps track a real semver-ish tag (nextcloud `31`, mastodon `v4.2.0`); *rolling* apps track a floating channel (`latest`/`stable`, ~32 of 50 lines). Version numbers are only comparable for the first kind; the **digest is the universal fingerprint** that makes "update available?" answerable for *both*. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2.
> 3. **The per-app version anchor is the `<slug>-service` image, not the first `image:` line.** Every app defines a `<slug>-service` (33/33); a few list a companion first (ollama: `ollama-webui`/open-webui before `ollama-service`), so `grep -m1 image:` mis-picks. Fixed live 2026-07-17 (`updaterPrimaryImage`). See revised §1.5.
>
> The digest-pinning idea (below) is unchanged in mechanism but **demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."**

---

## 0. Where we actually are (so we don't redesign what exists)

Most of the updater is **already built and working**. What's missing is narrow:

| Piece | Status |
|---|---|
| `updater apply` — snapshot → `compose pull` → `up`, auto-rollback on failure | ✅ real (`cli_updater_commands.sh`) |
| `history.json` (fail-closed from→to audit trail) | ✅ real |
| Signed hotfix channel (fetch/verify/apply/undo, `CFG_HOTFIX_AUTO`) | ✅ real |
| Auto-scan scheduling (`updater check auto`, 30-min throttle) | ✅ real |
| WebUI — fleet Updates tab, per-app Updates tab, Security/Recovery/History renderers, Update/Roll-back buttons wired to tasks | ✅ real, waiting on data |
| **"Update available" detection** | ❌ stub — `update_available` is hard-coded `false` (`webui_updater_scan.sh:55-58` is a marked hook) |
| **CVE data** (`cves.json`) | ❌ stub — written empty once, then never rewritten (the `[ ! -f ]` guard at `:82-89` must go) |
| Recovery fields (`dr_ready`, `last_snapshot*`) | ❌ never emitted; frontend infers from History |

So this is **not** a new tab or a new subsystem. The surfaces the feature needs already exist — the fleet **Updates** tab and the **per-app Updates tab** (the "version tab per app" — it's already there, beside Backups). The job is to make the two generators tell the truth, and to fix one real correctness hole described next.

## 1. The core problem: `latest` is a channel, not a version

Across `containers/*/docker-compose.yml` there are 49 active `image:` lines: **21 on `:latest`, 11 untagged (implicit latest), 17 on pinned or floating-stable tags**. Zero are variable-driven. Only nextcloud (`31-fpm-alpine`) and mastodon (`v4.2.0`) are meaningfully pinned among primary apps.

Floating tags break both halves of what we want:

- **Detection is textually impossible.** "Is `vaultwarden/server:latest` newer than `vaultwarden/server:latest`?" — the compose file can't answer; the string never changes.
- **Revert is currently a lie.** `updaterRollbackApp` restores the *data* snapshot and re-runs `compose up` — but the local image is already the *new* latest, so you get **new code on old data**. History records `from → to`, and for a floating tag both are the same string.

### The one idea

> **Keep the floating tag as the *channel*; pin the *digest* in the live compose.**
> `image: vaultwarden/server:latest@sha256:ab12…`

Compose supports `tag@digest` natively. The tag stays as the human-readable statement of *what we track* ("latest", "stable", "31-fpm-alpine"); the digest pins *exactly which build runs*. Then:

- **Detect:** resolve the channel tag at the registry (`docker manifest inspect`, no pull) and compare digests → a truthful `update_available`, even for `latest`.
- **Update:** snapshot → rewrite the pin to the new digest → pull → up. Nothing moves unless the user (or a future policy) says so.
- **Revert:** restore the data snapshot **and** rewrite the pin back to the previous digest. Now rollback means what the button says.

Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only `latest`, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a *pinned* tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the **live** copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's `set-compose-image` op is a working, drift-guarded, undoable sed for precisely this line).

### Multi-service apps — the anchor, not the first line

10 of 33 apps are multi-service, and they break three *different* naive assumptions (this is why "which image is the app's version?" can't be answered positionally):

- **Nextcloud (4 services): most images aren't the app.** `nextcloud:31-fpm-alpine` is the app; `mariadb:11` + `redis:alpine` + `nginx:alpine` are infra sidecars. The version the user cares about lives on *one* of four lines.
- **ollama (2 services): the first image isn't the app.** File order is `ghcr.io/open-webui/open-webui:main` **then** `ollama/ollama:latest`. `grep -m1 image:` reports open-webui as the version of an app *named ollama* — a live mislabel (fixed 2026-07-17).
- **jitsi (4 services): one app, one lock-step version, four images.** `jitsi/{web,prosody,jicofo,jvb}:stable` share one release channel and must move together.

Design consequences:

- **The version anchor = the app's `<slug>-service` image** (underscores → hyphens: `libreportal_catalog` → `libreportal-catalog-service`). Every app defines one (33/33), and it correctly resolves ollama to `ollama-service`. This is the primary for `updates.json`'s top-level fields, so the existing UI works unchanged. Helper: `updaterPrimaryImage` (`webui_updater_scan.sh`).
- **Track every `image:` line, keyed by service** — `updates.json` gains a `services[]` array. An app "has an update" if *any* tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services").
- **Lock-step sets → one channel drives several image lines.** A single `CFG_<APP>_VERSION` (see §2) can feed every `jitsi/*` sentinel, so the four move as a unit.
- **Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions.** They get pinned + CVE-scanned (you *do* want redis CVEs), but they're not "the app version" and are candidates for **fleet-shared** pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable.

## 2. Version identity — three facts, three sources, two types

The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing **three different questions** into one word, "version." They have different answers and you need all three:

| Question | Source | Who sets it | Nextcloud e.g. |
|---|---|---|---|
| **What version do I *want*?** | `CFG_<APP>_VERSION` → drives the anchor image's tag via the tagging system | **user** (defaults to the template's current tag) | `31` or `latest` |
| **What's *exactly* running?** | the resolved digest, recorded next to it | machine (on install / apply) | `sha256:ab12…` |
| **What's *available*?** | registry lookup on that channel tag | machine (scan) | `32` has shipped |

`CFG_<APP>_VERSION` is **the** source of truth, not a second one: the compose image tag is *derived* from it through the `#LIBREPORTAL|<APP>_VERSION_TAG|<default>` sentinel on the anchor's `image:` line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The **digest is a fingerprint, not the identity** — it disambiguates "`latest` yesterday vs today" and makes revert exact, but it's recorded *alongside* the config channel, never the thing the user edits. This is what lets a user **install/pin a specific version** (set the config, apply) while `latest` stays the default for hands-off freshness.

### Two version *types* (this is the "specific types" idea)

Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata):

- **Versioned** — anchor tag is a real version (`nextcloud:31`, `mastodon:v4.2.0`, `navidrome` via its OCI label `0.62.0`). UX: show the number, offer a version picker, "31 → 32". `update_available` when a newer tag or a newer build of the pinned tag exists.
- **Rolling** — anchor tag is a floating channel (`vaultwarden/server:latest`, `gitea:latest`, `jitsi/*:stable`; ~32 of 50 lines). There is no upstream number to compare, so UX shows the **channel + build freshness**: "latest · new build available (2 days newer)". `update_available` is purely a digest change.

The **digest comparison is the one uniform engine underneath both types** — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a *display enrichment* on top, as good as each upstream allows, never the detection mechanism.

### Human-readable version (display enrichment)

Digests are unreadable, so for display resolve in order: **`CFG_<APP>_VERSION` (if set) → OCI label `org.opencontainers.image.version` (`docker image inspect`; verified present on e.g. navidrome `0.62.0`, trivy `0.72.0`, absent on nginx/alpine) → meaningful tag (not `latest`/untagged) → short digest (`ab12cde`)**. Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it).

### `updates.json` additions

(existing fields keep their meaning) per app `type` (`versioned|rolling`), `channel` (the anchor tag), `pinned_digest`, `available_digest`, `version` (display, resolved as above), `available_version`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }`, plus the recovery fields the frontend already renders (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) from existing backup data. **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible.

## 3. Detection mechanics

- `docker manifest inspect <repo:tag>` (rootless, via `dockerCommandRunInstallUser`) returns the registry digest without pulling; compare against the pinned digest. Local `RepoDigests` and the registry manifest-list digest are the same identity, so the comparison is exact.
- **Throttle separately from the scan.** The 30-min `updater check auto` stat-gate stays cheap; registry lookups get their own stamp + knob, mirroring the established `/tmp` stamp idiom: `CFG_UPDATER_REGISTRY_INTERVAL` (minutes, **default 360**, `0` = manual-only). ~30 Docker Hub images once per 6 h sits comfortably under anonymous rate limits; add small jitter so fleets don't thundering-herd.
- **Failure = stale, not error.** A registry timeout keeps the previous verdict and stamps `last_checked`; never flip an app to "unscanned" because the network blinked (per-registry short-circuit after the first timeout in a run).
- **Reconcile existing installs:** on first scan, any installed app whose compose line is unpinned gets pinned to its *currently running* image's digest (`RepoDigests`) — zero behaviour change, purely recording reality. New installs pin right after the first pull. Apps using `build:` instead of `image:` are out of scope (scan already ignores them).

## 4. Apply, revert, and the per-app surface

`updaterApplyApp` becomes: resolve new digest (from `updates.json`) → **snapshot (unchanged, fail-closed)** → rewrite pin(s) via the factored-out first-class helper shared with the artifact op's sed → `compose pull` → `up` → history `ok` with digest-refs → rescan. On failure: auto-rollback now also **rewrites the pins back** before restoring the snapshot — that closes the "new code on old data" hole.

`updaterRollbackApp` (the user-facing Roll back button): read the last `update/ok` history entry's `from` refs → rewrite pins → restore data snapshot → `up` → history `rolled-back`. This is the "allow users to revert back" requirement, and it only works because of the pins.

**UI:** no new navigation. The per-app **Updates** tab gains a Version section (already scaffolded in `renderAppDetail`): channel · running build · available build · Update / Roll back. The fleet tab's rows finally show real `current → available`. Everything routes through the existing `updater_apply` / `updater_rollback` tasks — mutations stay task-only, no new API endpoints.

## 5. CVE scanning — decision

Three candidate shapes were on the table:

| Shape | Verdict |
|---|---|
| **A. Scanner as a catalog app** (trivy in the App Center, recommended alongside CrowdSec) | ✅ **Chosen** (2026-07-14). Installing/removing the scanner is the opt-in switch, visible and revertible like every other app; fits "everything is an app"; sits next to CrowdSec as the second security recommendation at first install. |
| **B. Scanner binary on the host** | ❌ Grows the host footprint we've deliberately kept lean (rootless + de-sudo); another thing to install/update outside the app model. |
| **C. Invisible ephemeral container** — the scan task does `docker run --rm` per scan, no catalog presence | ❌ Zero idle cost, but the scanner is invisible: no App Center card, no obvious opt-in/uninstall, and users can't see what has socket access. Visibility won. |

So: **trivy ships as a normal catalog app** (`containers/trivy/`), marked **recommended** and listed in the first-install recommended apps directly under CrowdSec. The updater's CVE step is gated on the app being installed — no trivy app, no scan, honest empty state pointing at the App Center ("Install the security scanner to enable CVE reports").

**Scanner: Trivy** (over grype — both fine, trivy has the larger ecosystem, single pinned OCI image, clean JSON, Apache-2.0). Details:

- The app runs `trivy server` (its long-running mode): one modest container that owns the vulnerability DB (~600 MB volume) and keeps it fresh. The updater scan executes scans **through the app's container** (`docker exec <trivy> trivy image --format json <image>`) — no per-scan container spawn, no host binary, and the DB downloads once instead of per run.
- Image access via the rootless docker socket mounted read-only into the trivy container (same daemon that owns the images) — the same trusted position CrowdSec-style security tooling already occupies, but here it's a visible, uninstallable app.
- **Privacy posture (worth stating in user-facing copy):** nothing about your images or apps ever leaves the box — Trivy matches locally against a downloaded DB; the only outbound call is the DB fetch from ghcr. The opt-in is installing the app (recommended, pre-ticked at first install like the other recommendations — skipping it = air-gap-friendly); `CFG_UPDATER_CVE_INTERVAL` (minutes, **default 1440** — daily; scans are the expensive step, results don't change hourly) lives in `configs/webui/webui_updater` next to the existing knobs.
- Output maps straight onto the schema the frontend already renders: per app `cves[] { id, severity, package, fixed_in, url }` + global `totals`. Dedupe per image (shared base layers repeat findings), scan each distinct image once per run. **Drop the `[ ! -f ]` guard** so re-scans overwrite `cves.json`.
- **Not alarmist** (house rule): the Security tab lists everything, but the per-app chip/badge only fires for **critical/high with a fix available** — "your box has 400 unfixable medium CVEs" red badges are noise, not signal.
- **Tie-in with updates:** a CVE whose `fixed_in` is satisfied by the available build marks that update as a *security update* — the Security filter chip and severity sort already exist in the UI, they just start meaning something.

## 6. Auto-update policy (deliberately later)

Once detection + pinned apply are trustworthy, add `CFG_UPDATER_AUTO=off|security|all` (default **off**): `security` auto-enqueues `updater_apply` only for security updates (mirrors `CFG_HOTFIX_AUTO`'s severity-split precedent, and like `artifactApplyAuto` it only *enqueues tasks*). Not part of the initial build — auto-updating before the revert story is proven live would be backwards.

## 7. Build phases (each independently shippable)

0. **P0 — Anchor fix.** ✅ *done 2026-07-17.* `updaterPrimaryImage` resolves each app's version image from its `<slug>-service`, not the first `image:` line — fixes the ollama mislabel. Used by both the version and CVE loops.
1. **P1 — Config-first pin foundation.** ✅ *done 2026-07-17.* (b) `CFG_<APP>_VERSION` drives the anchor tag via a `#LIBREPORTAL|<APP>_VERSION_TAG|<tag>` sentinel on every app's anchor image line (32 apps, opt-in, default = today's tag = no-op). (c) Digest recorded/reconciled by the scanner itself (no install hook needed — it inspects the running image each scan). `type` classified from the anchor tag. Shared image-line digest-rewrite = `updaterSetAnchorRef` (anchor-aware; artifact-op sharing deferred).
2. **P2 — Real detection + display.** ✅ *done 2026-07-17.* Registry digest compare (`docker buildx imagetools inspect` = the RepoDigest identity) in `webuiUpdaterScan` behind `CFG_UPDATER_REGISTRY_INTERVAL` (throttled, reuse-between-windows); `updates.json` carries `type`/`channel`/`services[]`/digests + resolved display `version` (config → OCI label → tag → short digest). Fleet + Updates tabs show truthful data. **Note:** digest-compare fully detects *rolling* new-builds; *versioned* newer-version enumeration (tag listing) is a P5 add. Recovery fields still inferred from History (deferred).
3. **P3 — Pinned apply/revert.** ✅ *done 2026-07-17.* `updaterApplyApp` records exact `repo:tag@sha256:…` from→to and un-pins before pulling; `updaterRollbackApp` re-pins the anchor to the pre-update build's digest before `up` (closes "new code on old data"). History carries digest refs. *(Helpers unit-tested; end-to-end apply/revert not yet exercised on a live install with a pending update.)*
4. **P4 — CVE scanner.** The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; `cves.json` for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4.
5. **P5 — Polish.** Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows.
6. **P6 — Auto-update policy** (§6), only after P3 has soaked on a real install.

## 8. Rejected alternatives (for the record)

- **~~`CFG_<APP>_VERSION` config vars~~ — ADOPTED 2026-07-17 (see §2).** Original objection: "second source of truth vs the compose file, collides with the `_TAG` namespace, doesn't solve floating `latest`." Reassessed: the tagging system makes the config *the* source (compose tag derived from it, like every other config value), and `CFG_<APP>_VERSION` → `<APP>_VERSION_TAG` fits the naming convention with no collision. The only surviving point — *doesn't detect floating `latest` alone* — is real and is exactly why the **digest fingerprint** rides alongside it; config = channel, digest = build.
- **Pure "compare version numbers" (no digest)** — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, *two types*).
- **Pin semver tags across all templates** — permanent curation burden, inconsistent upstreams, still blind to republished tags.
- **Watchtower-style auto-pull of latest** — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back."
- **Docker Scout / hosted scanners** — requires accounts / sends data off-box; against the ethos.
- **A new top-level "Versions" area** — unnecessary; the fleet Updates tab + per-app Updates tab already are that surface, they just need real data.