From 22d7bcf5058a9142d68851078c473376829dabad Mon Sep 17 00:00:00 2001 From: librelad Date: Fri, 17 Jul 2026 22:57:12 +0100 Subject: [PATCH] fix(updater): anchor app version to -service, not first image; revise design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scan read the app's "current image" via `grep -m1 image:` — the first image line. For apps that declare a companion service first this mislabels the app: ollama lists `ollama-webui` (open-webui) before `ollama-service` (ollama/ollama), so an app named ollama reported open-webui's version. Add `updaterPrimaryImage`: resolve the version anchor from the app's `-service` image (the universal primary-service naming convention, 33/33 apps; underscores→hyphens for slugs like libreportal_catalog). Falls back to the first image line off-convention. Used by both the version and CVE loops. Verified: only ollama changes (→ ollama/ollama:latest); nextcloud, mastodon, jitsi, gitea, vaultwarden anchors unchanged. Also revise docs/roadmap/app-version-updater-and-cve.md to the config-first direction agreed this session: - CFG__VERSION ADOPTED (was rejected): the #LIBREPORTAL tagging system makes the config the source (compose tag derived from it), not a second one. - Two version TYPES — versioned (real tag, version picker) vs rolling (floating channel, digest-freshness); digest is the uniform detection engine for both, version numbers are display enrichment. Answers "why not just compare numbers" (most upstreams publish none; no universal latest-version API). - Multi-service anchor = -service, not first line; lock-step sets (jitsi) = one channel → several image lines; sidecars tracked-by-digest, not headline. - Phases updated: P0 anchor (done) → P1 config-first pin → P2 detection+display. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: librelad --- docs/roadmap/app-version-updater-and-cve.md | 64 +++++++++++++++---- .../generators/updater/webui_updater_scan.sh | 28 ++++++-- 2 files changed, 77 insertions(+), 15 deletions(-) diff --git a/docs/roadmap/app-version-updater-and-cve.md b/docs/roadmap/app-version-updater-and-cve.md index 1b52634..aae087f 100644 --- a/docs/roadmap/app-version-updater-and-cve.md +++ b/docs/roadmap/app-version-updater-and-cve.md @@ -1,6 +1,13 @@ # LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design) -**Status:** Design — **not built.** Agreed direction 2026-07-14; fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pin/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing. +**Status:** Design — **not built** (except the Trivy/CVE half, shipped 2026-07-16). Fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing. + +> **Revised 2026-07-17 — config-first identity + version types + anchor fix.** Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the `#LIBREPORTAL|…|…` tagging system: +> 1. **`CFG__VERSION` is adopted, not rejected.** The tagging system (`scripts/config/tags/processors/tags_processor_app_config_values.sh`) already derives compose values from `CFG__` config — so a version config makes the config *the* source and the image tag *derived*, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8. +> 2. **Two version *types*, not one model.** *Versioned* apps track a real semver-ish tag (nextcloud `31`, mastodon `v4.2.0`); *rolling* apps track a floating channel (`latest`/`stable`, ~32 of 50 lines). Version numbers are only comparable for the first kind; the **digest is the universal fingerprint** that makes "update available?" answerable for *both*. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2. +> 3. **The per-app version anchor is the `-service` image, not the first `image:` line.** Every app defines a `-service` (33/33); a few list a companion first (ollama: `ollama-webui`/open-webui before `ollama-service`), so `grep -m1 image:` mis-picks. Fixed live 2026-07-17 (`updaterPrimaryImage`). See revised §1.5. +> +> The digest-pinning idea (below) is unchanged in mechanism but **demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."** --- @@ -43,16 +50,49 @@ Compose supports `tag@digest` natively. The tag stays as the human-readable stat Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only `latest`, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a *pinned* tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the **live** copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's `set-compose-image` op is a working, drift-guarded, undoable sed for precisely this line). -### Multi-service apps +### Multi-service apps — the anchor, not the first line -Nextcloud is 4 images, jitsi is 4. The scan currently reads only the first `image:` line. Design: **pin and track every `image:` line, keyed by service**. `updates.json` keeps its per-app top-level fields (primary service — first entry — so the existing UI works unchanged) and gains a `services[]` array; the UI can later aggregate ("1 of 4 services has an update"). An app "has an update" if *any* service does; apply updates all outdated services in one snapshot. +10 of 33 apps are multi-service, and they break three *different* naive assumptions (this is why "which image is the app's version?" can't be answered positionally): -## 2. Version identity — what we record and show +- **Nextcloud (4 services): most images aren't the app.** `nextcloud:31-fpm-alpine` is the app; `mariadb:11` + `redis:alpine` + `nginx:alpine` are infra sidecars. The version the user cares about lives on *one* of four lines. +- **ollama (2 services): the first image isn't the app.** File order is `ghcr.io/open-webui/open-webui:main` **then** `ollama/ollama:latest`. `grep -m1 image:` reports open-webui as the version of an app *named ollama* — a live mislabel (fixed 2026-07-17). +- **jitsi (4 services): one app, one lock-step version, four images.** `jitsi/{web,prosody,jicofo,jvb}:stable` share one release channel and must move together. -- **Source of truth:** the pin in the live compose file. No parallel `CFG__VERSION` variable — that would be a second source of truth that drifts, and the `_TAG` namespace already means template placeholders here (a real footgun). -- **Human-readable version:** digests are unreadable, so enrich for display: `docker image inspect` the local image for the OCI label `org.opencontainers.image.version` (most linuxserver/ghcr images carry it). Display order: label → tag (if not latest/untagged) → short digest (`sha256:ab12…` → `ab12cde`). `available_version` starts life as "new build of *latest*" plus short digest; label enrichment for the *remote* side is best-effort later (needs a config-blob fetch — not worth blocking on). -- **`updates.json` additions** (existing fields keep their meaning): per app `channel` (the tag), `pinned_digest`, `available_digest`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version }`, plus the recovery fields the frontend already knows how to render (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) sourced from the backup data that already exists. -- **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible. +Design consequences: + +- **The version anchor = the app's `-service` image** (underscores → hyphens: `libreportal_catalog` → `libreportal-catalog-service`). Every app defines one (33/33), and it correctly resolves ollama to `ollama-service`. This is the primary for `updates.json`'s top-level fields, so the existing UI works unchanged. Helper: `updaterPrimaryImage` (`webui_updater_scan.sh`). +- **Track every `image:` line, keyed by service** — `updates.json` gains a `services[]` array. An app "has an update" if *any* tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services"). +- **Lock-step sets → one channel drives several image lines.** A single `CFG__VERSION` (see §2) can feed every `jitsi/*` sentinel, so the four move as a unit. +- **Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions.** They get pinned + CVE-scanned (you *do* want redis CVEs), but they're not "the app version" and are candidates for **fleet-shared** pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable. + +## 2. Version identity — three facts, three sources, two types + +The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing **three different questions** into one word, "version." They have different answers and you need all three: + +| Question | Source | Who sets it | Nextcloud e.g. | +|---|---|---|---| +| **What version do I *want*?** | `CFG__VERSION` → drives the anchor image's tag via the tagging system | **user** (defaults to the template's current tag) | `31` or `latest` | +| **What's *exactly* running?** | the resolved digest, recorded next to it | machine (on install / apply) | `sha256:ab12…` | +| **What's *available*?** | registry lookup on that channel tag | machine (scan) | `32` has shipped | + +`CFG__VERSION` is **the** source of truth, not a second one: the compose image tag is *derived* from it through the `#LIBREPORTAL|_VERSION_TAG|` sentinel on the anchor's `image:` line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The **digest is a fingerprint, not the identity** — it disambiguates "`latest` yesterday vs today" and makes revert exact, but it's recorded *alongside* the config channel, never the thing the user edits. This is what lets a user **install/pin a specific version** (set the config, apply) while `latest` stays the default for hands-off freshness. + +### Two version *types* (this is the "specific types" idea) + +Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata): + +- **Versioned** — anchor tag is a real version (`nextcloud:31`, `mastodon:v4.2.0`, `navidrome` via its OCI label `0.62.0`). UX: show the number, offer a version picker, "31 → 32". `update_available` when a newer tag or a newer build of the pinned tag exists. +- **Rolling** — anchor tag is a floating channel (`vaultwarden/server:latest`, `gitea:latest`, `jitsi/*:stable`; ~32 of 50 lines). There is no upstream number to compare, so UX shows the **channel + build freshness**: "latest · new build available (2 days newer)". `update_available` is purely a digest change. + +The **digest comparison is the one uniform engine underneath both types** — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a *display enrichment* on top, as good as each upstream allows, never the detection mechanism. + +### Human-readable version (display enrichment) + +Digests are unreadable, so for display resolve in order: **`CFG__VERSION` (if set) → OCI label `org.opencontainers.image.version` (`docker image inspect`; verified present on e.g. navidrome `0.62.0`, trivy `0.72.0`, absent on nginx/alpine) → meaningful tag (not `latest`/untagged) → short digest (`ab12cde`)**. Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it). + +### `updates.json` additions + +(existing fields keep their meaning) per app `type` (`versioned|rolling`), `channel` (the anchor tag), `pinned_digest`, `available_digest`, `version` (display, resolved as above), `available_version`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }`, plus the recovery fields the frontend already renders (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) from existing backup data. **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible. ## 3. Detection mechanics @@ -96,8 +136,9 @@ Once detection + pinned apply are trustworthy, add `CFG_UPDATER_AUTO=off|securit ## 7. Build phases (each independently shippable) -1. **P1 — Pin foundation.** Factor the compose image-line rewrite into a shared helper (artifact op + updater both use it, all `image:` lines not just the first). Pin-on-install + reconcile-on-first-scan. No UI change yet. -2. **P2 — Real detection.** Registry digest compare in `webuiUpdaterScan` behind the new interval knob; `updates.json` gains services/digests; fleet + per-app tabs light up with truthful data. Emit recovery fields while in there. +0. **P0 — Anchor fix.** *(done 2026-07-17)* `updaterPrimaryImage` resolves each app's version image from its `-service`, not the first `image:` line — fixes the ollama mislabel. Used by both the version and CVE loops. +1. **P1 — Config-first pin foundation.** (a) Wire the anchor `image:` line into the tagging system: add a `#LIBREPORTAL|_VERSION_TAG|` sentinel so `CFG__VERSION` drives the tag (opt-in; default = today's tag = no-op). (b) Factor the compose image-line **digest** rewrite into a shared helper (artifact op + updater both use it, all `image:` lines keyed by service). (c) Record resolved digest on install + reconcile-on-first-scan. Classify `type` (versioned/rolling) from the anchor tag. No UI change yet. +2. **P2 — Real detection + display.** Registry digest compare in `webuiUpdaterScan` behind the new interval knob; `updates.json` gains `type`/`services[]`/digests + resolved display `version` (config → OCI label → tag → short digest); fleet + per-app tabs light up with truthful data, versioned vs rolling rendered per type. Emit recovery fields while in there. 3. **P3 — Pinned apply/revert.** `updaterApplyApp`/`updaterRollbackApp` rewrite pins as in §4; history carries digest refs; failure path restores pins. *After this, the Update and Roll back buttons are honest.* 4. **P4 — CVE scanner.** The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; `cves.json` for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4. 5. **P5 — Polish.** Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows. @@ -105,7 +146,8 @@ Once detection + pinned apply are trustworthy, add `CFG_UPDATER_AUTO=off|securit ## 8. Rejected alternatives (for the record) -- **`CFG__VERSION` config vars** — second source of truth vs the compose file, collides with the `_TAG` placeholder namespace, and doesn't solve floating `latest` by itself. +- **~~`CFG__VERSION` config vars~~ — ADOPTED 2026-07-17 (see §2).** Original objection: "second source of truth vs the compose file, collides with the `_TAG` namespace, doesn't solve floating `latest`." Reassessed: the tagging system makes the config *the* source (compose tag derived from it, like every other config value), and `CFG__VERSION` → `_VERSION_TAG` fits the naming convention with no collision. The only surviving point — *doesn't detect floating `latest` alone* — is real and is exactly why the **digest fingerprint** rides alongside it; config = channel, digest = build. +- **Pure "compare version numbers" (no digest)** — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, *two types*). - **Pin semver tags across all templates** — permanent curation burden, inconsistent upstreams, still blind to republished tags. - **Watchtower-style auto-pull of latest** — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back." - **Docker Scout / hosted scanners** — requires accounts / sends data off-box; against the ethos. diff --git a/scripts/webui/data/generators/updater/webui_updater_scan.sh b/scripts/webui/data/generators/updater/webui_updater_scan.sh index 015a295..498d42d 100644 --- a/scripts/webui/data/generators/updater/webui_updater_scan.sh +++ b/scripts/webui/data/generators/updater/webui_updater_scan.sh @@ -22,6 +22,26 @@ # output is honest: scanned=true, update_available=false, cves=[]. Always writes # valid JSON even when data gathering fails — it must never emit a broken file. +# An app's version anchor = the image of its `-service` service, the +# universal per-app primary-service naming convention (33/33 apps). NOT the +# first `image:` line: a few apps declare a companion service first — e.g. +# ollama lists `ollama-webui` (open-webui) before `ollama-service` +# (ollama/ollama) — so `grep -m1 image:` reports the wrong image as the app's +# version. Match the primary service header (underscores in the slug become +# hyphens in the service name, e.g. libreportal_catalog → libreportal-catalog- +# service), then take the first `image:` under it. Falls back to the first +# image line for anything that doesn't follow the convention. +updaterPrimaryImage() { + local app="$1" compose="$2" svc img + svc="${app//_/-}-service" + img="$(awk -v s="$svc" ' + m && /^[[:space:]]*image:/ { print; exit } + $0 ~ ("^[[:space:]]*" s ":") { m=1 } + ' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')" + [ -n "$img" ] || img="$(grep -m1 -E '^\s*image:' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')" + printf '%s' "$img" +} + webuiUpdaterScan() { local out_dir="$containers_dir/libreportal/frontend/data/updater/generated" runFileOp mkdir -p "$out_dir" 2>/dev/null || mkdir -p "$out_dir" 2>/dev/null @@ -49,9 +69,7 @@ webuiUpdaterScan() { local entries="" first=1 for app in "${apps[@]}"; do local image="" compose="$containers_dir/$app/docker-compose.yml" - if [ -f "$compose" ]; then - image="$(grep -m1 -E '^\s*image:' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')" - fi + [ -f "$compose" ] && image="$(updaterPrimaryImage "$app" "$compose")" # --- available-version hook ------------------------------------------- # Wire a registry check here (e.g. `docker manifest inspect`/skopeo) to # set available_image + update_available. Honest default: up to date. @@ -96,7 +114,9 @@ EOF && declare -f trivyScanImageCves >/dev/null 2>&1; then for app in "${apps[@]}"; do local cimg="" ccompose="$containers_dir/$app/docker-compose.yml" - [ -f "$ccompose" ] && cimg="$(grep -m1 -E '^\s*image:' "$ccompose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')" + # Same primary-image anchor as the version scan (scanning every + # service's image is a documented future step, see the roadmap). + [ -f "$ccompose" ] && cimg="$(updaterPrimaryImage "$app" "$ccompose")" [ -n "$cimg" ] || continue local cvej; cvej="$(trivyScanImageCves "$cimg")" [ -z "$cvej" ] || [ "$cvej" = "[]" ] && continue