Merge claude/2
This commit is contained in:
librelad
commit
fb4ad9a555
@ -1,6 +1,13 @@
|
||||
# LibrePortal — Per-App Version Updater & CVE Scanning (Roadmap / Design)
|
||||
|
||||
**Status:** Design — **not built.** Agreed direction 2026-07-14; fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pin/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing.
|
||||
**Status:** Design — **not built** (except the Trivy/CVE half, shipped 2026-07-16). Fills the two deliberately-unwired hooks in `webui_updater_scan.sh`. · **Audience:** us, future-self · **Scope:** real "update available" detection, per-app version identity (pick/track/revert), and the vulnerability scanner behind `/apps/overview/updates` · **Origin:** "the updates system is currently not set up" conversation 2026-07-14. Companion to [updates-and-distribution.md](updates-and-distribution.md) — that doc owns the *signed artifact/hotfix pipe*; this one owns the *generators* it always treated as pre-existing.
|
||||
|
||||
> **Revised 2026-07-17 — config-first identity + version types + anchor fix.** Three changes from the original direction, after auditing the compose corpus (33 apps, 50 image lines) and the `#LIBREPORTAL|…|…` tagging system:
|
||||
> 1. **`CFG_<APP>_VERSION` is adopted, not rejected.** The tagging system (`scripts/config/tags/processors/tags_processor_app_config_values.sh`) already derives compose values from `CFG_<APP>_<KEY>` config — so a version config makes the config *the* source and the image tag *derived*, not a second source of truth. The original §8 rejection assumed no such mechanism. See revised §2 / §8.
|
||||
> 2. **Two version *types*, not one model.** *Versioned* apps track a real semver-ish tag (nextcloud `31`, mastodon `v4.2.0`); *rolling* apps track a floating channel (`latest`/`stable`, ~32 of 50 lines). Version numbers are only comparable for the first kind; the **digest is the universal fingerprint** that makes "update available?" answerable for *both*. This is the answer to "why can't it just compare version numbers" — most upstreams don't publish a comparable one. See revised §2.
|
||||
> 3. **The per-app version anchor is the `<slug>-service` image, not the first `image:` line.** Every app defines a `<slug>-service` (33/33); a few list a companion first (ollama: `ollama-webui`/open-webui before `ollama-service`), so `grep -m1 image:` mis-picks. Fixed live 2026-07-17 (`updaterPrimaryImage`). See revised §1.5.
|
||||
>
|
||||
> The digest-pinning idea (below) is unchanged in mechanism but **demoted from "source of truth" to "exact-build fingerprint recorded alongside the config channel."**
|
||||
|
||||
---
|
||||
|
||||
@ -43,16 +50,49 @@ Compose supports `tag@digest` natively. The tag stays as the human-readable stat
|
||||
|
||||
Why not just pin semver tags in all ~45 templates instead? Because upstreams are inconsistent (some publish semver, some only `latest`, linuxserver uses their own scheme), it's a permanent manual maintenance burden, and it still wouldn't tell us when a *pinned* tag's build is republished. Digest pinning is universal, automatic, and template-free — templates stay exactly as they are; pinning happens on the **live** copy at install time, which is already the mutable source of truth (template→live copy-then-edit is the established model, and the artifact system's `set-compose-image` op is a working, drift-guarded, undoable sed for precisely this line).
|
||||
|
||||
### Multi-service apps
|
||||
### Multi-service apps — the anchor, not the first line
|
||||
|
||||
Nextcloud is 4 images, jitsi is 4. The scan currently reads only the first `image:` line. Design: **pin and track every `image:` line, keyed by service**. `updates.json` keeps its per-app top-level fields (primary service — first entry — so the existing UI works unchanged) and gains a `services[]` array; the UI can later aggregate ("1 of 4 services has an update"). An app "has an update" if *any* service does; apply updates all outdated services in one snapshot.
|
||||
10 of 33 apps are multi-service, and they break three *different* naive assumptions (this is why "which image is the app's version?" can't be answered positionally):
|
||||
|
||||
## 2. Version identity — what we record and show
|
||||
- **Nextcloud (4 services): most images aren't the app.** `nextcloud:31-fpm-alpine` is the app; `mariadb:11` + `redis:alpine` + `nginx:alpine` are infra sidecars. The version the user cares about lives on *one* of four lines.
|
||||
- **ollama (2 services): the first image isn't the app.** File order is `ghcr.io/open-webui/open-webui:main` **then** `ollama/ollama:latest`. `grep -m1 image:` reports open-webui as the version of an app *named ollama* — a live mislabel (fixed 2026-07-17).
|
||||
- **jitsi (4 services): one app, one lock-step version, four images.** `jitsi/{web,prosody,jicofo,jvb}:stable` share one release channel and must move together.
|
||||
|
||||
- **Source of truth:** the pin in the live compose file. No parallel `CFG_<APP>_VERSION` variable — that would be a second source of truth that drifts, and the `_TAG` namespace already means template placeholders here (a real footgun).
|
||||
- **Human-readable version:** digests are unreadable, so enrich for display: `docker image inspect` the local image for the OCI label `org.opencontainers.image.version` (most linuxserver/ghcr images carry it). Display order: label → tag (if not latest/untagged) → short digest (`sha256:ab12…` → `ab12cde`). `available_version` starts life as "new build of *latest*" plus short digest; label enrichment for the *remote* side is best-effort later (needs a config-blob fetch — not worth blocking on).
|
||||
- **`updates.json` additions** (existing fields keep their meaning): per app `channel` (the tag), `pinned_digest`, `available_digest`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version }`, plus the recovery fields the frontend already knows how to render (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) sourced from the backup data that already exists.
|
||||
- **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible.
|
||||
Design consequences:
|
||||
|
||||
- **The version anchor = the app's `<slug>-service` image** (underscores → hyphens: `libreportal_catalog` → `libreportal-catalog-service`). Every app defines one (33/33), and it correctly resolves ollama to `ollama-service`. This is the primary for `updates.json`'s top-level fields, so the existing UI works unchanged. Helper: `updaterPrimaryImage` (`webui_updater_scan.sh`).
|
||||
- **Track every `image:` line, keyed by service** — `updates.json` gains a `services[]` array. An app "has an update" if *any* tracked service does; apply updates all outdated services in one snapshot; the UI can aggregate ("1 of 4 services").
|
||||
- **Lock-step sets → one channel drives several image lines.** A single `CFG_<APP>_VERSION` (see §2) can feed every `jitsi/*` sentinel, so the four move as a unit.
|
||||
- **Sidecars (mariadb/redis/nginx) are tracked-by-digest, not headline versions.** They get pinned + CVE-scanned (you *do* want redis CVEs), but they're not "the app version" and are candidates for **fleet-shared** pinning rather than per-app (redis appears in 4 apps, mariadb/nginx in 2 each) — a secondary decision, deferrable.
|
||||
|
||||
## 2. Version identity — three facts, three sources, two types
|
||||
|
||||
The recurring confusion ("shouldn't it just give each app a version and compare to latest?") comes from collapsing **three different questions** into one word, "version." They have different answers and you need all three:
|
||||
|
||||
| Question | Source | Who sets it | Nextcloud e.g. |
|
||||
|---|---|---|---|
|
||||
| **What version do I *want*?** | `CFG_<APP>_VERSION` → drives the anchor image's tag via the tagging system | **user** (defaults to the template's current tag) | `31` or `latest` |
|
||||
| **What's *exactly* running?** | the resolved digest, recorded next to it | machine (on install / apply) | `sha256:ab12…` |
|
||||
| **What's *available*?** | registry lookup on that channel tag | machine (scan) | `32` has shipped |
|
||||
|
||||
`CFG_<APP>_VERSION` is **the** source of truth, not a second one: the compose image tag is *derived* from it through the `#LIBREPORTAL|<APP>_VERSION_TAG|<default>` sentinel on the anchor's `image:` line, exactly as ports/IPs/domains already derive from their configs. Undefined key → the template default stands, so this is opt-in per app and a no-op until someone sets it. The **digest is a fingerprint, not the identity** — it disambiguates "`latest` yesterday vs today" and makes revert exact, but it's recorded *alongside* the config channel, never the thing the user edits. This is what lets a user **install/pin a specific version** (set the config, apply) while `latest` stays the default for hands-off freshness.
|
||||
|
||||
### Two version *types* (this is the "specific types" idea)
|
||||
|
||||
Not every app can offer the same version UX, because upstreams don't all publish comparable version numbers. So classify each app (derivable from the anchor tag, overridable in app metadata):
|
||||
|
||||
- **Versioned** — anchor tag is a real version (`nextcloud:31`, `mastodon:v4.2.0`, `navidrome` via its OCI label `0.62.0`). UX: show the number, offer a version picker, "31 → 32". `update_available` when a newer tag or a newer build of the pinned tag exists.
|
||||
- **Rolling** — anchor tag is a floating channel (`vaultwarden/server:latest`, `gitea:latest`, `jitsi/*:stable`; ~32 of 50 lines). There is no upstream number to compare, so UX shows the **channel + build freshness**: "latest · new build available (2 days newer)". `update_available` is purely a digest change.
|
||||
|
||||
The **digest comparison is the one uniform engine underneath both types** — that's what keeps the system simple: track a tag, compare its digest to what's running, offer update/revert. Version numbers are a *display enrichment* on top, as good as each upstream allows, never the detection mechanism.
|
||||
|
||||
### Human-readable version (display enrichment)
|
||||
|
||||
Digests are unreadable, so for display resolve in order: **`CFG_<APP>_VERSION` (if set) → OCI label `org.opencontainers.image.version` (`docker image inspect`; verified present on e.g. navidrome `0.62.0`, trivy `0.72.0`, absent on nginx/alpine) → meaningful tag (not `latest`/untagged) → short digest (`ab12cde`)**. Remote-side label enrichment is best-effort later (needs a config-blob fetch — don't block on it).
|
||||
|
||||
### `updates.json` additions
|
||||
|
||||
(existing fields keep their meaning) per app `type` (`versioned|rolling`), `channel` (the anchor tag), `pinned_digest`, `available_digest`, `version` (display, resolved as above), `available_version`, `services[] { service, image, channel, pinned_digest, available_digest, update_available, version, is_anchor, is_sidecar }`, plus the recovery fields the frontend already renders (`dr_ready`, `last_snapshot`, `last_snapshot_version`, `last_snapshot_at`) from existing backup data. **History** entries record full pinned refs (`repo:tag@sha256:…`) in `from`/`to`, which is what makes revert-from-history possible.
|
||||
|
||||
## 3. Detection mechanics
|
||||
|
||||
@ -96,8 +136,9 @@ Once detection + pinned apply are trustworthy, add `CFG_UPDATER_AUTO=off|securit
|
||||
|
||||
## 7. Build phases (each independently shippable)
|
||||
|
||||
1. **P1 — Pin foundation.** Factor the compose image-line rewrite into a shared helper (artifact op + updater both use it, all `image:` lines not just the first). Pin-on-install + reconcile-on-first-scan. No UI change yet.
|
||||
2. **P2 — Real detection.** Registry digest compare in `webuiUpdaterScan` behind the new interval knob; `updates.json` gains services/digests; fleet + per-app tabs light up with truthful data. Emit recovery fields while in there.
|
||||
0. **P0 — Anchor fix.** *(done 2026-07-17)* `updaterPrimaryImage` resolves each app's version image from its `<slug>-service`, not the first `image:` line — fixes the ollama mislabel. Used by both the version and CVE loops.
|
||||
1. **P1 — Config-first pin foundation.** (a) Wire the anchor `image:` line into the tagging system: add a `#LIBREPORTAL|<APP>_VERSION_TAG|<current-tag>` sentinel so `CFG_<APP>_VERSION` drives the tag (opt-in; default = today's tag = no-op). (b) Factor the compose image-line **digest** rewrite into a shared helper (artifact op + updater both use it, all `image:` lines keyed by service). (c) Record resolved digest on install + reconcile-on-first-scan. Classify `type` (versioned/rolling) from the anchor tag. No UI change yet.
|
||||
2. **P2 — Real detection + display.** Registry digest compare in `webuiUpdaterScan` behind the new interval knob; `updates.json` gains `type`/`services[]`/digests + resolved display `version` (config → OCI label → tag → short digest); fleet + per-app tabs light up with truthful data, versioned vs rolling rendered per type. Emit recovery fields while in there.
|
||||
3. **P3 — Pinned apply/revert.** `updaterApplyApp`/`updaterRollbackApp` rewrite pins as in §4; history carries digest refs; failure path restores pins. *After this, the Update and Roll back buttons are honest.*
|
||||
4. **P4 — CVE scanner.** The trivy app (catalog + recommended + first-install list) plus the updater's scan step through it; `cves.json` for real, guard dropped, totals + severity chips live, security-update tie-in. The app itself can ship ahead of the rest of P4.
|
||||
5. **P5 — Polish.** Per-app critical-CVE / update chip on the app header (pattern exists for improvements), remote version-label enrichment, "N services" aggregation in rows.
|
||||
@ -105,7 +146,8 @@ Once detection + pinned apply are trustworthy, add `CFG_UPDATER_AUTO=off|securit
|
||||
|
||||
## 8. Rejected alternatives (for the record)
|
||||
|
||||
- **`CFG_<APP>_VERSION` config vars** — second source of truth vs the compose file, collides with the `_TAG` placeholder namespace, and doesn't solve floating `latest` by itself.
|
||||
- **~~`CFG_<APP>_VERSION` config vars~~ — ADOPTED 2026-07-17 (see §2).** Original objection: "second source of truth vs the compose file, collides with the `_TAG` namespace, doesn't solve floating `latest`." Reassessed: the tagging system makes the config *the* source (compose tag derived from it, like every other config value), and `CFG_<APP>_VERSION` → `<APP>_VERSION_TAG` fits the naming convention with no collision. The only surviving point — *doesn't detect floating `latest` alone* — is real and is exactly why the **digest fingerprint** rides alongside it; config = channel, digest = build.
|
||||
- **Pure "compare version numbers" (no digest)** — the intuitive model, but most upstreams don't publish a comparable version and there is no universal "latest version of X" API. Would require per-app curated version-source adapters (where to look, how to parse/compare) — permanent, fragile curation. The digest engine works for all apps automatically; version numbers are display enrichment on top (§2, *two types*).
|
||||
- **Pin semver tags across all templates** — permanent curation burden, inconsistent upstreams, still blind to republished tags.
|
||||
- **Watchtower-style auto-pull of latest** — maximum freshness, zero visibility, no revert; the exact opposite of "monitor, update deliberately, roll back."
|
||||
- **Docker Scout / hosted scanners** — requires accounts / sends data off-box; against the ethos.
|
||||
|
||||
@ -22,6 +22,26 @@
|
||||
# output is honest: scanned=true, update_available=false, cves=[]. Always writes
|
||||
# valid JSON even when data gathering fails — it must never emit a broken file.
|
||||
|
||||
# An app's version anchor = the image of its `<slug>-service` service, the
|
||||
# universal per-app primary-service naming convention (33/33 apps). NOT the
|
||||
# first `image:` line: a few apps declare a companion service first — e.g.
|
||||
# ollama lists `ollama-webui` (open-webui) before `ollama-service`
|
||||
# (ollama/ollama) — so `grep -m1 image:` reports the wrong image as the app's
|
||||
# version. Match the primary service header (underscores in the slug become
|
||||
# hyphens in the service name, e.g. libreportal_catalog → libreportal-catalog-
|
||||
# service), then take the first `image:` under it. Falls back to the first
|
||||
# image line for anything that doesn't follow the convention.
|
||||
updaterPrimaryImage() {
|
||||
local app="$1" compose="$2" svc img
|
||||
svc="${app//_/-}-service"
|
||||
img="$(awk -v s="$svc" '
|
||||
m && /^[[:space:]]*image:/ { print; exit }
|
||||
$0 ~ ("^[[:space:]]*" s ":") { m=1 }
|
||||
' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')"
|
||||
[ -n "$img" ] || img="$(grep -m1 -E '^\s*image:' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')"
|
||||
printf '%s' "$img"
|
||||
}
|
||||
|
||||
webuiUpdaterScan() {
|
||||
local out_dir="$containers_dir/libreportal/frontend/data/updater/generated"
|
||||
runFileOp mkdir -p "$out_dir" 2>/dev/null || mkdir -p "$out_dir" 2>/dev/null
|
||||
@ -49,9 +69,7 @@ webuiUpdaterScan() {
|
||||
local entries="" first=1
|
||||
for app in "${apps[@]}"; do
|
||||
local image="" compose="$containers_dir/$app/docker-compose.yml"
|
||||
if [ -f "$compose" ]; then
|
||||
image="$(grep -m1 -E '^\s*image:' "$compose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')"
|
||||
fi
|
||||
[ -f "$compose" ] && image="$(updaterPrimaryImage "$app" "$compose")"
|
||||
# --- available-version hook -------------------------------------------
|
||||
# Wire a registry check here (e.g. `docker manifest inspect`/skopeo) to
|
||||
# set available_image + update_available. Honest default: up to date.
|
||||
@ -96,7 +114,9 @@ EOF
|
||||
&& declare -f trivyScanImageCves >/dev/null 2>&1; then
|
||||
for app in "${apps[@]}"; do
|
||||
local cimg="" ccompose="$containers_dir/$app/docker-compose.yml"
|
||||
[ -f "$ccompose" ] && cimg="$(grep -m1 -E '^\s*image:' "$ccompose" 2>/dev/null | sed -E 's/^\s*image:\s*//; s/["'"'"']//g')"
|
||||
# Same primary-image anchor as the version scan (scanning every
|
||||
# service's image is a documented future step, see the roadmap).
|
||||
[ -f "$ccompose" ] && cimg="$(updaterPrimaryImage "$app" "$ccompose")"
|
||||
[ -n "$cimg" ] || continue
|
||||
local cvej; cvej="$(trivyScanImageCves "$cimg")"
|
||||
[ -z "$cvej" ] || [ "$cvej" = "[]" ] && continue
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user