A fresh install creates an empty containers root but leaves the rootless
daemon's own container state untouched. Restarting the daemon then runs its
container-restore pass, which resurrects the previous install's containers —
and Docker materialises each missing bind-mount source first, creating an empty
DIRECTORY even where the mount is a file.
That is the trigger behind the <app>.config stub directories: at 19:38:59 the
daemon re-created every missing mount source for a container built 40 minutes
earlier, runc then failed with "not a directory: Are you trying to mount a
directory onto a file", and the abandoned stubs collided with the install's own
copies 18 seconds later.
Add dockerRemoveStrandedContainers, run right after the rootless daemon
restart: remove containers whose compose project directory no longer exists, so
the next restart has nothing to resurrect. Scoped to project directories under
the LibrePortal containers root, so unrelated containers on the host are never
touched, and gated on the daemon answering.
Signed-off-by: librelad <librelad@digitalangels.vip>
Docker materialises a missing bind-mount source as an empty directory when a
container starts. The WebUI compose mounts ./libreportal.config as a file, so a
container start before the config landed left a directory at that path — and it
was self-perpetuating:
- copyFolder's tar extract aborted the whole source copy with
"libreportal/libreportal.config: Cannot open: File exists" (exit 2)
- dockerConfigSetupToContainer guards on [ ! -f ], which a directory fails, so
copyFile dropped the real config INSIDE the stub
- the closing -e / -r sanity checks both pass on a directory
The installer then reported success while libreportal-service crash-looped on
EISDIR reading /app/libreportal.config, leaving the WebUI unreachable.
Add repairStubDirForFile: promote a same-named file out of the stub, drop the
directory, and report if the path still isn't a regular file. Call it before the
WebUI source copy and before the per-app config copy (covers every app, not just
the WebUI), and tighten the closing existence check from -e to -f so a stub can
never pass validation again.
Signed-off-by: librelad <librelad@digitalangels.vip>
The per-app Updates tab pushed a manual "Check" button (header + empty state)
even though scans run automatically on CFG_UPDATER_SCAN_INTERVAL — so an app with
nothing to update read like an empty/actionable page. Replace the manual Check
with a calm status line inside the panel: "Checked automatically · last checked X
· next check ~Y", backup-schedule style. The genuine Apply/Roll back actions stay
(applying is still manual and safe). No auto-apply.
- webui_updater_scan.sh: stamp scan_interval_minutes alongside generated_at in
updates.json so the display needs no separate config fetch (0 = auto off).
- updater-page.js: renderAutoCheckLine() + fmtRelFuture().
- app-tabbed-manager.js: drop the header/empty-state Check buttons; render the
auto-check line; friendlier no-data copy.
- overview.css: style .updater-autocheck (green dot live / muted when off).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
For a generic app (no traefik files, not libreportal) fixPermissionsBeforeStart
prints nothing, so the "Updating file permissions before starting" step showed
its header followed by an empty line — looking like it did nothing. Add an
isSuccessful line after the call so the step reports like every other one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh install ran webuiLibrePortalUpdate twice back-to-back: once in
installLibrePortal step 11, then again in startScan at the end of preinstall.
The 30s time-debounce meant to collapse them is fragile (it never fired on a
recent install — >30s elapsed between the two), and debouncing is the wrong
lever anyway: startScan's pass runs AFTER scanConfigsForRandomPassword
finalises app passwords, so it — not the step-11 pass — is authoritative.
Defer the step-11 generation deterministically during a bootstrap install
(libreportal_bootstrap_install=true), leaving startScan's single pass to do the
work. Standalone reinstalls (no bootstrap flag) still generate in step 11. The
time-debounce stays as a general back-to-back backstop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the Update / Roll-back buttons tell the truth, closing the "new code on
old data" hole a floating tag creates.
updaterApplyApp:
- Anchor-correct: capture before/after from updaterPrimaryImage (the
<slug>-service image), not `grep -m1 image:` — fixes ollama et al.
- Records EXACT build refs in history from->to: repo:tag@sha256:<digest>
(via updaterRefDigest), so history is meaningful even when the tag doesn't
move (a rebuilt `latest`).
- Un-pins any digest a prior rollback pinned before pulling, so Update tracks
the channel again instead of freezing on the rolled-back build.
updaterRollbackApp:
- Before recreating, re-pins the anchor image to the pre-update build's digest
(from history's last update/ok `from`) via updaterSetAnchorRef, so `up` runs
the OLD code — not the current channel head. This is the fix for restoring a
data snapshot but recreating on a newer image.
New helpers (cli_updater_commands.sh): updaterRefDigest (local RepoDigest),
updaterSetAnchorRef (rewrite the anchor image line by service name, preserving
indent + the version sentinel; correct for companion-first apps like ollama),
updaterLastUpdateFrom (roll-back target from history).
Verified the helpers on nextcloud + ollama: pin adds @sha256 to the right
anchor only, sidecars untouched, sentinel preserved, unpin restores, YAML valid.
Caught and fixed a `local a=$1 b=...$a...` same-statement expansion bug that
would have silently no-op'd the rollback pin. End-to-end apply/revert not
exercised live here (no installed app has a pending update on this box).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The per-app object build used `available_version:($available_version|select(.!=""))`
to omit the field when empty. But in jq, a `{key: (empty)}` makes the WHOLE
object construction emit nothing — so every app with no available update (empty
available_version) produced no object and was silently dropped from updates.json.
Only apps WITH a pending update survived; an all-up-to-date fleet showed an empty
list. (Missed in P2's sandbox test because both fixture apps had updates.)
Emit an explicit null instead: `(if $available_version=="" then null else … end)`.
Verified: an up-to-date app (trivy, local==registry digest) now emits with
available_version:null; apps with updates still carry the string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
webuiUpdaterScan now tells the truth instead of hardcoding update_available=false.
Per app it resolves three facts (see roadmap §2): the running RepoDigest (one
`docker inspect` of the anchor image), the display version (OCI label →
versioned tag → channel·shortdigest), and — throttled — the registry's current
index digest for the channel tag (`docker buildx imagetools inspect`, the same
identity as RepoDigest, verified exact). update_available = the two digests
differ. Emits type (versioned|rolling), channel, current/available digests +
versions, and a services[] array (every image line, anchor flagged).
- Registry lookups throttled separately from the scan: CFG_UPDATER_REGISTRY_
INTERVAL (min, default 360; 0 = local-only), own /tmp stamp, reuse of the
prior available_digest between windows so the app list still refreshes every
scan. UPDATER_REGISTRY_FORCE=1 forces a live pull (the Check-now button).
Registry failure (offline/rate-limited) = "unknown", never a false "changed".
- Digest-compare fully detects rolling apps' new builds; for versioned apps it
catches rebuilds of the pinned tag (newer-version enumeration is a later
step) — honest per type, and versioned apps are user-picked via P1b anyway.
- Fixes a P1b regression: updaterPrimaryImage now strips the trailing version
sentinel comment (`s/ #.*//`) via _updaterCleanImageRef — without this the
anchor ref (and the live CVE scanner's image arg) carried the comment.
- JSON built with jq for safe escaping; jq-less fallback keeps output valid.
Verified via a simulated 2-app install: navidrome → "0.62.0" (OCI label) +
detected update; rolling app → channel·shortdigest + update; throttled re-run
reuses the prior digest; all output valid JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
"Control plane" is jargon. Rename the self-heal task and de-jargon all the
user-facing copy: task titles ("LibrePortal - Fix System Issues" / "System
Health Check"), the action label, the badge/banner/panel text and "Fix now"
button, the status summaries, and the heal task-log messages. Behaviour
unchanged; code comments keep the technical term where accurate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The scan read the app's "current image" via `grep -m1 image:` — the first
image line. For apps that declare a companion service first this mislabels the
app: ollama lists `ollama-webui` (open-webui) before `ollama-service`
(ollama/ollama), so an app named ollama reported open-webui's version.
Add `updaterPrimaryImage`: resolve the version anchor from the app's
`<slug>-service` image (the universal primary-service naming convention,
33/33 apps; underscores→hyphens for slugs like libreportal_catalog). Falls back
to the first image line off-convention. Used by both the version and CVE loops.
Verified: only ollama changes (→ ollama/ollama:latest); nextcloud, mastodon,
jitsi, gitea, vaultwarden anchors unchanged.
Also revise docs/roadmap/app-version-updater-and-cve.md to the config-first
direction agreed this session:
- CFG_<APP>_VERSION ADOPTED (was rejected): the #LIBREPORTAL tagging system
makes the config the source (compose tag derived from it), not a second one.
- Two version TYPES — versioned (real tag, version picker) vs rolling (floating
channel, digest-freshness); digest is the uniform detection engine for both,
version numbers are display enrichment. Answers "why not just compare numbers"
(most upstreams publish none; no universal latest-version API).
- Multi-service anchor = <slug>-service, not first line; lock-step sets (jitsi)
= one channel → several image lines; sidecars tracked-by-digest, not headline.
- Phases updated: P0 anchor (done) → P1 config-first pin → P2 detection+display.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Crash-loop detection keyed on `docker ps --filter status=restarting`, but a
backed-off loop sits "exited" between restarts once docker's backoff grows to
tens of seconds — so a slowed loop is missed. Detect via RestartCount CLIMBING
between scans (what a crash loop actually is), unioned with the instantaneous
restarting signal for fast loops. Baseline counts persist in
.health_restart_counts, written only by the throttled check so the heal's
re-scans don't disturb the delta.
Also give the system_health_heal / system_network_heal tasks proper display
(they fell through to the raw command + generic ⚙️): friendly titles in
formatCommandForUser, type icons (🩺 / 🌐) in getTaskTypeIcon, and action
labels in formatActionTitle — so they read as "LibrePortal - Repair Control
Plane" etc. with an icon, like install tasks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
An offline trivy install crash-looped (server FATALs when it can't fetch the
vuln DB), and on rootless docker the restart storm churned the shared network's
port-forwarder until the WebUI's own published host port was torn down — the
WebUI stayed healthy INSIDE its container but was unreachable from the host, with
nothing detecting or healing it.
Three fixes, in the house self-healing style (mirrors the network-drift trio):
1. Control-plane health checker wired into the existing task-processor idle poll
(maybeRegenPoll), no new daemon. dockerHealthScan (read-only) detects daemon
down, a WebUI running-but-host-port-unreachable (the port-forward corruption),
and crash-looping containers. webuiSystemHealthCheck writes
frontend/data/system/health_status.json + self-dispatches a heal — the user
can't click a button on a dead WebUI, so the poll drives the fix. Frontend
health-notifier surfaces a topbar badge + dashboard banner + details panel.
2. Failure cap, enforced centrally by dockerHealthHeal (task-gated): stops
crash-loopers (removing the churn), restarts the WebUI to re-publish a lost
port forward, and — only if that fails — recycles the rootless daemon and
restarts the core container. Caps every app immediately, no template churn.
3. Trivy no longer crash-loops offline: the server runs in a shell retry-loop so
the container stays Up and quietly retries on a backoff instead of exiting
FATAL. Verified: container stays Up across repeated DB-download failures.
Core WebUI compose gains restart: unless-stopped so it self-recovers after a
reboot / daemon recycle instead of staying down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Follow-up to the backup-refresh throttle/dedupe, cutting the cost of the
remote pulls that do still happen.
* restic stats now runs in --mode raw-data (restic_check.sh). The default
restore-size mode walks every snapshot's tree to sum logical file sizes —
the slowest restic op — just to fill a size readout. raw-data reads the
index only and reports the repository's actual deduplicated on-disk size,
which is exactly what the dashboard already labels "deduplicated,
encrypted". raw-data omits total_file_count, so the per-location card now
shows that location's snapshot count (already loaded client-side, and more
useful for a backup repo) instead of a file count.
* engineLocationStats now shares the same per-refresh memoiser as
engineSnapshotsJson (engine_dispatch.sh). Both the locations and dashboard
generators call it per location, so repo stats went from two restic calls
per location per refresh to one. Factored the cache into _engineCachedPull.
* SSH connection reuse for SFTP locations (backup_ssh.sh): ControlMaster=auto
with a self-reaping ControlPersist master, so the several restic
subprocesses a refresh/backup spawns against one location share a single
authenticated connection instead of a fresh handshake each — the dominant
per-call cost on a high-latency link. Toggle via CFG_BACKUP_SSH_MULTIPLEX.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The "Refreshing backup data..." step on every WebUI update fired one
`restic stats` (restore-size mode — the slowest restic op) plus ~4
identical unfiltered `restic snapshots --json` pulls per enabled
location (dashboard, snapshots, app-status, migrate each pulled their
own), all over a fresh SSH connection for remote repos, on every pass
with no throttle — the slow, "frozen"-looking line users hit on poor
links.
Two fixes:
1. Dedupe. engineSnapshotsJson transparently memoises the first
unfiltered whole-repo pull per location to a shared cache dir
(LP_SNAP_CACHE_DIR, set by webui_updater around the chain), so the
four generators reuse one restic call instead of four. Filtered and
failed/empty pulls always fall through to a live call.
2. Throttle. Backups and location changes already regenerate this data
live when they happen, so the routine pass is only a drift catch-up.
Split the cheap local-only generators (engines/schema/passwords —
no remote I/O) out to always run, and gate the remote pull behind
CFG_BACKUP_DASHBOARD_REFRESH_INTERVAL (minutes, default 30, 0 =
every update). A completed backup touches a dirty marker that forces
the next pass to pull; WEBUI_UPDATER_FORCE still forces a full
refresh.
Net: N locations x 5 remote restic calls every update -> 1 call per
location, only when something actually changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Trivy runs as a server whose vulnerability DB downloads on first boot; until it
lands no scan can produce results. Previously the updater generator wrote an
empty-but-valid cves.json the moment the file was missing, so installing Trivy
painted a green "no known vulnerabilities" all-clear that was actually a lie —
the DB hadn't even downloaded, and the Updates/Security view gave no signal.
Add an honest scanner state the WebUI branches on:
- containers/trivy/scripts/trivy_scan.sh — trivyScannerState (absent |
db_updating | ready) via `trivy version -f json`, trivyDbUpdatedAt, and
trivyScanImageCves (per-image scan normalized to {id,severity,package,
installed,fixed_in,url}, deduped). All degrade safely on error.
- webui_updater_scan.sh — stamp cves.json with scanner.state; only run real
per-image scans once the DB is ready. Always rewritten so state tracks live.
- updater-page.js — Security tab shows a loading box while the DB updates, an
install nudge when absent, and the genuine 🎉 only when ready+empty; Overview
CVE card sub + hint reflect the state.
- overview-manager.js — fleet Security row surfaces the "building CVE database"
pending state instead of silently omitting.
- function_manifest.sh — regenerated for the new trivy_scan.sh functions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New containers/trivy app (aquasec/trivy in server mode, private API port,
docker socket + cache volume) in the security,recommended categories, plus
placement in the setup wizard recommended step, the server-side install
tier after crowdsec, and the CLI first-install prompt. The updater's CVE
scan (design doc P4) will gate on this app being installed; §5 of the
design doc updated to record the app-based decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
After 'Generated apps-tools.json...' the next step reaches remote backup
locations (restic over SSH) with its output captured in $(...), so on a
slow connection the updater looked stalled with no message. Print an
in-progress notice before it, mirroring the per-app refresh hook loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Config sections (Basic, Mail, DNS, Firewall, …) rendered as a bare name. Add a
small data-driven icon before each: config files now carry a `# @icon <emoji>`
directive in their header, the config generator surfaces it as the subcategory's
`icon`, and the config editor prepends it to the section <h3> (covers every
render path — regular/master/toggle/domains/catalogs/whitelist — since they all
drop displaySubcategory straight into the heading).
- webui_generate_configs.sh: parse `# @icon` (first 10 lines) → emit "icon".
- config-manager.js: prepend subcategoryData.icon to the section title.
- configs/**: added an @icon to all 23 base config files (backup/general/network/
security/webui). The live install's config files get the same directive + a
configs.json regen so existing installs show the icons too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- Catalog block now shows the official catalog as a pinned row #1 (verified
badge + its URL from the generated data). Removing it opens the shared
confirmation dialog with a REQUIRED tickbox (generalized to accept custom
checkbox text) — it only hides official apps from browsing; system updates are
unaffected. Backed by a new CFG_CATALOG_OFFICIAL_ENABLED flag the scan honors.
- Trimmed the info-banner wording per feedback.
- Config option sections: the body (after the divider, before the buttons) now
sits in a darker boxed panel (bg + border), matching the content-box look used
elsewhere. Applies to regular grids + the block editors; scoped to the main
config editor.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the intended model: catalog sources should live in the config place (like
domains), and registry_catalog.json should be a purely GENERATED artifact
derived from them — not the source of truth. Replaces the earlier
$docker_dir/catalog/sources.json store.
- New configs/general/general_catalogs — CFG_CATALOG_1..9, one catalog base URL
per slot ("url" or "url|channel"), domains-style. Official stays pinned as
source #1 (derived from CFG_RELEASE_BASE_URL, not listed here). Slot N → source
idx N+1 (stable id for the Add picker / `app add --source`).
- catalog_sources.sh now reads/writes those CFG vars (via updateConfigOption)
instead of a JSON file; dropped catalogSourcesFile + the enable/disable toggle
(presence = enabled; remove = clear the slot).
- configUpdateBatch regenerates registry_catalog.json when a CFG_CATALOG_* key
changed — so pressing Save in the WebUI rebuilds the browse data.
- webuiRegistryCatalogScan is unchanged (still iterates catalogEnabledSources).
Verified: CFG_CATALOG_1/2 → sources at idx 2/3, empty slots skipped, url|channel
parsed, official pinned at idx 1.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
startPreInstall called installLibrePortalImageWebUI standalone AND then
installLibrePortalAppWebUI, whose app-install step 1 builds the same image
again — so a fresh install built libreportal-service twice back-to-back
(2nd mostly Docker-cache hits, but a redundant copyFolder + "this may take a
while" block). The standalone call only matters on a rebuild where the
app-install path SKIPS (libreportal already in the DB). Gate it on exactly
that: build standalone only when libreportal is already installed. Net: one
build in both cases (fresh → via app install; reinstall → standalone).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the "task loops eternally" bug: the lazy-autoload stub was
fn() { source "$file"; fn "$@"; }
If the source fails — the real case: an app-install/deploy rsync briefly
removes-then-replaces a generator file while a setup task calls it — `fn` is
never redefined, so `fn "$@"` re-invokes the *stub*, which sources the (still
missing) file, which re-invokes the stub… A single setup finalize recursed
12,050 levels, flooding the task log and taking 82s before it happened to
recover when the file reappeared. A permanently-missing file would never
recover.
Fix (root cause): drop the stub before sourcing —
fn() { unset -f fn; source "$file"; fn "$@"; }
so a failed source degrades to one "command not found" (rc 127) instead of
unbounded recursion. Regenerated function_manifest.sh (975 stubs, reformat
only — no function-set change).
Failsafes on the task processor (defence in depth, per request):
- FUNCNEST cap (TASK_FUNCNEST_MAX, default 1000) inside the task's eval
subshell — any runaway recursion now aborts in milliseconds instead of
spamming the log until the stack/disk gives out.
- Wall-clock cap (TASK_MAX_RUNTIME_SECS, default 7200s, 0=off) — the heartbeat
watcher TERM→KILLs a task's process group once exceeded and marks it failed
(distinct from a user cancel via a .timeout marker). Generous so real long
installs/backups/migrations finish.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Per feedback — shorten the install-failure error (drop the extra hint line) and
the rootless MTU notice.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An image-pull failure (e.g. the MTU-EOF black hole) left apps marked
installed+active with no container: the compose `up -d` failure was invisible
because (a) checkSuccess read $? after `_rc=$?`, always printing "✓ Started",
and (b) the exit code is dropped across dockerComposeUpdateAndStartApp →
dockerComposeUpdate → dockerComposeRestartAfterUpdate, and installApp never
checked it — so post-install integrations ran and set status=1 regardless.
- up_app.sh: restore $? to the compose exit before checkSuccess (both rootless
and rooted) so a failed `up -d` is reported as an error + logged, not "✓".
- app_install.sh: after `up`, reality-gate on the app's compose project having
at least one container (ps -a, so a slow-to-start container still counts).
If none exists, print a clear failure, skip _appPostStartIntegrations (which
is what records the app + sets status=1), and return non-zero.
Verified: the gate query passes for a running app (libreportal) and refuses
apps with no container (navidrome/ipinfo after their pulls EOF'd).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CFG_NETWORK_MTU=1500 was baked blindly into the rootless daemon uplink
(DOCKERD_ROOTLESS_ROOTLESSKIT_MTU) and every app's container network. On links
whose real path MTU is below 1500 (Qubes/NAT/VPN with PMTU discovery blocked),
image manifests + tiny images pull fine but large image LAYERS stall and reset
mid-blob with "httpReadSeeker: ... EOF" — apps silently fail to install. Probed
here: path MTU ~1328, Docker at 1500 → black hole.
- New scripts/network/network_mtu.sh: networkDetectMtu (don't-fragment ICMP
ladder → largest standard MTU that gets through, 1500 when ICMP gives no
signal) + networkEffectiveMtu (CFG_NETWORK_MTU: a number is verbatim, "auto"
probes once and caches $docker_dir/.network_mtu) + networkRedetectMtu.
- CFG_NETWORK_MTU default 1500 -> auto. Rootless setup now writes the resolved
MTU into the override (and re-detects per install); the per-app NETWORK_MTU_TAG
uses the resolved value too. Explicit numbers still win.
Verified on this box: auto -> 1300; alpine (multi-MB) + the full ~100MB navidrome
image now pull to completion where they previously EOF'd. Applied live (override
1300, config=auto, cache=1300, rootless docker restarted).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every file under $configs_dir is treated as a config (some get sourced at
startup), so a JSON file there was executed as bash — "[name:…]: command not
found" on every config load. Move it beside the apps DB in the manager state
root ($docker_dir/catalog/sources.json): persistent, manager-owned, and never
config-scanned.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First slice of multi-catalog ("taps") support. Today the App Center browses one
catalog (get.libreportal.org). This adds an ordered list of catalog sources and
teaches the browse scan to merge them.
- New scripts/catalog/catalog_sources.sh — the source list in its OWN file
($configs_dir/catalog/sources.json). Source #1 is ALWAYS the official catalog,
synthesized live from CFG_RELEASE_BASE_URL/CHANNEL (can't be edited/removed,
always pinned on top). Extra sources are stored as a small JSON array and are
UNVERIFIED (trust=community). Helpers: catalogSourcesJson / catalogEnabledSources
(priority order) / catalogSourceAdd|Remove|Toggle|List / catalogFetchCommunityIndex.
- webui_registry_scan.sh now walks catalogEnabledSources: the OFFICIAL source is
still signature-verified (lpFetchIndexInto, unchanged trust path); third-party
sources are fetched unverified. Apps are merged by slug into one card carrying a
sources[] array in priority order (highest first = default). Trust/verified are
taken from the SOURCE, never the artifact's self-claim, so a community index
can't promote itself to "official". Icons still mirror same-origin from the
official index only. registry_catalog.json gains top-level sources[] + per-app
sources[]; the old source{} object + signed/serial are kept for back-compat.
- New `libreportal catalog source list|add|remove|enable|disable` + `catalog
refresh` CLI (dynamic-routed). Mutations go through the task system (cliTaskRun
"…" "catalog"), never a new mutating API.
Scope firewall: this governs APP BROWSE + ADD only. LibrePortal's own updates and
hotfixes still resolve from the official CFG_RELEASE_BASE_URL alone — lpFetchIndex
is untouched, so a third-party catalog can never become a system-update channel.
Next: `app add --source`, then the WebUI (domains-style block source manager +
Add-dialog source picker with the Official badge / unverified warning).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three coupled defects in the first-run wizard flow, all surfacing as
"it said complete but nothing was set up":
1. Zero-app installs sailed through. With no apps ticked, setup was just
config+finalize, finished in seconds having installed nothing, and
fast-forwarded to an empty App Center. Add a self-contained in-wizard
confirm ("Install with no apps?") before submitting. Can't reuse the
shared confirmation-dialog component — it isn't loaded this early in
boot — so the dialog is rendered by the wizard itself and mounted on
<body> to escape the aurora surface's FX-stacking rule.
2. finalize declared success unconditionally. It never inspected the
per-app install tasks, so a failed app still yielded "your install is
ready" + a redirect. Pass the setup group id to `setup finalize`; it
now rolls up the group's app-install task results and logs a clear
partial/failed verdict. The WebUI completion watcher gates the welcome
toast + App Center hand-off on the whole group succeeding, not just on
finalize completing — a failed app now keeps the user on the tasks page
with an error toast instead of a false all-clear.
3. Setup task count was confusing (banner "of 2" while the page listed 3).
On dev/git installs the topbar's dev-mode auto-enable raced the wizard's
own CFG_DEV_MODE write and spawned a second, un-grouped config-update
task mid-setup. Skip that auto-enable while a setup handoff is active
(the wizard already persists CFG_DEV_MODE); it runs on the next load if
still needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The backup-data refresh in the WebUI update chain shelled out to
`restic snapshots` once PER APP (dashboard, app-status) and once per
(host, app) (migrate) — every call a `sudo`+`restic` spawn that re-opens
and decrypts the repo. With N apps and L locations that's O(N·L) (app-status)
plus O(hosts·apps·L) (migrate) spawns on every update — the silent stall
after "Refreshed backup dashboard data...".
One `restic snapshots --json` already returns the whole repo, so fetch it
ONCE per location and derive the per-app / per-host / system views in jq.
Validated against a live restic 0.19 repo: the jq filter reproduces
`restic snapshots --tag <t> --host <h>` exactly (incl. empty-host = all
hosts, and newest = last), and count/latest extraction matches the old grep.
- dashboard: one primary-location fetch → all app tiles + system status
(was N snapshot calls + 1).
- app-status: one fetch per location → filter per app (was N×L calls);
also de-recursed into a flat app loop.
- migrate: filter the already-fetched per-location list (was a spawn per
host×app). Fixes a latent bug too — `grep -oc` over single-line JSON
always reported a count of 1; jq `length` is now accurate.
restic snapshot spawns per refresh drop from ~O(N·L + hosts·apps·L) to ~O(L).
`restic stats` (still run per location in locations + dashboard) is a
separate, non-multiplicative cost left untouched to preserve its semantics.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
webuiLibrePortalUpdate runs the per-app refresh hooks on every update (the
task processor fires it repeatedly), and gluetun's hook silently pulls a
~7MB servers.json from GitHub each time. The hook's stdout is captured by
`result=$($_hook)`, so on a slow link the update just sits with no output
right after "Generated apps-tools.json..." — it reads as a freeze.
- Throttle the fetch to once per CFG_GLUETUN_PROVIDERS_REFRESH_HOURS (24h
default; 0 = manual-only). Direct callers (Tools refresh, install hook)
pass GLUETUN_PROVIDERS_FORCE=1 to still get a guaranteed pull.
- curl --compressed (this JSON gzips ~7x) + --connect-timeout 15 so a slow
or dead link shrinks/fails fast instead of stalling every update.
- Print "Refreshing <app> WebUI data..." from the updater loop (outside the
output capture) so the step is visible instead of looking hung.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
'Marketplace' read as a place to buy things; this is a free, self-hosted app
catalog, so rename the app to LibrePortal Catalog. Slug is libreportal_catalog
(underscore — the slug becomes a CFG_<SLUG>_ prefix and a bash identifier via
declare "${app_name}=i"; a dash would break install). Docker-facing names use
dashes (libreportal-catalog-service / hostname libreportal-catalog), declared
explicitly in the PORT config + compose, not derived from the slug.
- containers/marketplace/ -> containers/libreportal_catalog/ (+ .config, .svg, hook file)
- CFG_MARKETPLACE_* -> CFG_LIBREPORTAL_CATALOG_*, APP_NAME + TITLE + PORT_1 updated
- install hook fn marketplace_install_post_setup -> libreportal_catalog_install_post_setup
- served browse site reworded Marketplace -> Catalog, icon refs updated
- regenerated function_manifest.sh (autoload stub now points at the new file)
Not installed yet, so there is no live config/container/volume to migrate — the
cheapest moment to rename. The client-side registry ('View full page on the
marketplace') wording is a separate subsystem and left unchanged for now.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
start_scan.sh runs updateDockerInstallPassword every system scan, doing
`sudo passwd $CFG_DOCKER_INSTALL_USER` via runSystem. Model A's scoped
sudoers grants only LP_HELPERS/LP_SYSTEM + run-as-install-user — not passwd
— so at runtime (manager, non-root) it fails exit 1 every scan. This is the
exact sibling of the updateDockerSudoPassword failure fixed in 9050a8c; that
guard was added to the manager/sudo user but the dockerinstall user was
missed, so error_report.log kept logging "Updating the password for the
dockerinstall user" on every scan.
The password is set at install (root path, startPreInstall →
installDockerRootlessUser) and the rootless docker user is driven by tooling,
not a password login, so the runtime re-sync is legacy + impossible under
de-sudo. Guard it to skip unless EUID 0, mirroring the sudo-pass fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The featured passthrough was removed when the dedicated-section commit was
reverted; the Featured sidebar row needs it. spec.json {"featured":true} now
sets meta.featured on the published app again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replaces the extra on-card marketplace tags (the separate Official trust
badge) with a richer, more elegant flow: a registry card keeps a single
'Available' pill, and clicking Add (or the card) opens a modal outlining the
app — full description, publisher + trust, version, the add command, and a
'View full page on the marketplace ↗' link to the app's page on the source
it came from — with the actual add as the confirm. Gives a beat to review a
community app before pulling it in.
- webui_registry_scan.sh re-emits source{base,channel}; loadApps stashes it
(window.registryCatalogSource) + carries per-app version.
- openRegistryDetails() builds the eo-modal; addRegistryApp() is now just the
task dispatch (the modal's confirm / a fallback).
- The marketplace website supports a #<slug> deep-link (focus one app, with a
'Show all' bar), so the modal's link lands on that app's page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Reverts a4e65df. The maintainer prefers the more fluid model where registry
apps flow into the App Center grid as 'Available — Add' cards alongside
installed/local apps, rather than a separate /apps/marketplace destination.
Restores the Stage-4 merged-grid behaviour (loadApps merges
registry_catalog.json; createAppCard renders registry cards; addRegistryApp
dispatches app_add). The make_app.sh featured passthrough + the generator's
featured/source fields go with it (unused by the merged grid).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Splits the catalog out of the main grid into its own destination, the
WordPress 'Add Plugins' vs 'Installed Plugins' model: the grid is now
purely 'your apps' (local definitions), and the new Marketplace section is
'get more apps' (the remote signed catalog).
- New MarketplacePage (components/apps/marketplace/) mounts at
/apps/marketplace inside the apps feature (same sub-dispatch pattern as
/apps/overview — no new top-level component). Pinned sidebar entry with a
live 'available to add' count badge.
- Status strip: signed/unsigned, available + catalog counts, serial, source
base+channel, freshness, and a Refresh that re-runs the host-side registry
scan via updater_check.
- Publisher-curated Featured shelf (meta.featured, set at publish time — no
tracking/popularity), category chips + search, per-app detail modal
(long description, publisher/trust/version, add command), and the chained
Add & set-up flow: dispatch app_add, and when the definition lands, hand
off to the app's config/install page.
- State-aware cards: Available (Add) / Added (Set up →) / Installed (Open).
- Backend: make_app.sh passes through meta.featured; webui_registry_scan.sh
emits featured + source{base,channel} in registry_catalog.json.
- Removed the grid's registry-merge + registry card path + its CSS (moved to
the namespaced marketplace surface); app_add task wiring + completion
handler retained and reused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
containers/marketplace — nginx:alpine app (standard drop-in contract:
config + tagged compose + icon + install hook) whose docroot serves BOTH
halves of the marketplace: the signed catalog channel tree (index.json /
payloads, published into data/<channel>/ by the release tools) and a
self-contained client-rendered browse site over the same file (search,
category chips, trust badges, copyable 'libreportal app add <slug>' —
no third-party assets, no backend, no build step). The official
marketplace is an instance of this app; self-hosting one = installing it
and pointing CFG_RELEASE_BASE_URL at it. Boxes only ever trust the
minisign signature on the catalog, never the website.
New generic gating convention: CFG_<APP>_DEV_ONLY=true keeps an app out
of the App Center grid unless Developer Mode is on (CFG_DEV_MODE, the
same flag the **DEV** config-field filter uses); an installed dev-only
app always stays visible. The marketplace app is the first user.
Cache policy: catalog/channel manifests no-cache; payloads short
revalidating cache (same-id re-publish); version-pinned release
artifacts immutable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
webuiRegistryCatalogScan (run by updater check, same atomic keep-prior
pattern as webuiArtifactScan) writes apps/generated/registry_catalog.json:
the type:"app"/kind:"bundle" rows of the signed index annotated with
defined/installed, browse metadata from the envelope meta, and icons
mirrored into core/icons/apps/registry/ ONLY when their bytes match the
sha256 pin in the signed index — the browser stays same-origin; a tampered
or oversized icon is skipped, never served.
webuiArtifactScan now selects type=="hotfix" so app rows never render as
pseudo-hotfixes in the Improvements tab, and counts+logs artifacts of
unrecognized type instead of surfacing them (the §8.1 forward-compat
firewall on the scan path).
Harness vs a locally served registry: 14/14 (catalog row + meta + flags,
icon pin verify + tamper skip, hotfix-only stream, unknown-type skip+log,
unreachable-registry keeps prior files).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Opens the two designed seams (roadmap §8.4): _artifactResolve accepts
type:"app" (slug validated, the installed-app gate skipped — presence is
the collision policy's call), and payload.kind:"bundle" gets its own APPLY
flow. The download core (sha256 pin vs the signed index + minisig +
refuse-unsigned) is factored into _artifactDownloadVerified, shared by ops
and bundle payloads.
A bundle add: fetch → quarantine-validate → place in the definition tree
(staging + one rename, manager funnel) → lpRegenWebui → verify the app
surfaced in apps.json → applied-record with a precise undo → History.
The validator is fail-closed (traversal/absolute paths, links/devices,
single top-level dir == slug, charset, size/entry caps, set-id strip,
config TITLE+CATEGORY + compose present, bash -n every .sh) because the
definition tree is live-sourced on every CLI start — nothing lands there
before trust + quarantine pass. Collision policy: installed-live refused,
local definitions win, registry-owned re-add = reversible definition
update (prior tree packed into the undo). Revert removes/restores the
definition (refused while installed) and regens. Apps never auto-apply
(type filter kept + publisher forces auto:false).
New verb: libreportal app add <slug|artifact-id> (app_add task; resolves
by slug via appAddFromRegistry, ambiguity refused).
Also fixes the second half of the sigstate-propagation bug class:
artifactApply captured $(_artifactResolve) in a subshell, stranding
_ART_INDEX/_ART_APP/_ART_SCOPE AND the LP_INDEX_SIGSTATE the apply gate
enforces — on a signed box every apply would have refused as unsigned.
Resolve now assigns globals (_ART_JSON) and is called directly.
Source-and-mock harness: 46/46 (resolve gates, 14 validator refusals,
happy add, collision matrix, definition-update round-trip, revert
semantics, postcheck + record-failure rollbacks, apply-auto exclusion,
app add verb).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The marketplace publisher tool: packs containers/<slug>/ (the drop-in app
contract, unchanged) into a deterministic tarball (commit-clamped mtimes +
gzip -n; unchanged apps repack byte-identical and keep their published
version), signs it, copies a sha256-pinned catalog icon, and upserts a
type:"app" / payload.kind:"bundle" envelope into the same team-signed
index hotfixes ride. Catalog metadata (title/category/descriptions) is
parsed line-wise from the app's own .config — one source of truth with the
App Center generators. auto:false is hard-forced: apps never auto-apply.
The index-upsert/serial/freshness/publishers-map and signing logic is
factored out of make_hotfix.sh into lib/release_index.sh, shared by both
tools (make_hotfix.sh behavior preserved; regression-tested alongside an
app entry: serial bump, both payload kinds coexist, valid_until refreshed).
LP_INDEX_VALID_DAYS is the shared freshness knob (LP_HOTFIX_VALID_DAYS
kept as a legacy alias).
Verified: speedtest publish → deterministic repack (identical sha256) →
served via local http.server → real lpFetchIndex/accessors harness 10/10.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Every caller captured the index with var=$(lpFetchIndex), which runs the
fetch in a command-substitution subshell — the LP_INDEX_SIGSTATE global it
sets never reached the caller. On a box with real signing active the
artifactApply/apply-auto gates would therefore refuse a correctly signed
index (fail-closed, but the apply path would be dead on arrival the day
signing activates), and artifact index / the WebUI scan would report a
verified feed as UNSIGNED.
New lpFetchIndexInto <var> [cache] runs the fetch in the calling shell and
assigns via printf -v; all four call sites converted. Verified with a
source-and-mock harness against a locally served index: 10/10 (sigstate
reaches caller, serial high-water, anti-rollback refuse, staleness refuse,
id enumeration, envelope round-trip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Describe only the current useradd behaviour; drop the narration of the old
silent-failure bug (per the repo's no-tombstone-comments convention).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Drop the GitHub-release version comparison entirely. We install slirp4netns
from apt regardless, so comparing against the GitHub-latest tag only produced
a perpetual 'outdated' loop and a no-op re-install. apt-get install -y is
already idempotent, so run it unconditionally and report the resulting
version.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
slirp4netns --version prints multiple lines (version, commit, libslirp,
SLIRP_CONFIG_VERSION_MAX). The old 'awk {print $2}' ran on every line and
also picked the literal word 'version' from line 1, producing a multi-line
blob that leaked into the 'is outdated' notice. Read only the first line and
take field 3 (the actual number), strip the leading v from the GitHub tag so
the comparison is meaningful, and skip the check if the tag fetch fails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Initial rootless setup ran 'systemctl --user start docker' immediately
after install, but the rootless net/port-driver override.conf (and the
daemon-reload that loads it) aren't written until further down. So the
first start always failed — 'Job for docker.service failed' plus a
spurious '✗ Error Setting up Rootless' in the error report — even though
the later 'systemctl --user restart docker' brought the daemon up fine
once the override was in place.
Drop the premature start from the install step (keep install + enable);
the restart after the override is written is now the first real start.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
