The live Nextcloud 31→34 climb left 4.4 GB of images behind — one per
rung, each ~1.5 GB, all still present after it finished. On a small VPS
that is the difference between working and full.
`system reclaim` cannot help: it collects DANGLING images, and every rung
is a real tag, so all of them stay tagged and stay on disk. (Rolling apps
never hit this — moving a floating tag orphans the old image, which
reclaim then collects. It is specific to laddering.)
After a SUCCESSFUL climb only, remove the images stepped through, keeping
the immediately-previous version so a roll-back needs no download.
CFG_UPDATER_UPGRADE_PRUNE=false keeps everything. Never runs on failure,
where the older images are exactly what recovery may need.
Tested: a 3-rung climb removes 31 and 32 and keeps 33; a single-step
climb removes nothing (its previous version IS the rollback target); the
config switch disables it.
Found by looking at the box after the first real ladder run — the feature
worked, and then quietly cost 4.4 GB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`updater upgrade <app> --detach` parsed "--detach" as the target version
and refused with "no safe path from 31-fpm-alpine to --detach". It failed
safe, but blaming the version for a misplaced flag is a poor way to say
the flag is not supported here. Unknown options now say so.
Found during the first live ladder run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by actually installing it. The compose bind-mounts
./resources/nginx.conf into the web container, but nothing ever copied
that file into the container tree, so Docker created a DIRECTORY in its
place and nginx died with:
error mounting ".../resources/nginx.conf" to rootfs at
"/etc/nginx/nginx.conf": not a directory
Worse than a hard failure: the app still recorded as installed. Three of
four containers came up, the DB and the app itself were fine, and only
the web front end was missing — a quiet, partial install.
Apps needing a resource file declare the copy in a hook (authelia does
exactly this); Nextcloud simply never had one. Adds
nextcloud_install_post_compose — after the compose file is written,
before permissions and `up` — which repairs any stub directory left by a
previous attempt and then copies the file.
The stub repair matters: without it the copy lands INSIDE the directory
(resources/nginx.conf/nginx.conf) and the mount fails identically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ties the ladder and the verifiers together behind a new verb:
libreportal updater upgrade <app> [version] [--dry-run]
Per rung, and every part is load-bearing:
snapshot (fail-closed) -> set version -> pull -> up -> VERIFY -> next
On failure anywhere: restore THIS rung's snapshot, put the version back,
stop, and leave the app on the last version it actually verified at. The
ladder never continues past a doubt.
A snapshot PER RUNG rather than one at the start, because upstream
migrations are usually one-way — Nextcloud 32's schema cannot be undone
by putting the 31 image back. The recovery guarantee is "restore the
snapshot from sixty seconds ago", which only holds if every rung has one.
Two gates before anything moves. An app with no <app>_upgrade_verify is
refused outright: the generic health check cannot see a half-finished
migration, so laddering on it would be a guess wearing a safety label.
And a ladder that cannot be computed end to end refuses rather than
attempting a partial climb.
`updater upgrade` is a separate verb from `apply` on purpose: apply moves
you WITHIN a release line (and may be automatic), upgrade moves you
BETWEEN lines and is always a deliberate act. Dry runs execute inline so
the plan is instant to read.
updaterSetAnchorVersion rewrites the image tag AND its version sentinel
together — updating only the image would leave the sentinel advertising
the old version, and the next config regeneration would silently revert
the app.
Tested with stubs against the real code paths: the no-verifier gate holds
and changes nothing; a dry run has zero side effects; the happy path
snapshots at each current version before moving; a verify failure on rung
2 of 3 stops with the app on rung 1, restored, and never touches rung 3;
a failed snapshot moves no version and pulls nothing; a container that
will not start is rolled back.
NOT yet exercised on a live install — no app here needs a ladder. The
first real run should be a dry run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Foundation for stepped upgrades. Answers one question only — WHICH
versions, in WHICH order — with no side effects, so it can be tested
exhaustively. Applying the rungs is a separate job.
Nextcloud refuses to skip a major ("Updates between multiple major
versions and downgrades are unsupported") and will not start; databases
behave the same way about their data directory. For those apps 31 -> 34
is three upgrades, each with a migration that must finish before the
next begins.
Built by PROBING each candidate rung, not by enumerating tags — because
enumeration is provably unsafe here. Docker Hub pages at 100 ordered by
recency, and the first real-registry run proved the danger: it produced
v4.2 -> v4.4 -> v4.5 -> v4.6 for mastodon, silently skipping v4.3, which
exists (HTTP 200) but had fallen off the newest-100 listing. Skipping a
rung is the precise failure this file exists to prevent, so the ladder is
now built by incrementing and probing: v4.2 -> v4.3 -> v4.4 -> v4.5 ->
v4.6, 4 steps.
Guarantees: same shape only (never 31-fpm-alpine onto 31-apache),
strictly ascending, never a downgrade, rolling tags refused outright, and
a version upstream never published is stepped over only because the probe
said so. If a continuous path to the target cannot be constructed it
returns 1 and prints nothing — refusing to guess, because a wrong ladder
means a skipped migration.
20 unit tests, including the exact listing-truncation case above and the
numeric ordering that would otherwise drive an app backwards (0.9 vs
0.10). Real registry: nextcloud 3 steps, mastodon 4, stalwart current.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The digest compare only ever asks about the tag already pinned, so it
answers "has my tag been rebuilt?" and can never answer "does a newer
version exist?". An app on v0.16 reports up to date forever while 0.17
ships. That is the gap between an app that updates and an app that is
current, and it silently affects every pinned app.
Adds tag enumeration for VERSIONED tags only (rolling tags already move
on their own): list the repo's tags, keep those sharing the current tag's
SHAPE, and pick the numerically greatest.
Shape matching is the whole safety story — v0.16 -> v#.# so it can never
"upgrade" you onto v0.16-alpine, 31-fpm-alpine onto 31-apache, or a date
tag onto a semver one. Comparison is component-wise numeric, so 0.10 > 0.9
and 1.0 > 0.99 (a string sort gets both wrong), with 10# forcing base ten
so an upstream "08" cannot be read as octal. 15 unit tests cover it.
Docker Hub only, deliberately: all three pinned apps live there, it needs
no auth, and the generic OCI tags/list wants a per-registry token dance.
Other registries stay quiet rather than guess. Throttled inside the
existing registry window and cached between windows so it cannot flicker.
Surfaced as INFORMATION, never an action: no button applies it, because a
version move can carry a data migration. `update_available` and the "up
to date" badge keep their exact meaning; the new state sits beside them
and points at the Version field.
Against the live registry: stalwart v0.16 is current, nextcloud is on
31-fpm-alpine with 34-fpm-alpine out, mastodon on v4.2.0 with v4.6.5 out.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It sets the image tag, so a wrong value stops the app starting — that
belongs with the other expert settings, not beside feature toggles.
Tooltip now explains the split it participates in: automatic updates
apply rebuilds OF this version, changing it moves between releases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both found in a user's console log.
1. ReferenceError: setupMobileMenu is not defined (dashboard.js:98)
core/topbar/js/mobile-menu.js defines that global, and index.html
never loaded it. dashboard.js called it unguarded as the FIRST line
of setupEventListeners, so dashboard init threw every page load and
took loadInstalledApps() with it — and the burger menu was dead on
mobile. system-loader already guarded its own call with a typeof
check, which is why this survived unnoticed.
Loads the script (before dashboard.js) and guards the call, so
optional nav chrome can never take down the page below it again.
2. Endless 404s on /api/tasks/<id> for tasks that no longer exist
queue.json is append-only from the enqueue side and nothing ever
pruned it, so any task file removed afterwards left an id the WebUI
re-fetched forever, one 404 per poll per orphan. Adds
cleanupOrphanQueueEntries to the idle housekeeping pass: entries with
no task file are dropped and logged. Self-heals existing strays.
(Provoked by my own clean-up of two test tasks earlier in this
session, but the gap is real and predates it.)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One container providing SMTP/IMAP/POP3/JMAP plus CalDAV/CardDAV, an admin
UI and spam filtering — chosen over mailcow (owns its own installer, which
is what killed the earlier attempt now sitting in scripts/unused/) and
over Mailu (~7 containers) because a single image with a single data dir
is the only shape that fits the existing conventions cleanly: one anchor
service the updater can version, one path the backup engine can snapshot.
Mail-specific departures from the usual app template, each deliberate:
* Ports are FIXED, not random. Other mail servers connect to :25 by
number and clients expect 465/587/993 — a randomised external port
would silently make the server unreachable. Only the admin UI takes a
random port, since that one really is just a browser behind Traefik.
143/995/4190/443 ship disabled; the port processor comments them out.
* UPDATE_TYPE=manual and the image pinned to v0.16, not :latest.
Stalwart is pre-1.0 and has said the storage schema is still being
finalised, so an unattended minor bump could carry a data migration on
the message store. This is the one app where the auto default is wrong.
* BACKUP_STRATEGY=stop-snapshot-start. The message store is written
continuously; a live copy can land mid-transaction. Seconds of queued
delivery (senders retry) buys a consistent snapshot.
* The install hook checks outbound port 25 and reverse DNS, then prints
the MX/SPF/DMARC records with real values. A mail server whose
container started is not a working mail server, and every remaining
requirement lives at the registrar or the VPS provider.
Admin credentials are seeded via STALWART_RECOVERY_ADMIN from the app
config rather than left to Stalwart's first-run random password, which
would otherwise exist only in the container log.
Icon is a drawn placeholder, not the upstream trademark.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The portal was found crash-looping with EACCES on /app/webui_logins.
configs/webui/* are bind-mounted into the container, which reads them
through its GROUP (files 0640, group = container owner). Two paths reset
that group to the manager and never gave it back:
* init.sh setupConfigsFromRepo — `chown -R manager:manager` over the
whole configs tree on every install/redeploy (the documented local-
mode deploy), and
* the runtime config reconcile — rewriting a live config replaces the
file as the manager, so ANY release that merely adds a key to a
webui_* config would break the portal.
Neither breaks anything immediately: the running container holds its
open files, so the failure only appears at the next restart, long after
the change that caused it. That is exactly how it surfaced here — a
deploy in the evening, a dead WebUI later.
init.sh gains restoreWebuiBindAccess (prefers the root ownership helper,
inline chown fallback for the first install, no-op when the container
user does not exist yet) called right after the chown; the reconcile
calls the existing reconcileWebuiDirOwnership when it is in scope.
Verified by reproducing the break (chown -R manager over configs), then
running the fixed deploy and force-recreating the container: group is
restored to the container owner and the portal serves on 3179.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The missing piece of hands-off updates/backups: when a task fails while
nobody has the WebUI open, LibrePortal now says so — email (via the
existing Mail settings), ntfy, Gotify, Discord, Slack, Telegram, or
Pushover, configured under Settings → Notifications.
One hook, everywhere: the task processor reports every terminal task to
`libreportal notify task <id>` (detached, never load-bearing — hard curl
timeouts, failures ignored). The POLICY lives in the notify command, not
the daemon: CFG_NOTIFY_EVENTS = failures (default) | all | off, and
cancelled tasks never notify. Failure copy is task-aware — a failed
update says the app was already rolled back and won't be retried, so the
reader knows the box is safe before opening the WebUI.
`libreportal notify test` sends to every enabled channel with per-channel
results. Verified against a local mock endpoint: all webhook payloads,
JSON escaping (quotes/newlines), the events policy, and fail-fast on
dead endpoints (8ms, exit nonzero).
The v0.1.0 per-app NOTIFY_* field-mapping scaffolding (never wired to a
sender) stays as-is; this global channel is the system it was waiting on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes that make the auto-updater a trustworthy background system:
* CFG_UPDATER_WINDOW (default 06:00-08:00 host time, right after the
05:00 backup cron; HH:MM-HH:MM wraps midnight, 'always' = any time).
Gates only the enqueue — scans keep running all day, so the Updates
page stays current and pending updates visibly wait for the window.
Malformed values fail closed and are rejected by the WebUI validator.
* "Check now" actually checks: an explicit `updater check` sets
UPDATER_REGISTRY_FORCE=1. The flag existed but nothing ever set it,
so the button silently reused the 6h digest cache and could not find
a build the user knew had shipped. Force also overrides interval 0,
which now means "manual-only" as documented in the roadmap.
* Registry stamp moved from /tmp to <system>/logs: the task processor
runs under PrivateTmp, so daemon and CLI each kept a separate 6h
clock and the daemon's reset on every service restart.
* A failed automatic attempt is no longer invisible: the scan emits
auto_attempted_digest (the one-shot no-retry stamp), and when it
matches the available build the UI stops promising an install that
will never come — per-app detail explains, the fleet row gets an
"auto failed" chip, and the Overview board counts it as needing you.
Also corrects the CFG_TIMEZONE label: it sets the containers' TZ only;
scheduled tasks follow the host clock (timedatectl), and the old
"Timezone for scheduled tasks" wording promised a knob that never
existed. The window + auto_window display state plainly WHEN updates
land, answering "how does the user know when the next update happens".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
navidrome's install died on 2026-08-01 and left no explanation: the log
had only "Started container for navidrome (exit 1, up_app.sh:130)". The
compose output was captured into `result` and never read, and stderr was
not captured at all — so the one thing that says WHY (image pull EOF,
port clash, missing external network) was thrown away at the moment it
mattered.
Capture stderr and print the tail of the output on failure, before
checkSuccess (which can exit). Both the rootless and rooted call sites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First real end-to-end auto-update on a live install failed like this:
Automatically updating trivy (a recovery snapshot is taken first)
Snapshotting trivy before update…
Pulling new image(s) for …
Update of failed — rolling back…
Could not roll back automatically
The app name went empty after the snapshot. Cause: bash is dynamically
scoped, so a callee assigning an undeclared variable writes the CALLER's
local of that name — and a `while read app` loop leaves it EMPTY at EOF.
webuiBackupAppStatus's dashboard generator runs at the end of every backup
and did exactly that to updaterApplyApp's `app`.
Nothing was damaged: the pull ran against an empty name, failed before
touching the image, and the rollback was a no-op on a nonexistent app.
Fixed both ends. The generator (and three gluetun loops with the same
latent leak) now declare `local app`. updaterApplyApp/updaterRollbackApp
hold the name in `_upd_app` so they no longer depend on every callee's
hygiene, and updaterApplyAll stops leaking its own loop var.
This is exactly the untested path the roadmap flagged: "apply/revert not
yet exercised end-to-end on a live install with a pending update."
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the decision half of the app updater. Detection (P2) and the
snapshot-first apply/revert (P3) were already real, but nothing ever
pressed the button — every update waited for a click.
CFG_<APP>_UPDATE_TYPE=auto|manual per app, default auto (33 templates)
CFG_UPDATER_AUTO=true|false master switch, default true
updaterAppPolicy resolves the two the way backupResolveStrategy already
resolves backup strategy: the global switch can only make things more
manual. updaterApplyAuto runs at the end of `updater check` and enqueues
the ordinary updater_apply task for each auto app that has an update —
never applies inline, so an automatic update is the same code path, task
log, History entry and Roll back button as a manual one.
Safety: each attempt stamps its target digest under generated/auto/, so a
build that fails is rolled back and then left alone rather than retried on
every scan; in-flight updater tasks are skipped so scans can't stack.
Tracked end to end: updates.json carries each app's resolved update_type,
History entries carry trigger=manual|auto. The WebUI says whether updates
install themselves, chips only the apps that opted out, labels automatic
history, and — since an auto app's pending update needs no decision — keeps
it off the Overview board's "Needs action" view.
Also fixes artifactApplyAuto enqueueing without --detach: it runs inside
the single-threaded task processor's own poll, so following the new task in
the foreground waits for a task that cannot start until it returns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sudo-rs — the default sudo from Ubuntu 25.10, so on 26.04 — does not
implement bare -E. It does not reject it either: it warns to stderr
("preserving the entire environment is not supported, '-E' is ignored")
and runs the command with the environment DROPPED, leaving the exit
status untouched. Callers capture stderr, so the warning is invisible and
the backup engines simply never receive RESTIC_PASSWORD / BORG_PASSPHRASE
/ KOPIA_PASSWORD and cannot open the repository.
Name the nine vars explicitly via --preserve-env=<list>, which sudo-rs
and classic sudo (>=1.8.21, so Debian 10's 1.8.27) both honour, so this
needs no version gate. The list is cross-checked against every
RESTIC_/BORG_/KOPIA_ var the engine env scripts export.
The list lives in variables.sh with a literal fallback in runBackupOp,
because init.sh sources run_privileged.sh directly during install without
ever loading variables.sh — an unguarded empty list would silently
reproduce the same dropped-credential bug.
restoreFirstRunDiscover now goes through runBackupOp rather than issuing
its own sudo. It was the only backup-engine call bypassing that funnel,
which is why it missed this fix by construction; routing it back also
gives it the -H that keeps restic's cache under the backup user's HOME.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get.docker.com/rootless aborts when ip_tables/ip6_tables aren't loaded.
Ubuntu 24.04 and 26.04 ship both modules but don't autoload them on a
fresh box, so rootless setup died there — and because the caller captures
its output into $result, the reason never reached the console or the
error report. The install continued, reported success, and printed
credentials for a WebUI that was never running.
initPrerequires now modprobes both modules and persists them to
/etc/modules-load.d/libreportal-rootless.conf for subsequent boots,
failing with an actionable message when the kernel genuinely lacks them
(container/VM kernels without netfilter).
installDockerRootless gets its own guard, since it also runs outside
init.sh via start_docker / rootless_start_setup. It only attempts
modprobe when it can — the de-sudoed manager has no modprobe in the
LP_SYSTEM allowlist, matching how ubuntu.sh handles sudo-apt — and
returns non-zero rather than proceeding into a failure whose message
would be swallowed. Already-loaded modules are a clean no-op, so the
normal post-install re-run path is unaffected.
Uninstall removes the drop-in alongside the sysctl ones, and it's listed
in the footprint summary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`apt install p7zip*` only still resolves on Debian 13 / Ubuntu 24.04+
because the renamed `7zip` package happens to declare `Provides: p7zip`.
That is an alias we don't control, so pick the real package name against
the freshly-updated lists instead: `7zip` where it exists, `p7zip-full`
otherwise. Match on a real package stanza rather than apt-cache's exit
status, which returns 0 with empty output for provided-only names.
Also in this path:
- apt -> apt-get for the scripted calls, so the "apt does not have a
stable CLI interface" warning stops polluting the captured $result.
- drop a duplicated `pv` from the package list.
- move the package list below `apt-get update` so the 7-Zip probe reads
current lists.
Debian 10 sysctl check now reads OS_TYPE/OS_VERSION from detectOS rather
than shelling out to lsb_release, which minimal images don't ship. This
also stops a non-Debian release numbered "10" from matching.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
24.04 is the current LTS and was still hitting the untested-OS prompt
while the newer 26.04 did not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ubuntu 26.04 hit the "untested and may not be fully supported" prompt
and blocked non-interactive installs. Add it to the supported list.
Also escape the dots in the version alternation — unescaped they matched
any character, so e.g. "18X04" was treated as a tested release.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fresh install creates an empty containers root but leaves the rootless
daemon's own container state untouched. Restarting the daemon then runs its
container-restore pass, which resurrects the previous install's containers —
and Docker materialises each missing bind-mount source first, creating an empty
DIRECTORY even where the mount is a file.
That is the trigger behind the <app>.config stub directories: at 19:38:59 the
daemon re-created every missing mount source for a container built 40 minutes
earlier, runc then failed with "not a directory: Are you trying to mount a
directory onto a file", and the abandoned stubs collided with the install's own
copies 18 seconds later.
Add dockerRemoveStrandedContainers, run right after the rootless daemon
restart: remove containers whose compose project directory no longer exists, so
the next restart has nothing to resurrect. Scoped to project directories under
the LibrePortal containers root, so unrelated containers on the host are never
touched, and gated on the daemon answering.
Signed-off-by: librelad <librelad@digitalangels.vip>
Docker materialises a missing bind-mount source as an empty directory when a
container starts. The WebUI compose mounts ./libreportal.config as a file, so a
container start before the config landed left a directory at that path — and it
was self-perpetuating:
- copyFolder's tar extract aborted the whole source copy with
"libreportal/libreportal.config: Cannot open: File exists" (exit 2)
- dockerConfigSetupToContainer guards on [ ! -f ], which a directory fails, so
copyFile dropped the real config INSIDE the stub
- the closing -e / -r sanity checks both pass on a directory
The installer then reported success while libreportal-service crash-looped on
EISDIR reading /app/libreportal.config, leaving the WebUI unreachable.
Add repairStubDirForFile: promote a same-named file out of the stub, drop the
directory, and report if the path still isn't a regular file. Call it before the
WebUI source copy and before the per-app config copy (covers every app, not just
the WebUI), and tighten the closing existence check from -e to -f so a stub can
never pass validation again.
Signed-off-by: librelad <librelad@digitalangels.vip>
The per-app Updates tab pushed a manual "Check" button (header + empty state)
even though scans run automatically on CFG_UPDATER_SCAN_INTERVAL — so an app with
nothing to update read like an empty/actionable page. Replace the manual Check
with a calm status line inside the panel: "Checked automatically · last checked X
· next check ~Y", backup-schedule style. The genuine Apply/Roll back actions stay
(applying is still manual and safe). No auto-apply.
- webui_updater_scan.sh: stamp scan_interval_minutes alongside generated_at in
updates.json so the display needs no separate config fetch (0 = auto off).
- updater-page.js: renderAutoCheckLine() + fmtRelFuture().
- app-tabbed-manager.js: drop the header/empty-state Check buttons; render the
auto-check line; friendlier no-data copy.
- overview.css: style .updater-autocheck (green dot live / muted when off).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
For a generic app (no traefik files, not libreportal) fixPermissionsBeforeStart
prints nothing, so the "Updating file permissions before starting" step showed
its header followed by an empty line — looking like it did nothing. Add an
isSuccessful line after the call so the step reports like every other one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh install ran webuiLibrePortalUpdate twice back-to-back: once in
installLibrePortal step 11, then again in startScan at the end of preinstall.
The 30s time-debounce meant to collapse them is fragile (it never fired on a
recent install — >30s elapsed between the two), and debouncing is the wrong
lever anyway: startScan's pass runs AFTER scanConfigsForRandomPassword
finalises app passwords, so it — not the step-11 pass — is authoritative.
Defer the step-11 generation deterministically during a bootstrap install
(libreportal_bootstrap_install=true), leaving startScan's single pass to do the
work. Standalone reinstalls (no bootstrap flag) still generate in step 11. The
time-debounce stays as a general back-to-back backstop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the Update / Roll-back buttons tell the truth, closing the "new code on
old data" hole a floating tag creates.
updaterApplyApp:
- Anchor-correct: capture before/after from updaterPrimaryImage (the
<slug>-service image), not `grep -m1 image:` — fixes ollama et al.
- Records EXACT build refs in history from->to: repo:tag@sha256:<digest>
(via updaterRefDigest), so history is meaningful even when the tag doesn't
move (a rebuilt `latest`).
- Un-pins any digest a prior rollback pinned before pulling, so Update tracks
the channel again instead of freezing on the rolled-back build.
updaterRollbackApp:
- Before recreating, re-pins the anchor image to the pre-update build's digest
(from history's last update/ok `from`) via updaterSetAnchorRef, so `up` runs
the OLD code — not the current channel head. This is the fix for restoring a
data snapshot but recreating on a newer image.
New helpers (cli_updater_commands.sh): updaterRefDigest (local RepoDigest),
updaterSetAnchorRef (rewrite the anchor image line by service name, preserving
indent + the version sentinel; correct for companion-first apps like ollama),
updaterLastUpdateFrom (roll-back target from history).
Verified the helpers on nextcloud + ollama: pin adds @sha256 to the right
anchor only, sidecars untouched, sentinel preserved, unpin restores, YAML valid.
Caught and fixed a `local a=$1 b=...$a...` same-statement expansion bug that
would have silently no-op'd the rollback pin. End-to-end apply/revert not
exercised live here (no installed app has a pending update on this box).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The per-app object build used `available_version:($available_version|select(.!=""))`
to omit the field when empty. But in jq, a `{key: (empty)}` makes the WHOLE
object construction emit nothing — so every app with no available update (empty
available_version) produced no object and was silently dropped from updates.json.
Only apps WITH a pending update survived; an all-up-to-date fleet showed an empty
list. (Missed in P2's sandbox test because both fixture apps had updates.)
Emit an explicit null instead: `(if $available_version=="" then null else … end)`.
Verified: an up-to-date app (trivy, local==registry digest) now emits with
available_version:null; apps with updates still carry the string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
webuiUpdaterScan now tells the truth instead of hardcoding update_available=false.
Per app it resolves three facts (see roadmap §2): the running RepoDigest (one
`docker inspect` of the anchor image), the display version (OCI label →
versioned tag → channel·shortdigest), and — throttled — the registry's current
index digest for the channel tag (`docker buildx imagetools inspect`, the same
identity as RepoDigest, verified exact). update_available = the two digests
differ. Emits type (versioned|rolling), channel, current/available digests +
versions, and a services[] array (every image line, anchor flagged).
- Registry lookups throttled separately from the scan: CFG_UPDATER_REGISTRY_
INTERVAL (min, default 360; 0 = local-only), own /tmp stamp, reuse of the
prior available_digest between windows so the app list still refreshes every
scan. UPDATER_REGISTRY_FORCE=1 forces a live pull (the Check-now button).
Registry failure (offline/rate-limited) = "unknown", never a false "changed".
- Digest-compare fully detects rolling apps' new builds; for versioned apps it
catches rebuilds of the pinned tag (newer-version enumeration is a later
step) — honest per type, and versioned apps are user-picked via P1b anyway.
- Fixes a P1b regression: updaterPrimaryImage now strips the trailing version
sentinel comment (`s/ #.*//`) via _updaterCleanImageRef — without this the
anchor ref (and the live CVE scanner's image arg) carried the comment.
- JSON built with jq for safe escaping; jq-less fallback keeps output valid.
Verified via a simulated 2-app install: navidrome → "0.62.0" (OCI label) +
detected update; rolling app → channel·shortdigest + update; throttled re-run
reuses the prior digest; all output valid JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
"Control plane" is jargon. Rename the self-heal task and de-jargon all the
user-facing copy: task titles ("LibrePortal - Fix System Issues" / "System
Health Check"), the action label, the badge/banner/panel text and "Fix now"
button, the status summaries, and the heal task-log messages. Behaviour
unchanged; code comments keep the technical term where accurate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The scan read the app's "current image" via `grep -m1 image:` — the first
image line. For apps that declare a companion service first this mislabels the
app: ollama lists `ollama-webui` (open-webui) before `ollama-service`
(ollama/ollama), so an app named ollama reported open-webui's version.
Add `updaterPrimaryImage`: resolve the version anchor from the app's
`<slug>-service` image (the universal primary-service naming convention,
33/33 apps; underscores→hyphens for slugs like libreportal_catalog). Falls back
to the first image line off-convention. Used by both the version and CVE loops.
Verified: only ollama changes (→ ollama/ollama:latest); nextcloud, mastodon,
jitsi, gitea, vaultwarden anchors unchanged.
Also revise docs/roadmap/app-version-updater-and-cve.md to the config-first
direction agreed this session:
- CFG_<APP>_VERSION ADOPTED (was rejected): the #LIBREPORTAL tagging system
makes the config the source (compose tag derived from it), not a second one.
- Two version TYPES — versioned (real tag, version picker) vs rolling (floating
channel, digest-freshness); digest is the uniform detection engine for both,
version numbers are display enrichment. Answers "why not just compare numbers"
(most upstreams publish none; no universal latest-version API).
- Multi-service anchor = <slug>-service, not first line; lock-step sets (jitsi)
= one channel → several image lines; sidecars tracked-by-digest, not headline.
- Phases updated: P0 anchor (done) → P1 config-first pin → P2 detection+display.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Crash-loop detection keyed on `docker ps --filter status=restarting`, but a
backed-off loop sits "exited" between restarts once docker's backoff grows to
tens of seconds — so a slowed loop is missed. Detect via RestartCount CLIMBING
between scans (what a crash loop actually is), unioned with the instantaneous
restarting signal for fast loops. Baseline counts persist in
.health_restart_counts, written only by the throttled check so the heal's
re-scans don't disturb the delta.
Also give the system_health_heal / system_network_heal tasks proper display
(they fell through to the raw command + generic ⚙️): friendly titles in
formatCommandForUser, type icons (🩺 / 🌐) in getTaskTypeIcon, and action
labels in formatActionTitle — so they read as "LibrePortal - Repair Control
Plane" etc. with an icon, like install tasks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
An offline trivy install crash-looped (server FATALs when it can't fetch the
vuln DB), and on rootless docker the restart storm churned the shared network's
port-forwarder until the WebUI's own published host port was torn down — the
WebUI stayed healthy INSIDE its container but was unreachable from the host, with
nothing detecting or healing it.
Three fixes, in the house self-healing style (mirrors the network-drift trio):
1. Control-plane health checker wired into the existing task-processor idle poll
(maybeRegenPoll), no new daemon. dockerHealthScan (read-only) detects daemon
down, a WebUI running-but-host-port-unreachable (the port-forward corruption),
and crash-looping containers. webuiSystemHealthCheck writes
frontend/data/system/health_status.json + self-dispatches a heal — the user
can't click a button on a dead WebUI, so the poll drives the fix. Frontend
health-notifier surfaces a topbar badge + dashboard banner + details panel.
2. Failure cap, enforced centrally by dockerHealthHeal (task-gated): stops
crash-loopers (removing the churn), restarts the WebUI to re-publish a lost
port forward, and — only if that fails — recycles the rootless daemon and
restarts the core container. Caps every app immediately, no template churn.
3. Trivy no longer crash-loops offline: the server runs in a shell retry-loop so
the container stays Up and quietly retries on a backoff instead of exiting
FATAL. Verified: container stays Up across repeated DB-download failures.
Core WebUI compose gains restart: unless-stopped so it self-recovers after a
reboot / daemon recycle instead of staying down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Follow-up to the backup-refresh throttle/dedupe, cutting the cost of the
remote pulls that do still happen.
* restic stats now runs in --mode raw-data (restic_check.sh). The default
restore-size mode walks every snapshot's tree to sum logical file sizes —
the slowest restic op — just to fill a size readout. raw-data reads the
index only and reports the repository's actual deduplicated on-disk size,
which is exactly what the dashboard already labels "deduplicated,
encrypted". raw-data omits total_file_count, so the per-location card now
shows that location's snapshot count (already loaded client-side, and more
useful for a backup repo) instead of a file count.
* engineLocationStats now shares the same per-refresh memoiser as
engineSnapshotsJson (engine_dispatch.sh). Both the locations and dashboard
generators call it per location, so repo stats went from two restic calls
per location per refresh to one. Factored the cache into _engineCachedPull.
* SSH connection reuse for SFTP locations (backup_ssh.sh): ControlMaster=auto
with a self-reaping ControlPersist master, so the several restic
subprocesses a refresh/backup spawns against one location share a single
authenticated connection instead of a fresh handshake each — the dominant
per-call cost on a high-latency link. Toggle via CFG_BACKUP_SSH_MULTIPLEX.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The "Refreshing backup data..." step on every WebUI update fired one
`restic stats` (restore-size mode — the slowest restic op) plus ~4
identical unfiltered `restic snapshots --json` pulls per enabled
location (dashboard, snapshots, app-status, migrate each pulled their
own), all over a fresh SSH connection for remote repos, on every pass
with no throttle — the slow, "frozen"-looking line users hit on poor
links.
Two fixes:
1. Dedupe. engineSnapshotsJson transparently memoises the first
unfiltered whole-repo pull per location to a shared cache dir
(LP_SNAP_CACHE_DIR, set by webui_updater around the chain), so the
four generators reuse one restic call instead of four. Filtered and
failed/empty pulls always fall through to a live call.
2. Throttle. Backups and location changes already regenerate this data
live when they happen, so the routine pass is only a drift catch-up.
Split the cheap local-only generators (engines/schema/passwords —
no remote I/O) out to always run, and gate the remote pull behind
CFG_BACKUP_DASHBOARD_REFRESH_INTERVAL (minutes, default 30, 0 =
every update). A completed backup touches a dirty marker that forces
the next pass to pull; WEBUI_UPDATER_FORCE still forces a full
refresh.
Net: N locations x 5 remote restic calls every update -> 1 call per
location, only when something actually changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Trivy runs as a server whose vulnerability DB downloads on first boot; until it
lands no scan can produce results. Previously the updater generator wrote an
empty-but-valid cves.json the moment the file was missing, so installing Trivy
painted a green "no known vulnerabilities" all-clear that was actually a lie —
the DB hadn't even downloaded, and the Updates/Security view gave no signal.
Add an honest scanner state the WebUI branches on:
- containers/trivy/scripts/trivy_scan.sh — trivyScannerState (absent |
db_updating | ready) via `trivy version -f json`, trivyDbUpdatedAt, and
trivyScanImageCves (per-image scan normalized to {id,severity,package,
installed,fixed_in,url}, deduped). All degrade safely on error.
- webui_updater_scan.sh — stamp cves.json with scanner.state; only run real
per-image scans once the DB is ready. Always rewritten so state tracks live.
- updater-page.js — Security tab shows a loading box while the DB updates, an
install nudge when absent, and the genuine 🎉 only when ready+empty; Overview
CVE card sub + hint reflect the state.
- overview-manager.js — fleet Security row surfaces the "building CVE database"
pending state instead of silently omitting.
- function_manifest.sh — regenerated for the new trivy_scan.sh functions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New containers/trivy app (aquasec/trivy in server mode, private API port,
docker socket + cache volume) in the security,recommended categories, plus
placement in the setup wizard recommended step, the server-side install
tier after crowdsec, and the CLI first-install prompt. The updater's CVE
scan (design doc P4) will gate on this app being installed; §5 of the
design doc updated to record the app-based decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
After 'Generated apps-tools.json...' the next step reaches remote backup
locations (restic over SSH) with its output captured in $(...), so on a
slow connection the updater looked stalled with no message. Print an
in-progress notice before it, mirroring the per-app refresh hook loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Config sections (Basic, Mail, DNS, Firewall, …) rendered as a bare name. Add a
small data-driven icon before each: config files now carry a `# @icon <emoji>`
directive in their header, the config generator surfaces it as the subcategory's
`icon`, and the config editor prepends it to the section <h3> (covers every
render path — regular/master/toggle/domains/catalogs/whitelist — since they all
drop displaySubcategory straight into the heading).
- webui_generate_configs.sh: parse `# @icon` (first 10 lines) → emit "icon".
- config-manager.js: prepend subcategoryData.icon to the section title.
- configs/**: added an @icon to all 23 base config files (backup/general/network/
security/webui). The live install's config files get the same directive + a
configs.json regen so existing installs show the icons too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- Catalog block now shows the official catalog as a pinned row #1 (verified
badge + its URL from the generated data). Removing it opens the shared
confirmation dialog with a REQUIRED tickbox (generalized to accept custom
checkbox text) — it only hides official apps from browsing; system updates are
unaffected. Backed by a new CFG_CATALOG_OFFICIAL_ENABLED flag the scan honors.
- Trimmed the info-banner wording per feedback.
- Config option sections: the body (after the divider, before the buttons) now
sits in a darker boxed panel (bg + border), matching the content-box look used
elsewhere. Applies to regular grids + the block editors; scoped to the main
config editor.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the intended model: catalog sources should live in the config place (like
domains), and registry_catalog.json should be a purely GENERATED artifact
derived from them — not the source of truth. Replaces the earlier
$docker_dir/catalog/sources.json store.
- New configs/general/general_catalogs — CFG_CATALOG_1..9, one catalog base URL
per slot ("url" or "url|channel"), domains-style. Official stays pinned as
source #1 (derived from CFG_RELEASE_BASE_URL, not listed here). Slot N → source
idx N+1 (stable id for the Add picker / `app add --source`).
- catalog_sources.sh now reads/writes those CFG vars (via updateConfigOption)
instead of a JSON file; dropped catalogSourcesFile + the enable/disable toggle
(presence = enabled; remove = clear the slot).
- configUpdateBatch regenerates registry_catalog.json when a CFG_CATALOG_* key
changed — so pressing Save in the WebUI rebuilds the browse data.
- webuiRegistryCatalogScan is unchanged (still iterates catalogEnabledSources).
Verified: CFG_CATALOG_1/2 → sources at idx 2/3, empty slots skipped, url|channel
parsed, official pinned at idx 1.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
startPreInstall called installLibrePortalImageWebUI standalone AND then
installLibrePortalAppWebUI, whose app-install step 1 builds the same image
again — so a fresh install built libreportal-service twice back-to-back
(2nd mostly Docker-cache hits, but a redundant copyFolder + "this may take a
while" block). The standalone call only matters on a rebuild where the
app-install path SKIPS (libreportal already in the DB). Gate it on exactly
that: build standalone only when libreportal is already installed. Net: one
build in both cases (fresh → via app install; reinstall → standalone).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the "task loops eternally" bug: the lazy-autoload stub was
fn() { source "$file"; fn "$@"; }
If the source fails — the real case: an app-install/deploy rsync briefly
removes-then-replaces a generator file while a setup task calls it — `fn` is
never redefined, so `fn "$@"` re-invokes the *stub*, which sources the (still
missing) file, which re-invokes the stub… A single setup finalize recursed
12,050 levels, flooding the task log and taking 82s before it happened to
recover when the file reappeared. A permanently-missing file would never
recover.
Fix (root cause): drop the stub before sourcing —
fn() { unset -f fn; source "$file"; fn "$@"; }
so a failed source degrades to one "command not found" (rc 127) instead of
unbounded recursion. Regenerated function_manifest.sh (975 stubs, reformat
only — no function-set change).
Failsafes on the task processor (defence in depth, per request):
- FUNCNEST cap (TASK_FUNCNEST_MAX, default 1000) inside the task's eval
subshell — any runaway recursion now aborts in milliseconds instead of
spamming the log until the stack/disk gives out.
- Wall-clock cap (TASK_MAX_RUNTIME_SECS, default 7200s, 0=off) — the heartbeat
watcher TERM→KILLs a task's process group once exceeded and marks it failed
(distinct from a user cancel via a .timeout marker). Generous so real long
installs/backups/migrations finish.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Per feedback — shorten the install-failure error (drop the extra hint line) and
the rootless MTU notice.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An image-pull failure (e.g. the MTU-EOF black hole) left apps marked
installed+active with no container: the compose `up -d` failure was invisible
because (a) checkSuccess read $? after `_rc=$?`, always printing "✓ Started",
and (b) the exit code is dropped across dockerComposeUpdateAndStartApp →
dockerComposeUpdate → dockerComposeRestartAfterUpdate, and installApp never
checked it — so post-install integrations ran and set status=1 regardless.
- up_app.sh: restore $? to the compose exit before checkSuccess (both rootless
and rooted) so a failed `up -d` is reported as an error + logged, not "✓".
- app_install.sh: after `up`, reality-gate on the app's compose project having
at least one container (ps -a, so a slow-to-start container still counts).
If none exists, print a clear failure, skip _appPostStartIntegrations (which
is what records the app + sets status=1), and return non-zero.
Verified: the gate query passes for a running app (libreportal) and refuses
apps with no container (navidrome/ipinfo after their pulls EOF'd).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
