Phase 2 and 4 of docs/roadmap/storage-locations.md. Apps can now be
placed on a location and moved between them.
CFG_<APP>_STORAGE lands in all 37 app templates, holding a location NAME
rather than a path: names survive a migrate to a host with different
disks, paths do not. The 11 infrastructure apps that other apps reach by
literal path (traefik, prometheus, grafana, adguard, gluetun, crowdsec,
headscale, dashy, pihole, unbound, wireguard) are pinned. libreportal
itself never gets the key — it is pinned structurally by webuiDir.
Pinning needed no second config key. "Pinned" is not a fact about a value,
it is a statement about whether the field may be edited, so it goes in the
comment beside **ADVANCED** and **DEV** as **READONLY**, and the field
factory renders those disabled. That marker earns its keep beyond this
feature: derived fields already warned in prose that editing them does
nothing (crowdsec.config:72) next to a perfectly editable input.
storage_app_config.sh keeps the comment honest — it carries the resolved
path for hand-recovery and regenerates the dropdown from the registry, but
only writes when something actually changed, since the app .config is
user-editable and lives in the container-owned tree.
app move stops the app (a live copy of a running Postgres is a corrupt
copy), snapshots it, copies, verifies, and only then removes the source.
The copy runs in libreportal-ownership because it must: app data holds
rootless sub-UID files the manager can neither read nor recreate.
Verified against two real ext4 filesystems that a cross-device move
preserves uid 231141 and the payload, and that the source survives every
refusal path — unregistered destination, the WebUI app, a traversal in the
app name, and an occupied destination.
Task titles registered in both tables, with a specific rule so a move
renders as "Nextcloud - Move to bigdisk" rather than the generic fallback
dropping the destination. lp-task-names could not be run to confirm — it
borrows the WebUI container's node and no containers are running.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 of docs/roadmap/storage-locations.md — locations can now exist.
Nothing places an app on one yet; that is phase 2.
libreportal-storage is the only writer of the root-owned registry, and
its admission rules are what make adding a location safe: absolute and
canonical (a symlinked path is refused), outside the protected system
set, non-nesting with any known root in either direction, and EMPTY — or
already carrying our marker, which is the adopt case for a drive that
already holds app data. Root only ever chowns an empty directory, so
acceptance cannot hand away anything that existed. The parent must also
not be manager-writable, which is what closes the validate-then-chown
race; /mnt and /srv qualify, a path inside the manager's home does not.
The fitness checks answer a different question — "will app data actually
work here" — and grade rather than refuse. Only checks 1-5 (filesystem
type, mount options, ownership, sub-UID range, write/read-back) can block.
Reboot persistence and removability warn, because both describe supported
setups and start-up is already gated by the marker test.
The ownership probe had to move into the root helper. For a candidate the
directory is not ours yet — a fresh /mnt/disk is root-owned 0755 — so an
unprivileged probe could only ever report "cannot create a directory
here", which says nothing about the filesystem. Verified against a real
loopback ext4: it now reports ownership, sub-UID and read-back cleanly.
The Disks view unions the registry with attached hardware, registry
first. Verified on the live box that pulling a drive leaves its row in
place as not-attached, naming the app stranded on it, rather than the row
silently disappearing at exactly the moment someone needs it.
Also registers storage_scripts with both loaders, adds the CLI category
(auto-dispatched by cli_initialize.sh), and bumps footprint_version to 6
for the new root helper and the widened sudoers allowlist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The main sweep — ~260 call sites across ~100 files move from string
concatenation on a single root to appDir/storageAppDirs/storageAppConfigs.
On a single-root install the resolved paths are identical, so this is a
no-op until a location is registered.
Enumerators were the interesting half. `for d in "$containers_dir"/*/`
appears in the menus, the registry/artifact scanners and the DNS setup —
and a shell glob cannot list a rootless 751 tree at all, which is the
same bug config_find_file.sh already documents in a comment. Routing them
through storageAppDirs (which enumerates as the owning user) fixes that
alongside the multi-root work.
Three places needed judgement rather than substitution:
db_app_scan.sh deletes database rows and port allocations for apps whose
folder is missing, and reaps "empty" app dirs. With a storage location
unmounted, every app on it looks exactly like that. Each of those
branches now gates on appStorageAvailable first — an app on an unplugged
drive is skipped with a notice, never deleted.
instance_create.sh rewrites cloned hooks so an instance touches its own
directory instead of the base app's. Its sed matched ${containers_dir}<type>,
which this sweep just replaced with $(appDir <type>) — so it would have
silently stopped redirecting, and an instance would have written to the
original's files (the adguard auth adapter case its own comment warns
about). Now matches both appDir forms, verified against bare, quoted,
unrelated-app, legacy and prose cases.
peer_shell/peer_pull streamed and extracted relative to the primary root.
Both now use the app's own root, and peer_shell keeps a single-root
fallback since it runs as a restricted SSH shell with no LibrePortal env.
Also fixes a pre-existing bug found on the way: webui_app_config.sh
tested "$containers_dir/frontend/data/last_update", one level short of the
real tree under the libreportal app dir, so the WebUI refresh trigger
after a config update has never once fired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two mechanical sweeps, no behaviour change on a single-root install.
The 14 `[[ "$p" == "$containers_dir"* ]]` prefix tests that decide
manager-vs-container-user elevation become pathIsContainerData, so a file
on a second storage root is no longer misclassified as manager-owned —
which would have written it with the wrong owner and failed later, far
from the cause. The 65 references to the WebUI's own tree become
webuiDir(), which is pinned to the primary root by design.
Two traps found while doing it:
run_privileged.sh is sourced directly by init.sh without paths.sh, so it
needs a fallback. Defining one named pathIsContainerData was wrong:
generate_function_manifest.sh indexes top-level definitions, and the
resulting autoload stub would have shadowed the real multi-root
implementation with the primary-only fallback — silently classifying
every file on a second disk as manager-owned, which is exactly the bug
this sweep exists to prevent. Renamed to _runCfgIsContainerPath, which
delegates when the real one is loaded.
setup_lock.sh built its path in a top-level assignment, so it was
evaluated at source time and needed the file flagged eager. Made it a
function instead: the path resolves on call, and the file drops off
LP_EAGER_FILES entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixing _instanceRewriteTools does nothing for an instance already on disk, and
a clone from the old code is broken in ways that never announce themselves:
every Tools action answers "App '<slug>' has no tool '<id>'" because
dockerAppRunTool wants app<Ucfirst><Pascal>; `authPersistCfg <type>` writes the
instance's new admin credential into the BASE app's config; and the clone
defines the base app's adapter and tool names while its bodies exec against the
instance's container, so the loader keeps whichever it sourced last and the
base app's user tools can end up administering the instance — decided by
nothing but find(1) order. Seen on a live install: the generated manifest
resolved [appBookstackListUsers] to bookstack_test's copy.
libreportal instance repair [slug] [--dry-run]
Rewrites the template dir only — no container is touched, nothing reinstalled,
so it does not route through the task system the way create/remove do.
Idempotent by construction. Two of the three renames match their own output
(appMattermost_teest… still starts with appMattermost), and a clone from the
old code is only PARTLY wrong — its suffix hooks were always correct and end at
the slug with no trailing underscore, which the infix rule would otherwise read
as type + id + () and append the id twice
(appSetupComposeTags_nextcloud_family_family). Three sentinels park the
already-correct spellings before the rewrite and restore them after, so a
healthy instance is a no-op and an interrupted run can just be re-run.
Verified against fixtures built with the old rule set for all five
multi-instance apps that ship tools: after repair each tree is byte-identical
to a fresh clone from the fixed cloner, a second pass reports "already
correct", and --dry-run leaves checksums untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four defects that all reduce to "a function an instance defines is invisible
to the code that dispatches it". Reported as `mattermost_teest has no upgrade
verifier`, for an app whose verifier was on disk the whole time.
* generate_function_manifest.sh shipped 0664. lpRegenArrays invokes it as an
executable, so it died rc=126 on every call and `|| true` swallowed it — the
manifest was never rebuilt on any live system, only laid down at deploy.
Its sibling generate_arrays.sh is 0775, which is why the files_*.sh arrays
looked current while the manifest was byte-identical to the shipped copy.
* lpRegenArrays now runs both generators through bash rather than depending on
the exec bit, reports a manifest failure instead of hiding it, and treats a
new containers/<app> dir as stale — the one event on a live box that adds
functions was the one the scripts/-only mtime check could not see.
* updaterHasVerifier consults the disk before answering no. The CLI runs
LP_LAZY=1, where the container scan is skipped and every function must come
from the build-time manifest, so an app created after the build reads as
having no verifier. GATE 1 then refuses an upgrade that is fully verifiable,
and updaterUpgradeAuto's `|| continue` drops the app in silence for good.
Self-healing regardless of manifest staleness, which matters because a
self-update restores the shipped manifest and drops instance entries again.
* _instanceRewriteTools gains three renames. authAdapter_<type>_<method>() was
caught by neither the prefix rule (no word boundary before _<type>) nor the
suffix rule (needs () right after the type), so the clone defined the base
app's adapter name while pointing at its own container — every instance user
tool answered "does not implement", and which definition survived came down
to find(1) order. Bare-app arguments to authAdapterCall/authPersistCfg went
unrewritten too, so an instance's password reset wrote the credential into
the base app's config. And dockerAppRunTool wants app<Ucfirst><Pascal>, which
no rule produced, so every tool on every instance was unreachable. The infix
rename runs before the suffix rename: the reverse order appends the id half
twice (appSetupComposeTags_nextcloud_work_work).
Verified on a live install: the upgrade ladder now plans mattermost_teest
11.9 -> 11.10, and `regen arrays --force` indexes the instance hooks.
Also carries in-flight instance-removal regen work from a concurrent session
on the same worktree (_lpRegenOrphanedApp, instanceRemove's WebUI refresh).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The /tasks page's right-side tick + dynamic Select all / Clear All ⇄
Delete Selected layout now covers the other three management surfaces:
- App detail → Tasks tab: filter bar gains the Clear All button and
master tick; Clear All there scopes to that app's tasks only. The
selection set is resolved through window.tasksManager everywhere —
TasksManager is constructed in several places, and ticks previously
landed on one instance while Delete Selected read another's empty set.
- Apps overview → Updates: the header's Update all button now morphs to
Update Selected (N) + Clear in place as rows are ticked, replacing the
separate selection bar between toolbar and list.
- App detail → Backups: each snapshot row gains Delete + a right-side
tick; a toolbar atop the list morphs Delete All ⇄ Delete Selected (N).
The whole selection rides in ONE task (delete <app> 1:a,2:b,…) since
the backup surfaces hold one task per subject at a time.
- CLI: backup app delete accepts comma-separated <idx>:<snap> pairs, and
both delete and delete_all now regenerate the WebUI backup JSON so
deleted snapshots leave the screen instead of lingering until the next
backup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unmaintained warning runs on one field — when upstream last rebuilt
the image — and off-Hub apps had no value for it. Hub answers in a single
call; the OCI API does not expose it at all, so an app on ghcr.io, quay.io
or lscr.io simply could not be assessed for staleness, which is the one
signal a user cannot work out for themselves.
It is in the image, just further down: manifest -> (if a multi-arch
index) a platform manifest -> config blob, whose "created" is the build
time. Three requests instead of Hub's one, once per registry window, and
only for the apps Hub cannot answer for — which is why Hub keeps its
cheap path rather than being routed through this.
Index and single-arch manifests are distinguished explicitly rather than
by position: in an index the first digest is a CHILD manifest, in an
image manifest it is the config itself, so reading "the first digest"
would silently fetch the wrong blob for one of the two shapes.
Live: stoat 2026-08-08, bookstack 2026-08-17, speedtest 2026-08-16,
invidious 2026-08-05 — all previously null. Hub unchanged, navidrome
still answered by the single-call path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version discovery spoke only hub.docker.com, and every other registry
got a shrug: updaterTagExists returned "no" and updaterRegistryTags
returned nothing. Five apps live off Hub — stoat and wireguard on
ghcr.io, bookstack and speedtest on lscr.io, invidious on quay.io — and
for all of them the updater reported "up to date" having never asked.
That is the same dishonesty as a scan that never ran: an absence of
evidence rendered as a clean bill of health.
There was never a barrier, only unwritten code. The standard
Distribution API needs one extra step: request, read the
WWW-Authenticate challenge, fetch a token from the realm it names,
retry. ghcr.io, quay.io and lscr.io all answer anonymously for public
images — lscr.io by pointing its realm at ghcr.io, quay.io by not
challenging at all.
Docker Hub deliberately keeps its own path. hub.docker.com returns tags
NEWEST-first, so the 100 it pages are the 100 that matter, and it draws
on a different budget from the pull limit — registry-1.docker.io
manifest reads count against the anonymous 100/hour that the updater
needs for actual pulls, and a ladder probes a tag per rung.
Tag LISTING off Hub is a weaker signal and the comment says so: /v2/
tags/list is lexical, not newest-first, and large repos cap the page, so
the newest release can legitimately be absent. Probing backfills it,
which is why the probe fallback added earlier matters more off Hub than
on it.
Verified against all four registries: existence probing correct on eight
cases including true negatives; stoat climbs v0.15.0 -> v0.15.1 through
ghcr.io, and correctly reports nothing above v0.15.1 — the same answer
as before, but now because it looked. Hub unregressed: matrix still
resolves v1.158.0 -> v1.159.0 and nextcloud still ladders 31 -> 32 33 34.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
updaterSetAnchorVersion located the anchor by looking for a service
literally named "<app>-service". That is a convention, not a rule:
matrix names its anchor service matrix-synapse and stoat names its api.
For those apps nothing matched, so the rewrite changed no lines and the
upgrade aborted at step 2 with "could not set version" — after having
already taken a snapshot. Dry runs never showed it because they return
before that step. It now finds the anchor by its bare <APP>_VERSION_TAG
sentinel, the same way updaterPrimaryImage does.
It also moved only the anchor. Some apps are one product shipped as many
images: stoat is nine stoatchat services released together, all on
v0.15.1, expecting matching versions of each other. Stepping the anchor
alone would have put api on v0.16 while events stayed on v0.15.1 — the
exact mismatch that once justified keeping the app off automatic
updates.
The lock-step set is DERIVED from the compose rather than configured,
because the compose already states it: a service moves with the anchor
when it carries a version sentinel, sits on the SAME tag, and shares the
anchor's registry namespace. Both tests are load-bearing and each
rejects a real case — livekit-server is same-namespace but on its own
cadence, for-web is pinned to a commit hash, element-web is a different
namespace entirely, and mongo has no namespace at all. Verified against
copies of five composes: stoat moves all eight sibling services and
nothing else; matrix, rocketchat and nextcloud move exactly one image.
Every sentinel that moved gets its CFG_*_VERSION key set, not just the
anchor's, or the next config-driven regeneration would quietly pull the
locked-step services back to the old version.
One trap worth naming: quotes were stripped with sed 's/["\047]//g',
but \047 is an octal escape awk honours and sed does not — in a sed
bracket expression it is the literal characters \ 0 4 7, so it deleted
every 0, 4 and 7 it saw and v0.15.1 arrived as v.15.1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GATE 1 refuses to ladder an app that cannot prove a rung landed, and
only mastodon, nextcloud and stalwart could. None of those are installed
here, so the stepped upgrade — button or automatic — was unreachable for
every app on the box.
Three fixes.
_updaterPrimaryContainer assumed the container is "<app>-service". It is
a convention, not a rule: matrix names its anchor service matrix-synapse
and stoat names its api (container stoat-api). The verifier therefore
inspected a container that does not exist, saw no state, and could only
time out — on exactly the stateful apps that most need verifying. It now
reads the anchor service's container_name from the compose, buffering
per service block because container_name may sit either side of the
image line.
Added updaterVerifyHttpVersion: poll the app over its PUBLISHED port
from the host, pull the version from a JSON field or a response header,
and require agreement three polls running. Probed from the host rather
than `docker exec … curl` because half these images ship no curl at all
(mattermost is one), so exec-based probing is a coin flip on the
vendor's base image. Version comparison matches only the components both
sides state, since tags and self-reported builds rarely share precision:
v1.158.0 vs 1.158.0, 11.9 vs 11.9.1, 8.7.0 vs 8.7 all agree; 11.9 vs
11.10 does not.
Each app hook is then three facts. Verified live: all three confirm at
the version they are actually on, and all three REFUSE a version they
are not — which is the property that makes stepping them safe.
updaterUpgradeAuto now skips apps with no verifier instead of queueing a
task that GATE 1 will reject, which would otherwise mean a failure
notification every day for an app that was never eligible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hook substituted `simple_style: auto`, a line that only exists in SearXNG's
full bundled settings.yml. The file generated here is the minimal
`use_default_settings: true` form with no ui: block at all, so the sed matched
nothing and the theme setting had never taken effect on any install.
It could not have worked even with the right pattern: the entrypoint chowns
settings.yml to searxng:searxng (uid 977) mode 644 on first start, so the
host-side docker user cannot write to it. The edit now runs inside the
container via docker exec, targeting the real key path
ui.theme_args.simple_style.
Three shapes are handled so the hook stays correct on repeat installs and
alongside hand edits: substitute in place when simple_style already exists,
nest theme_args inside an existing ui: block rather than appending a second one
(a duplicate YAML key SearXNG refuses to load), and otherwise append the whole
block. All three were exercised against the running container and produce valid
YAML with exactly one ui: block. awk rather than `sed a\` for the nesting case,
since busybox sed does not expand \n in appended text.
The value is validated against auto|light|dark|black before being written.
SearXNG checks it at startup and exits on anything else, so an unrecognised
CFG_SEARXNG_THEME would have taken the app down instead of merely looking
wrong; it is now reported and the default left alone.
Verified end to end on a base install and a --local instance: both come up,
serve 200, and report Dark as the selected style on /preferences, each with its
own settings.yml and secret_key. The instance's cloned hook correctly reads
CFG_SEARXNG_PROBE_THEME and targets its own container, since the container name
is built from $app_name. Both test installs were removed afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two halves: the ladder could not climb the commonest versioning scheme,
and nothing ever climbed it on its own.
The ladder stepped by bumping a tag's LAST numeric component, so
v1.158.0 went v1.158.1, v1.158.2, … and never arrived at v1.159.0. It
then failed closed, refusing to build a path. Synapse publishes
v1.159.0 and no v1.158.1 at all, so Matrix could not be laddered by the
button either — three-part semver minor bumps were simply unreachable.
updaterNextRung now considers a bump of every component, keeps the
candidates that exist upstream and takes the smallest: the next release
by definition, whether it lands in the patch position or crosses into a
new major. Shape discipline is unchanged, so 31-fpm-alpine still never
becomes 31-apache, and each rung is still probed, so none can be
skipped. updaterTagBumpAt moves here from the scan, its natural home,
which also breaks a source cycle.
updaterUpgradeAuto then climbs at most ONE rung per app per calendar
day, inside the install window, for apps set to auto. One rung because a
ladder run unattended can be several migrations deep before anyone
looks, and "restore the snapshot from a minute ago" stops comforting
once four have stacked; one a day so there is time to notice. It crosses
a major if that is genuinely the next release — refusing would strand an
app on the last version of its line forever — but one step at a time,
never as a leap. Two stamps: the target rung (a failure is not retried
until something newer ships) and the day.
Every rung goes through updaterUpgradeApp unchanged, so GATE 1 still
refuses any app without a real verifier, and the per-rung contract is
identical to the button: snapshot fail-closed, set version, pull, up,
verify, restore that rung and stop on any failure. History now records
the trigger instead of hardcoding "manual", including on the rollback
paths. CFG_UPDATER_LADDER_AUTO gates the whole thing separately from
CFG_UPDATER_AUTO, because "keep my apps patched" and "move my apps
between versions unattended" are different appetites for risk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audit of per-app hooks/tools found 19 of 33 apps whose helpers would have
operated on the BASE app after cloning. Two general causes, both fixed by
rewriting classes rather than patching apps:
- Container references escaped the rewrite whenever a flag sat between the
docker verb and the target (`docker exec -u git gitea-service …`), since the
old rule only matched a name immediately after the verb — and the hyphenated
form missed the `<type>_` rule too. Hook trees now get the same discovered
identity rename the compose does, reading names from the TYPE's compose since
the clone has already been rewritten by then. Safe to apply broadly: the
compose pass runs first and aborts for any app whose identities aren't
<type>-prefixed, so a bare word like stoat's `api` never reaches it.
- Hooks that build the deployed path as "${containers_dir}<type>/..." instead
of "$containers_dir$app_name/..." read and WROTE the base app's files —
adguard's auth adapter edits AdGuardHome.yaml, so an instance would have
rewritten the original's config. The trailing slash is optional in the match:
dashy tests [[ -d "${containers_dir}dashy" ]] and gluetun cds into it, both
ending at the quote. Only the first path component is touched, so
${containers_dir}prometheus/prometheus/... keeps its inner segment.
Re-audit: all 33 apps with hook trees are clean. Stoat still leaks, but it is
refused at the compose stage and never reaches this code.
Volumes audited too, and need no changes: no app uses named volumes, so the
./relative bind mounts every app uses resolve inside each instance's own
deployed dir. The absolute sources that exist are host or in-container paths
correctly shared read-only (/etc/localtime, /sys, /etc/ssl/certs). Jitsi's
${CONFIG} is set per-app by its own hook to $containers_dir$app_name/... and so
follows the slug.
Bookstack's rewritten tool tree is byte-identical to the live instance's across
all 8 files, so the running instances are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An instance's isolation never needed a domain — its own slug, dir, secrets,
IP and randomly-allocated host port already make two copies independent. But
the routing layer assumed one, so a LAN-only box got a broken instance rather
than a port-served one. Four fixes:
- instanceCreate now rewrites the parent-service column of the cloned config's
PORT_ rows to match the service names it stamps into the compose. That value
is stored as network_resources.parent_service and joined against the
compose-derived service names, so an instance left carrying the TYPE's
service name matched nothing: it rendered in the WebUI with no port, no URL
and no login row despite being up and reachable.
- `instance create --local` (plus a LAN-only toggle in the modal) forces every
port to access=private, traefik=false, for a second copy that should stay
off the domain even when one is configured.
- initializeAppVariables forces the traefik column false when no CFG_DOMAIN_n
is set. Previously a traefik=true port with an empty domain stamped
Host(`app.`) — a trailing-dot host matching nothing — and dragged APP_URL to
https://app. with it, breaking every app that builds its links from APP_URL.
host_setup is blanked for the same reason. The published host port is
untouched; access type, not the traefik flag, gates allocation.
- APP_URL's direct host-port branch now prefers a new $local_ip_v4 (the source
IP for the default route) over $public_ip_v4, which is the WAN address from
an external resolver. LibrePortal never forwards ports, so the WAN address
was unreachable for exactly the LAN/VPN clients that branch serves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The registry helpers rejected any repo containing a dotted host segment,
which caught 'docker.io/authelia/authelia' — Docker Hub spelled out in
full. Those apps were silently skipped by tag enumeration and version
laddering. Strip the docker.io/ and index.docker.io/ prefixes before the
host check; genuinely third-party registries (ghcr.io, quay.io, lscr.io)
are still correctly skipped.
Found by auditing every app's anchor image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The live Nextcloud 31→34 climb left 4.4 GB of images behind — one per
rung, each ~1.5 GB, all still present after it finished. On a small VPS
that is the difference between working and full.
`system reclaim` cannot help: it collects DANGLING images, and every rung
is a real tag, so all of them stay tagged and stay on disk. (Rolling apps
never hit this — moving a floating tag orphans the old image, which
reclaim then collects. It is specific to laddering.)
After a SUCCESSFUL climb only, remove the images stepped through, keeping
the immediately-previous version so a roll-back needs no download.
CFG_UPDATER_UPGRADE_PRUNE=false keeps everything. Never runs on failure,
where the older images are exactly what recovery may need.
Tested: a 3-rung climb removes 31 and 32 and keeps 33; a single-step
climb removes nothing (its previous version IS the rollback target); the
config switch disables it.
Found by looking at the box after the first real ladder run — the feature
worked, and then quietly cost 4.4 GB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`updater upgrade <app> --detach` parsed "--detach" as the target version
and refused with "no safe path from 31-fpm-alpine to --detach". It failed
safe, but blaming the version for a misplaced flag is a poor way to say
the flag is not supported here. Unknown options now say so.
Found during the first live ladder run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ties the ladder and the verifiers together behind a new verb:
libreportal updater upgrade <app> [version] [--dry-run]
Per rung, and every part is load-bearing:
snapshot (fail-closed) -> set version -> pull -> up -> VERIFY -> next
On failure anywhere: restore THIS rung's snapshot, put the version back,
stop, and leave the app on the last version it actually verified at. The
ladder never continues past a doubt.
A snapshot PER RUNG rather than one at the start, because upstream
migrations are usually one-way — Nextcloud 32's schema cannot be undone
by putting the 31 image back. The recovery guarantee is "restore the
snapshot from sixty seconds ago", which only holds if every rung has one.
Two gates before anything moves. An app with no <app>_upgrade_verify is
refused outright: the generic health check cannot see a half-finished
migration, so laddering on it would be a guess wearing a safety label.
And a ladder that cannot be computed end to end refuses rather than
attempting a partial climb.
`updater upgrade` is a separate verb from `apply` on purpose: apply moves
you WITHIN a release line (and may be automatic), upgrade moves you
BETWEEN lines and is always a deliberate act. Dry runs execute inline so
the plan is instant to read.
updaterSetAnchorVersion rewrites the image tag AND its version sentinel
together — updating only the image would leave the sentinel advertising
the old version, and the next config regeneration would silently revert
the app.
Tested with stubs against the real code paths: the no-verifier gate holds
and changes nothing; a dry run has zero side effects; the happy path
snapshots at each current version before moving; a verify failure on rung
2 of 3 stops with the app on rung 1, restored, and never touches rung 3;
a failed snapshot moves no version and pulls nothing; a container that
will not start is rolled back.
NOT yet exercised on a live install — no app here needs a ladder. The
first real run should be a dry run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stepping 31 -> 32 -> 33 is arithmetic. Knowing 32 FINISHED before
touching 33 is the whole safety story, and it is invisible from outside
the app: Nextcloud runs its migration on boot and sits in maintenance
mode — or fails halfway — while Docker reports the container perfectly
healthy. Advance a rung there and a migration has been skipped on live
data.
Contract: <app>_upgrade_verify <app> <expected-tag> <deadline> -> 0
Returns 0 ONLY on positive confirmation that the app serves at the
expected version with nothing outstanding. Unhealthy, indeterminate and
timed-out all return non-zero — uncertainty is a failure, not a maybe,
because the alternative gambles with data.
nextcloud `occ status`: installed, NOT in maintenance, no pending DB
upgrade, and the running major matches the tag. Maintenance
mid-migration is expected and simply keeps waiting.
mastodon /health serving, ZERO "down" rows in db:migrate:status, and
the version from /api/v1/instance matching. /health alone is
insufficient — Puma answers before migrations finish.
stalwart /healthz/ready (per its documented probes), required to hold
stable rather than flash once. Weaker by design: the probes
confirm serving but report no version, and the file says so
rather than implying more.
updaterVerifyGeneric (running + healthy + no restart during a settle
window) is the fallback for everything else, and is explicitly NOT
sufficient to justify climbing a rung — the engine will refuse to ladder
an app with no declared verifier.
9 tests drive the dangerous states directly: maintenance mode, pending DB
upgrade, and a wrong major all correctly REFUSE to verify; clean states
pass. Those three negatives are the ones that would have corrupted data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Foundation for stepped upgrades. Answers one question only — WHICH
versions, in WHICH order — with no side effects, so it can be tested
exhaustively. Applying the rungs is a separate job.
Nextcloud refuses to skip a major ("Updates between multiple major
versions and downgrades are unsupported") and will not start; databases
behave the same way about their data directory. For those apps 31 -> 34
is three upgrades, each with a migration that must finish before the
next begins.
Built by PROBING each candidate rung, not by enumerating tags — because
enumeration is provably unsafe here. Docker Hub pages at 100 ordered by
recency, and the first real-registry run proved the danger: it produced
v4.2 -> v4.4 -> v4.5 -> v4.6 for mastodon, silently skipping v4.3, which
exists (HTTP 200) but had fallen off the newest-100 listing. Skipping a
rung is the precise failure this file exists to prevent, so the ladder is
now built by incrementing and probing: v4.2 -> v4.3 -> v4.4 -> v4.5 ->
v4.6, 4 steps.
Guarantees: same shape only (never 31-fpm-alpine onto 31-apache),
strictly ascending, never a downgrade, rolling tags refused outright, and
a version upstream never published is stepped over only because the probe
said so. If a continuous path to the target cannot be constructed it
returns 1 and prints nothing — refusing to guess, because a wrong ladder
means a skipped migration.
20 unit tests, including the exact listing-truncation case above and the
numeric ordering that would otherwise drive an app backwards (0.9 vs
0.10). Real registry: nextcloud 3 steps, mastodon 4, stalwart current.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The missing piece of hands-off updates/backups: when a task fails while
nobody has the WebUI open, LibrePortal now says so — email (via the
existing Mail settings), ntfy, Gotify, Discord, Slack, Telegram, or
Pushover, configured under Settings → Notifications.
One hook, everywhere: the task processor reports every terminal task to
`libreportal notify task <id>` (detached, never load-bearing — hard curl
timeouts, failures ignored). The POLICY lives in the notify command, not
the daemon: CFG_NOTIFY_EVENTS = failures (default) | all | off, and
cancelled tasks never notify. Failure copy is task-aware — a failed
update says the app was already rolled back and won't be retried, so the
reader knows the box is safe before opening the WebUI.
`libreportal notify test` sends to every enabled channel with per-channel
results. Verified against a local mock endpoint: all webhook payloads,
JSON escaping (quotes/newlines), the events policy, and fail-fast on
dead endpoints (8ms, exit nonzero).
The v0.1.0 per-app NOTIFY_* field-mapping scaffolding (never wired to a
sender) stays as-is; this global channel is the system it was waiting on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes that make the auto-updater a trustworthy background system:
* CFG_UPDATER_WINDOW (default 06:00-08:00 host time, right after the
05:00 backup cron; HH:MM-HH:MM wraps midnight, 'always' = any time).
Gates only the enqueue — scans keep running all day, so the Updates
page stays current and pending updates visibly wait for the window.
Malformed values fail closed and are rejected by the WebUI validator.
* "Check now" actually checks: an explicit `updater check` sets
UPDATER_REGISTRY_FORCE=1. The flag existed but nothing ever set it,
so the button silently reused the 6h digest cache and could not find
a build the user knew had shipped. Force also overrides interval 0,
which now means "manual-only" as documented in the roadmap.
* Registry stamp moved from /tmp to <system>/logs: the task processor
runs under PrivateTmp, so daemon and CLI each kept a separate 6h
clock and the daemon's reset on every service restart.
* A failed automatic attempt is no longer invisible: the scan emits
auto_attempted_digest (the one-shot no-retry stamp), and when it
matches the available build the UI stops promising an install that
will never come — per-app detail explains, the fleet row gets an
"auto failed" chip, and the Overview board counts it as needing you.
Also corrects the CFG_TIMEZONE label: it sets the containers' TZ only;
scheduled tasks follow the host clock (timedatectl), and the old
"Timezone for scheduled tasks" wording promised a knob that never
existed. The window + auto_window display state plainly WHEN updates
land, answering "how does the user know when the next update happens".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First real end-to-end auto-update on a live install failed like this:
Automatically updating trivy (a recovery snapshot is taken first)
Snapshotting trivy before update…
Pulling new image(s) for …
Update of failed — rolling back…
Could not roll back automatically
The app name went empty after the snapshot. Cause: bash is dynamically
scoped, so a callee assigning an undeclared variable writes the CALLER's
local of that name — and a `while read app` loop leaves it EMPTY at EOF.
webuiBackupAppStatus's dashboard generator runs at the end of every backup
and did exactly that to updaterApplyApp's `app`.
Nothing was damaged: the pull ran against an empty name, failed before
touching the image, and the rollback was a no-op on a nonexistent app.
Fixed both ends. The generator (and three gluetun loops with the same
latent leak) now declare `local app`. updaterApplyApp/updaterRollbackApp
hold the name in `_upd_app` so they no longer depend on every callee's
hygiene, and updaterApplyAll stops leaking its own loop var.
This is exactly the untested path the roadmap flagged: "apply/revert not
yet exercised end-to-end on a live install with a pending update."
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the decision half of the app updater. Detection (P2) and the
snapshot-first apply/revert (P3) were already real, but nothing ever
pressed the button — every update waited for a click.
CFG_<APP>_UPDATE_TYPE=auto|manual per app, default auto (33 templates)
CFG_UPDATER_AUTO=true|false master switch, default true
updaterAppPolicy resolves the two the way backupResolveStrategy already
resolves backup strategy: the global switch can only make things more
manual. updaterApplyAuto runs at the end of `updater check` and enqueues
the ordinary updater_apply task for each auto app that has an update —
never applies inline, so an automatic update is the same code path, task
log, History entry and Roll back button as a manual one.
Safety: each attempt stamps its target digest under generated/auto/, so a
build that fails is rolled back and then left alone rather than retried on
every scan; in-flight updater tasks are skipped so scans can't stack.
Tracked end to end: updates.json carries each app's resolved update_type,
History entries carry trigger=manual|auto. The WebUI says whether updates
install themselves, chips only the apps that opted out, labels automatic
history, and — since an auto app's pending update needs no decision — keeps
it off the Overview board's "Needs action" view.
Also fixes artifactApplyAuto enqueueing without --detach: it runs inside
the single-threaded task processor's own poll, so following the new task in
the foreground waits for a task that cannot start until it returns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make the Update / Roll-back buttons tell the truth, closing the "new code on
old data" hole a floating tag creates.
updaterApplyApp:
- Anchor-correct: capture before/after from updaterPrimaryImage (the
<slug>-service image), not `grep -m1 image:` — fixes ollama et al.
- Records EXACT build refs in history from->to: repo:tag@sha256:<digest>
(via updaterRefDigest), so history is meaningful even when the tag doesn't
move (a rebuilt `latest`).
- Un-pins any digest a prior rollback pinned before pulling, so Update tracks
the channel again instead of freezing on the rolled-back build.
updaterRollbackApp:
- Before recreating, re-pins the anchor image to the pre-update build's digest
(from history's last update/ok `from`) via updaterSetAnchorRef, so `up` runs
the OLD code — not the current channel head. This is the fix for restoring a
data snapshot but recreating on a newer image.
New helpers (cli_updater_commands.sh): updaterRefDigest (local RepoDigest),
updaterSetAnchorRef (rewrite the anchor image line by service name, preserving
indent + the version sentinel; correct for companion-first apps like ollama),
updaterLastUpdateFrom (roll-back target from history).
Verified the helpers on nextcloud + ollama: pin adds @sha256 to the right
anchor only, sidecars untouched, sentinel preserved, unpin restores, YAML valid.
Caught and fixed a `local a=$1 b=...$a...` same-statement expansion bug that
would have silently no-op'd the rollback pin. End-to-end apply/revert not
exercised live here (no installed app has a pending update on this box).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
An offline trivy install crash-looped (server FATALs when it can't fetch the
vuln DB), and on rootless docker the restart storm churned the shared network's
port-forwarder until the WebUI's own published host port was torn down — the
WebUI stayed healthy INSIDE its container but was unreachable from the host, with
nothing detecting or healing it.
Three fixes, in the house self-healing style (mirrors the network-drift trio):
1. Control-plane health checker wired into the existing task-processor idle poll
(maybeRegenPoll), no new daemon. dockerHealthScan (read-only) detects daemon
down, a WebUI running-but-host-port-unreachable (the port-forward corruption),
and crash-looping containers. webuiSystemHealthCheck writes
frontend/data/system/health_status.json + self-dispatches a heal — the user
can't click a button on a dead WebUI, so the poll drives the fix. Frontend
health-notifier surfaces a topbar badge + dashboard banner + details panel.
2. Failure cap, enforced centrally by dockerHealthHeal (task-gated): stops
crash-loopers (removing the churn), restarts the WebUI to re-publish a lost
port forward, and — only if that fails — recycles the rootless daemon and
restarts the core container. Caps every app immediately, no template churn.
3. Trivy no longer crash-loops offline: the server runs in a shell retry-loop so
the container stays Up and quietly retries on a backoff instead of exiting
FATAL. Verified: container stays Up across repeated DB-download failures.
Core WebUI compose gains restart: unless-stopped so it self-recovers after a
reboot / daemon recycle instead of staying down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Per the intended model: catalog sources should live in the config place (like
domains), and registry_catalog.json should be a purely GENERATED artifact
derived from them — not the source of truth. Replaces the earlier
$docker_dir/catalog/sources.json store.
- New configs/general/general_catalogs — CFG_CATALOG_1..9, one catalog base URL
per slot ("url" or "url|channel"), domains-style. Official stays pinned as
source #1 (derived from CFG_RELEASE_BASE_URL, not listed here). Slot N → source
idx N+1 (stable id for the Add picker / `app add --source`).
- catalog_sources.sh now reads/writes those CFG vars (via updateConfigOption)
instead of a JSON file; dropped catalogSourcesFile + the enable/disable toggle
(presence = enabled; remove = clear the slot).
- configUpdateBatch regenerates registry_catalog.json when a CFG_CATALOG_* key
changed — so pressing Save in the WebUI rebuilds the browse data.
- webuiRegistryCatalogScan is unchanged (still iterates catalogEnabledSources).
Verified: CFG_CATALOG_1/2 → sources at idx 2/3, empty slots skipped, url|channel
parsed, official pinned at idx 1.
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First slice of multi-catalog ("taps") support. Today the App Center browses one
catalog (get.libreportal.org). This adds an ordered list of catalog sources and
teaches the browse scan to merge them.
- New scripts/catalog/catalog_sources.sh — the source list in its OWN file
($configs_dir/catalog/sources.json). Source #1 is ALWAYS the official catalog,
synthesized live from CFG_RELEASE_BASE_URL/CHANNEL (can't be edited/removed,
always pinned on top). Extra sources are stored as a small JSON array and are
UNVERIFIED (trust=community). Helpers: catalogSourcesJson / catalogEnabledSources
(priority order) / catalogSourceAdd|Remove|Toggle|List / catalogFetchCommunityIndex.
- webui_registry_scan.sh now walks catalogEnabledSources: the OFFICIAL source is
still signature-verified (lpFetchIndexInto, unchanged trust path); third-party
sources are fetched unverified. Apps are merged by slug into one card carrying a
sources[] array in priority order (highest first = default). Trust/verified are
taken from the SOURCE, never the artifact's self-claim, so a community index
can't promote itself to "official". Icons still mirror same-origin from the
official index only. registry_catalog.json gains top-level sources[] + per-app
sources[]; the old source{} object + signed/serial are kept for back-compat.
- New `libreportal catalog source list|add|remove|enable|disable` + `catalog
refresh` CLI (dynamic-routed). Mutations go through the task system (cliTaskRun
"…" "catalog"), never a new mutating API.
Scope firewall: this governs APP BROWSE + ADD only. LibrePortal's own updates and
hotfixes still resolve from the official CFG_RELEASE_BASE_URL alone — lpFetchIndex
is untouched, so a third-party catalog can never become a system-update channel.
Next: `app add --source`, then the WebUI (domains-style block source manager +
Add-dialog source picker with the Official badge / unverified warning).
Signed-off-by: librelad <librelad@digitalangels.vip>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three coupled defects in the first-run wizard flow, all surfacing as
"it said complete but nothing was set up":
1. Zero-app installs sailed through. With no apps ticked, setup was just
config+finalize, finished in seconds having installed nothing, and
fast-forwarded to an empty App Center. Add a self-contained in-wizard
confirm ("Install with no apps?") before submitting. Can't reuse the
shared confirmation-dialog component — it isn't loaded this early in
boot — so the dialog is rendered by the wizard itself and mounted on
<body> to escape the aurora surface's FX-stacking rule.
2. finalize declared success unconditionally. It never inspected the
per-app install tasks, so a failed app still yielded "your install is
ready" + a redirect. Pass the setup group id to `setup finalize`; it
now rolls up the group's app-install task results and logs a clear
partial/failed verdict. The WebUI completion watcher gates the welcome
toast + App Center hand-off on the whole group succeeding, not just on
finalize completing — a failed app now keeps the user on the tasks page
with an error toast instead of a false all-clear.
3. Setup task count was confusing (banner "of 2" while the page listed 3).
On dev/git installs the topbar's dev-mode auto-enable raced the wizard's
own CFG_DEV_MODE write and spawned a second, un-grouped config-update
task mid-setup. Skip that auto-enable while a setup handoff is active
(the wizard already persists CFG_DEV_MODE); it runs on the next load if
still needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
webuiRegistryCatalogScan (run by updater check, same atomic keep-prior
pattern as webuiArtifactScan) writes apps/generated/registry_catalog.json:
the type:"app"/kind:"bundle" rows of the signed index annotated with
defined/installed, browse metadata from the envelope meta, and icons
mirrored into core/icons/apps/registry/ ONLY when their bytes match the
sha256 pin in the signed index — the browser stays same-origin; a tampered
or oversized icon is skipped, never served.
webuiArtifactScan now selects type=="hotfix" so app rows never render as
pseudo-hotfixes in the Improvements tab, and counts+logs artifacts of
unrecognized type instead of surfacing them (the §8.1 forward-compat
firewall on the scan path).
Harness vs a locally served registry: 14/14 (catalog row + meta + flags,
icon pin verify + tamper skip, hotfix-only stream, unknown-type skip+log,
unreachable-registry keeps prior files).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Opens the two designed seams (roadmap §8.4): _artifactResolve accepts
type:"app" (slug validated, the installed-app gate skipped — presence is
the collision policy's call), and payload.kind:"bundle" gets its own APPLY
flow. The download core (sha256 pin vs the signed index + minisig +
refuse-unsigned) is factored into _artifactDownloadVerified, shared by ops
and bundle payloads.
A bundle add: fetch → quarantine-validate → place in the definition tree
(staging + one rename, manager funnel) → lpRegenWebui → verify the app
surfaced in apps.json → applied-record with a precise undo → History.
The validator is fail-closed (traversal/absolute paths, links/devices,
single top-level dir == slug, charset, size/entry caps, set-id strip,
config TITLE+CATEGORY + compose present, bash -n every .sh) because the
definition tree is live-sourced on every CLI start — nothing lands there
before trust + quarantine pass. Collision policy: installed-live refused,
local definitions win, registry-owned re-add = reversible definition
update (prior tree packed into the undo). Revert removes/restores the
definition (refused while installed) and regens. Apps never auto-apply
(type filter kept + publisher forces auto:false).
New verb: libreportal app add <slug|artifact-id> (app_add task; resolves
by slug via appAddFromRegistry, ambiguity refused).
Also fixes the second half of the sigstate-propagation bug class:
artifactApply captured $(_artifactResolve) in a subshell, stranding
_ART_INDEX/_ART_APP/_ART_SCOPE AND the LP_INDEX_SIGSTATE the apply gate
enforces — on a signed box every apply would have refused as unsigned.
Resolve now assigns globals (_ART_JSON) and is called directly.
Source-and-mock harness: 46/46 (resolve gates, 14 validator refusals,
happy add, collision matrix, definition-update round-trip, revert
semantics, postcheck + record-failure rollbacks, apply-auto exclusion,
app add verb).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Every caller captured the index with var=$(lpFetchIndex), which runs the
fetch in a command-substitution subshell — the LP_INDEX_SIGSTATE global it
sets never reached the caller. On a box with real signing active the
artifactApply/apply-auto gates would therefore refuse a correctly signed
index (fail-closed, but the apply path would be dead on arrival the day
signing activates), and artifact index / the WebUI scan would report a
verified feed as UNSIGNED.
New lpFetchIndexInto <var> [cache] runs the fetch in the calling shell and
assigns via printf -v; all four call sites converted. Verified with a
source-and-mock harness against a locally served index: 10/10 (sigstate
reaches caller, serial high-water, anti-rollback refuse, staleness refuse,
id enumeration, envelope round-trip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The rootless WebUI container reads its bind-mount sources (configs/webui/*)
through the container-owner GROUP since a2376e2 switched those files from
world-readable to 0640 group=container-owner. But the WebUI credential
randomizer rewrites webui_logins via `sed -i` as the non-root manager, which
recreates the file with the manager's own group — dropping the container-owner
group. The installer then started the container immediately, so node hit
EACCES on /app/webui_logins at require-time (parseConfigFile) and exited 1;
nothing listened on the WebUI port. `libreportal webui login reset` had the
same latent bug (rewrite → restart). Under the old world-readable model a
post-sed file stayed o+r so the container could still read it, which is why
this only surfaced on fresh rootless installs after a2376e2.
Fix: make reconcileWebuiDirOwnership the single "ready the WebUI for its
container" pass — it now also restores the configs/webui bind access (new
`webui-bind` ownership action) on top of the container-dir chown. Reorder the
installer so the credential randomizer runs BEFORE the before-start permission
pass, making that pass the last ownership touch before the container starts;
and call reconcileWebuiDirOwnership before the restart in login reset.
Live box recovered via `libreportal-ownership reconcile`; WebUI 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The Services tab restart button POSTed to a backend endpoint that (a)
checked the app's compose path from INSIDE the webui container, where
the host's containers root isn't mounted — so every restart failed with
'Compose file not found' — and (b) queued a raw 'docker compose restart'
that the host task processor would run as the manager user, which can't
talk to the rootless daemon anyway. Errors surfaced via a bare alert().
Per-service restart now follows the exact shape of the whole-app verbs:
- CLI: 'libreportal app restart <app> [service]' — the optional service
arg makes dockerRestartApp restart just that compose service, via
dockerCommandRun (right user in rootless mode) from the app dir on the
host, where the compose file actually lives. Service names validated
against compose-legal characters before touching a shell line.
- WebUI: the button dispatches a 'service_restart' task action through
the task router (mutations-via-tasks), runs in the background with the
standard task toast + link — no page switch — and failures use the
notification system instead of alert(). Because the task runs host-
side, restarting the WebUI's own libreportal-service now works too.
- Backend: the mutating restart endpoint and its now-unused helpers are
removed; service-routes.js is read-only surface (status + log tails).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Move the WebUI-updater settings out of general_terminal into their own
advanced webui-category file (webui_logs precedent): new
configs/webui/webui_updater holds CFG_UPDATER_SCAN_INTERVAL and the
migrated CFG_HOTFIX_AUTO, listed in webui/.category.
The move only reaches existing installs if the config convergence
machinery works, and three pieces of it silently didn't:
- checkConfigFilesMissingFiles walked a stale hardcoded category list
('general features network' — features doesn't exist; webui/backup/
security never healed). Derive the categories from the template tree
instead, and heal .category metadata too: copy it when absent and
merge missing SUBCATEGORY_ORDER entries when present, so healed files
actually appear in the WebUI Config editor. core_categories removed.
- Option reconciliation never touched ANY nested config file: configs_dir
carries a trailing slash, so rel stripping missed ('configs//'), the
template lookup failed, and reconcileConfigFile early-returned for
every file. Strip the slash before matching.
- reconcileConfigFile's AUTO_DELETE=false branch read a never-populated
live_line array, losing the dropped keys it promised to keep. Populate
it alongside live_value.
Also exclude *.bak from config sourcing (reconciliation writes <file>.bak
next to live configs — now that it runs, sourcing backups would resurrect
deleted keys), and add 'libreportal config check' as a non-interactive
front door to the converge pass (was only reachable via install flows and
the interactive menu).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replace the click-to-scan-only flow with a self-throttled auto-scan that
rides the existing task-processor idle poll (the same shape as the
network-drift check — no new daemon, unit, or endpoint):
- 'libreportal updater check auto' gates on the age of the generated
updates.json vs CFG_UPDATER_SCAN_INTERVAL (minutes, default 30,
0 disables); a fresh file makes the 60s tick a single stat() + return.
Manual checks and post-update rescans reset the clock for free, and a
missing file means the first scan runs ~a minute after install.
- Eligible signed hotfixes keep flowing through artifactApplyAuto, which
only enqueues ordinary tasks — mutations stay on the task path.
- Open updater surfaces (standalone /updater and the fleet Overview's
headless UpdaterPage) follow along with a 60s static-JSON re-read that
repaints only when a generated_at stamp changed; timer released via
dispose() on unmount, ticks skipped while hidden.
- Empty states now say the first scan happens automatically; Check now
stays as the immediate manual override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Lets a *multi-instance-capable* app run as several fully isolated instances
on one box (e.g. two Bookstack/WordPress sites, or a "family" + "work"
Nextcloud) — distinct data, DB, subdomain, backups and update cadence.
Design: an instance is just another app. It gets its own slug (<type>_<id>),
its own CFG_<SLUG>_* namespace, deployed dir, DB row, IP/port allocation and
host, so the entire existing pipeline (scan, install, services, routing,
updater, backups) treats it like any app with zero changes. All
instance-specific rewriting is confined to a clone of the type's template;
the shipped template and the core engine are untouched.
Gating: opt-in per app via CFG_<TYPE>_MULTI_INSTANCE=true. Only Bookstack
carries it for now (the validated reference). The other 31 apps are
unaffected — the feature is invisible unless the flag is present.
- scripts/instance/instance_create.sh — clone + re-namespace config, rewrite
compose identity (container_name / Traefik routers / backup labels) and
per-app tools, set a hostname-safe subdomain (PORT field 10), then hand off
to dockerInstallApp. Plus instanceList / instanceRemove.
- libreportal instance create|remove|list — new CLI category; mutations route
through the task system (no new mutating API endpoint).
- WebUI: "instance of <type>" badge + a "New instance" card action on capable
apps, and a create modal (name + domain# + subdomain, live host preview)
that dispatches the standard task. Capability/instance-of read straight off
the already-exposed app config.
Known follow-ups (documented): flip the flag on more apps after a compose
identity check (Nextcloud next); per-app tools are best-effort isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Closes the gap behind the vpn-recreate bug: when the shared network is
recreated with a different /24, every app's stored static IP is left
outside it and adoptDockerSubnet only realigns CFG, not the apps.
- networkScanConflicts (network_conflicts.sh): read-only scan diffing each
active network_resources IP against docker's real subnet (via ipInSubnet).
Per-service routing-aware — skips gateway-routed services whose ipv4 is
commented out in the deployed compose, so gluetun apps don't false-positive.
Distinguishes 'daemon down' (benign) from 'network missing' (real).
- webuiSystemNetworkCheck (webui_system_network.sh): self-throttled generator
that writes frontend/data/system/network_status.json (modelled on
verify_status.json). Wired into webuiSystemUpdate AND run unconditionally
every ~60s from the task-processor poll (regen webui is mtime-gated and
would never fire on drift, which touches no source file).
- networkHealConflicts (network_heal.sh) + 'libreportal system network
check|heal [app]': the heal adopts docker's subnet in-process, then re-IPs
stranded apps with reset_network=ip (ports preserved), gluetun first.
Mutating path runs only through the task system (dual-mode, like update
apply); read-only check runs inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CFG_HOTFIX_AUTO (security-breakage|all|off, default security-breakage) seeded in
general_terminal; reaches existing installs via the add-only config reconciler.
- webui_artifact_scan.sh (webuiArtifactScan): fetch+verify the signed index, write
artifacts_available.json ATOMICALLY (build in temp → jq-validate → one write;
keep the prior file on any failure — never emits broken JSON). Annotates each
artifact with applied (a per-id record exists) + applicable (target installed).
- artifactApplyAuto + `libreportal artifact apply-auto`: enqueue apply tasks for
the eligible signed hotfixes — only when the index is VERIFIED-signed, only
auto==true + in the severity policy + applicable + not already applied. Each
apply is its own task (visible in the log + History), never applied inline.
- `updater check` now also refreshes the index (webuiArtifactScan) and runs
artifactApplyAuto — one front door, no second phone-home.
Unit-tested 13/13: policy filtering (security-breakage / off / all), auto:false
exclusion, already-applied skip, non-installed-app skip, unsigned-index fail-closed,
and the scan transform's signed/applied/applicable fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
A 4-lens adversarial security review of the Phase 2 applier raised 19 issues
and confirmed 17 after per-finding verification. All are trust-boundary (they
require the signing key), but several break the explicit "no code-exec, always
reversible, nothing-silent" contract, so all 17 are fixed:
Trust path — fail CLOSED, never misreport:
- lpFetchIndex now surfaces the real signature state (LP_INDEX_SIGSTATE);
artifactApply REFUSES to mutate unless the index is actually verified, and
_artifactFetchPayload refuses an unsigned payload. The read path still
tolerates dev/unsigned but now says "UNSIGNED" instead of "Signed + verified".
- valid_until and index_serial are now MANDATORY + numeric in lpFetchIndex
(missing = refuse) — closes the anti-withholding / anti-rollback fail-opens.
Injection / code-exec (defense in depth even for a signed payload):
- runFileWrite rootless branch no longer builds a `bash -c` shell string with the
destination interpolated — it uses the argv form (like runFileOp), so a path
with a quote can't inject a command as the install user. (shared-helper fix)
- op paths must match a safe-filename charset (no quotes/$/backtick/;/newline);
set-config-key values and set-compose-image refs are charset-guarded too.
- content_b64 is validated as real base64 at precheck.
Reversibility / honest failure:
- dockerComposeUp now returns the real compose exit status (it always returned 0,
so the updater's rollback gate AND the apply's start-failure detection were
fail-open). (shared-helper fix)
- set-config-key undo captures the WHOLE config file (lossless) instead of a
lossy re-parsed scalar; edit-only (rejects an absent key).
- _artifactReplayUndoFile returns non-zero if any inverse op fails; auto-rollback
and revert now record "rollback-incomplete"/"revert-incomplete" + isError
instead of falsely claiming success, and revert keeps the record for retry.
- applied-record write failure is checked — apply rolls back rather than leave an
un-revertable change. System-scope regen failure is no longer swallowed.
- Writes are path-aware (configs/ -> runInstallWrite, container tree ->
runFileWrite) so system-scope hotfixes write/restore correctly.
- Checked lazy-sourcing surfaces a clear error instead of a bare exit 127.
Unit-tested 35/35 (adds: command-sub value rejection, bad image-ref, invalid
base64, quote/metachar path-injection rejection, replay-failure reporting).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The mutating side of the unified distribution primitive (spec §8.3). Hotfixes
can now be applied and reverted, first-party, through the task system.
New scripts/cli/commands/artifact/cli_artifact_apply.sh:
- artifactApply <id>: resolve+gate (applies_when / min_lp / max_lp /
max_footprint / publishers-map role) → fetch+verify payload (sha256 pinned by
the signed index + minisig) → dry-precheck ALL ops (all-or-nothing) → best-
effort snapshot → apply each op recording a precise inverse → bring app up →
auto-rollback (replay undo LIFO, snapshot fallback) → applied-record + History.
- artifactRevert <id>: replay the applied-record's undo log (LIFO).
- Bounded, CLOSED op vocabulary (no run-script/exec, ever): set-config-key,
set-compose-image, patch-file-if-checksum-matches, set-data-file. An
unsupported op rejects the whole artifact at precheck (fail-closed).
- Write-target firewall: scope:app → containers/<app>/ only; scope:system →
configs/ only; the install tree (our code) is off-limits to hotfixes (fork 1).
Drift guards (expect_current / checksum) skip cleanly rather than clobber.
- Two-tier trust: index minisig-verified vs the footprint key (lpFetchIndex)
covers the envelope; payload sha256-pinned + minisig-verified; publishers-map
role gate (a non-official publisher can't claim official). Community per-
artifact-key sigs are gated off until that tier is enabled.
cli_artifact_commands.sh: apply/revert via the task system (artifact_apply /
artifact_revert types — no allowlist needed), + read-only `applied` list.
cli_updater_commands.sh:
- FIX verified safety bug: updaterApplyApp/RollbackApp called `libreportal backup
app "$app"` and `... restore latest`, which parse the app name as the ACTION,
hit the dispatcher's `*)` default (exits 0) — so updates ran with NO snapshot
and rollback was a silent no-op. Call backupAppStart / restoreAppStart directly.
- FIX updaterRecordHistory jq-silent-skip: was `command -v jq || return 0`
(silently dropped the audit entry). Now fail-closed with a brace-agnostic
bash-native prepend fallback; extended with artifact_id/serial/undo_id.
fetch.sh: add _lpJsonEsc (shared JSON-escape for the jq-free fallbacks).
Regenerated source arrays + lazy-load manifest for the new file/functions.
Unit-tested 31/31: every op apply+precheck+undo round-trip, the path-allowlist
firewall (incl. .. traversal + install-tree + cross-app rejection), all-or-
nothing abort, unsupported-op rejection, and the History bash-native fallback
(records + preserves prior entries without jq). A full signed-apply e2e needs
minisign + the signing key (Phase 5 make_hotfix.sh).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Build the read side of the unified distribution primitive from
docs/roadmap/updates-and-distribution.md: one team-signed catalog
(index.json) on the same channel as latest.json, listing type-tagged
artifact envelopes. A hotfix is the first artifact type; apps/themes/
components are future envelope rows through the SAME pipe — the
marketplace seam is just the `type` + `payload.kind` fields.
Phase 1 is fetch + verify + parse only (NO mutation; the snapshot →
ops → rollback → History apply verb is Phase 2):
- Factor `lpVerifyMinisig` out of `lpFetchRelease` (scripts/source/
fetch.sh) — one trust anchor (the root-owned footprint key) now
shared by releases and the index; refactor `lpFetchRelease` to use
it (behaviour-preserving, still fail-closed).
- scripts/source/artifacts.sh: `lpFetchIndex` — download →
verify-before-parse → `valid_until` freshness (anti-withholding) →
`index_serial` monotonic high-water (anti-rollback, TUF-lite) → emit
verified JSON. Trust core is jq-free; parsing accessors prefer jq
with a grep fallback.
- `libreportal artifact index` (scripts/cli/commands/artifact/) —
read-only front door that fetches, verifies and lists. Runs directly
like `updater check` (no task; no mutation).
- Regenerate the source arrays + lazy-load function manifest for the
new files.
Doc: promote the format from vision to spec (§8) — 3 layers
(INDEX/ENVELOPE/PIPELINE), the bounded declarative op vocabulary (no
run-script, ever), the apply pipeline mapped onto existing functions,
the marketplace seam, and resolutions for all five open forks.
Self-tested 12/12: trust core fails closed (real key + no minisign →
refuse), happy path, stale-refused, rollback-refused, signature-refused,
jq + grep parsing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- scripts/webui/data/generators/updater/webui_updater_scan.sh (webuiUpdaterScan):
writes frontend/data/updater/generated/{updates,cves,history}.json from the
installed-apps DB (current image per app from compose). Available-version +
CVE-scanner are clearly-marked pluggable hooks; always emits valid JSON.
- scripts/cli/commands/updater/{cli_updater_commands.sh,cli_updater_header.sh}:
auto-dispatched as 'libreportal updater <sub>' (check/apply/apply-all/rollback).
apply does disaster-recovery FIRST — snapshots the app via the backup engine,
then pulls + recreates (real dockerComposeUp/compose-pull helpers), records
history, and auto-rolls-back on failure. Standard LIBREPORTAL_TASK_EXEC
enqueue/exec split so WebUI + CLI share locking + audit trail.
New .sh files: the array/function-manifest regen self-heals on deploy; the
check path also sources its generator on demand to cover the gap.
NOTE: host-side bash — written to the repo's conventions but not runnable in
this env; this is the surface to test (the WebUI feature is lp-shot-verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replaces the read-only "Largest images" top-10 table with a Tasks-style list of
ALL Docker images, with select-one / select-multiple / clear-all removal that
mirrors the Tasks page UX (row checkboxes, master select-all, a button that
morphs Clear All ↔ Delete Selected (N), an eo confirm modal).
Deletion routes through the task system, NOT a new web API: a new
`libreportal system image rm [--force] <ids>` CLI subcommand (validates each
ref, loops runFileOp docker image rm, reports a tally) is invoked via the
system_image_rm task action — same pattern as Reclaim. The web backend change
is read-only (uncap the existing /storage image list). In-use images are
skipped by default with an opt-in "force-remove" toggle (warned). The page
stays put, toasts, and refreshes on the task's completion event.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Adds per-file integrity attestation on top of the existing signed-tarball
release flow. make_release now generates a SHA256SUMS manifest over the shipped
tree and (when a key is configured) signs it, riding both inside the release
tarball so they land in the install tree with no extra download.
lpVerifyInstall (scripts/source/verify.sh) re-hashes the install tree against
that manifest and verifies the manifest's minisign signature against the
root-owned footprint pubkey, yielding states: verified / modified / tampered /
unsigned / unverifiable / development. webuiSystemVerify writes verify_status.json
(throttled daily, force on demand, also after each update apply), surfaced as an
Integrity line + "Verify now" button on the Admin → Overview Updates card and a
row in the update details panel. `libreportal verify` exposes the same check on
the CLI.
Honest framing: this is a self-check (run by the software it verifies), so red
fires only for genuine modified/tampered states; the badge tooltip points to
out-of-band `minisign -Vm` for an independent guarantee.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Three fixes from testing the storage page:
- Placement: the "Reclaim space" button moves into the page header,
top-right (matching the metric page), instead of sitting in the body.
- It now actually reclaims: build cache needs -a to drop (docker reports
0 B "reclaimable" without it, but it's pure cache — safe to clear), so
the CLI uses `docker builder prune -af`. Previously the safe scope
freed ~nothing on a box whose reclaimable was mostly cache.
- Honest "Reclaimable" number: /api/system/storage was counting the
whole build cache AND unused tagged images, overstating what the safe
prune frees (e.g. 340 MB shown, ~96 MB per docker, button cleared 0).
Reclaimable now = dangling images + build cache only; stopped
containers and volumes are never counted (the safe prune never touches
them). Headline now matches the button's effect.
Also simplify the CLI output (drop the jargony scope notice and the
reclaimed-total greps) and re-enable the now-persistent header button
after the post-reclaim refreshes.
Signed-off-by: librelad <librelad@digitalangels.vip>
Adds a `libreportal system reclaim` CLI command and an orange "Reclaim
space" button on /admin/config/system/storage (the v2 prune control the
page always hinted at).
Scope is deliberately SAFE: build cache + dangling (untagged) images
only (docker builder prune -f + docker image prune -f via the
rootless-aware runFileOp). It never touches volumes (app data) or
tagged/in-use images, so nothing an app relies on is removed.
Wiring mirrors system_update: a systemReclaim() action + system_reclaim
route case run the command verbatim through the task processor. The
button confirms via showConfirmation, shows a spinner, and re-reads
storage usage as the prune lands. Button styled with --status-warning to
match the Reclaimable stat it sits under, with a note clarifying scope.
Signed-off-by: librelad <librelad@digitalangels.vip>
Extends the install-routing spike (e5273a4) to every long-running CLI
command, so CLI and WebUI now share one execution path everywhere:
app install ← already done
app uninstall
app start / stop / restart / up / down / reload
app backup
app restore
update apply
backup app create (matches `app backup` — same end target)
Each handler now has the same shape:
if [[ "$LIBREPORTAL_TASK_EXEC" == "1" ]]; then
<inline call> # processor's recursive invocation
else
cliTaskRun "<cmd>" <type> <app> # user invocation: enqueue + follow
fi
Processor change — crontab_task_processor.sh:
Adds `export LIBREPORTAL_TASK_EXEC=1` next to LIBREPORTAL_NONINTERACTIVE.
Universal bypass: every task command the processor runs (CLI-queued OR
pre-existing WebUI-queued like `libreportal app install adguard`)
inherits the env var, so the inline branch fires and we never
re-enqueue. This also lets us drop the env-var prefix the install spike
was baking into the command string (e5273a4) — cleaner task files +
one place to think about the bypass.
`backup app schedule` (the cron-driven path that already enqueues via
createTaskFile in backup_app_schedule.sh) is left alone — different
entry point, different runtime context, already correctly task-routed.
Why route the fast ones too (start/stop/restart/up/down):
Consistency beats the ~1s task-roundtrip latency for a CLI button.
Locking now serialises a CLI `app stop foo` against a WebUI restart of
the same app; the audit trail covers every state change. Cheap to
revert any individually if the latency turns out to bother someone.
Validated live earlier with `libreportal app install dashy` — task file
written, processor dispatched, follower streamed install live, exit 0
propagated. Same machinery now powers the other 9 handlers.
Signed-off-by: librelad <librelad@digitalangels.vip>
librelad