The task processor is a systemd-service daemon, not a cron job — move it out
of the misleadingly-named scripts/crontab/task/ to scripts/task/.
To stop the systemd unit from baking the processor's in-tree path (the footprint
coupling that forces a reinstall on every reorg), the unit now ExecStarts the
stable wrapper: /usr/local/bin/libreportal __task-processor. start.sh intercepts
that early (after paths.sh, before the heavy load), exports install_scripts_dir,
and exec's the processor with start_script. Future moves/renames need only the
one hand-off updated + a regen — no footprint bump.
- git mv scripts/crontab/task -> scripts/task (filenames kept; cron-watchdog grep
+ function names unchanged)
- libreportal-svc: ExecStart -> stable wrapper launcher
- start.sh: __task-processor internal launcher (export install_scripts_dir; exec)
- crontab_task_processor.sh: fix self-location ../.. -> .. for the new 1-level
depth (latent bug the move would otherwise have introduced)
- regen files_*/function_manifest; add task_scripts to the app/cli aggregates
- footprint_version 3 -> 4 (root-owned svc unit changed -> needs a root reinstall)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replaces the read-only "Largest images" top-10 table with a Tasks-style list of
ALL Docker images, with select-one / select-multiple / clear-all removal that
mirrors the Tasks page UX (row checkboxes, master select-all, a button that
morphs Clear All ↔ Delete Selected (N), an eo confirm modal).
Deletion routes through the task system, NOT a new web API: a new
`libreportal system image rm [--force] <ids>` CLI subcommand (validates each
ref, loops runFileOp docker image rm, reports a tally) is invoked via the
system_image_rm task action — same pattern as Reclaim. The web backend change
is read-only (uncap the existing /storage image list). In-use images are
skipped by default with an opt-in "force-remove" toggle (warned). The page
stays put, toasts, and refreshes on the task's completion event.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Adds per-file integrity attestation on top of the existing signed-tarball
release flow. make_release now generates a SHA256SUMS manifest over the shipped
tree and (when a key is configured) signs it, riding both inside the release
tarball so they land in the install tree with no extra download.
lpVerifyInstall (scripts/source/verify.sh) re-hashes the install tree against
that manifest and verifies the manifest's minisign signature against the
root-owned footprint pubkey, yielding states: verified / modified / tampered /
unsigned / unverifiable / development. webuiSystemVerify writes verify_status.json
(throttled daily, force on demand, also after each update apply), surfaced as an
Integrity line + "Verify now" button on the Admin → Overview Updates card and a
row in the update details panel. `libreportal verify` exposes the same check on
the CLI.
Honest framing: this is a self-check (run by the software it verifies), so red
fires only for genuine modified/tampered states; the badge tooltip points to
out-of-band `minisign -Vm` for an independent guarantee.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Spike — closes the gap where the CLI install bypassed the very task system
the WebUI uses. Now both surfaces hit the same path:
user types `libreportal app install dashy`
→ CLI enqueues a task file in $TASK_DIR (identical shape to the
WebUI's createTaskFile)
→ pokes $TASK_DIR/.queue.fifo so the processor dispatches in <100ms
instead of waiting up to IDLE_POLL_SECS
→ CLI tails the task log + polls .status, exits with the task's
exit_code on terminal state
→ Ctrl-C detaches the follower without killing the task — the
WebUI's tasks panel keeps showing it
Bypass: the recursive command in the task file is prefixed
`LIBREPORTAL_TASK_EXEC=1 libreportal app install <name>`. The install
branch in cli_app_commands.sh honours that env var by running inline,
which is what the processor's eval invocation hits. No processor
changes — the bypass travels with the task.
Wins:
- one log file per install, shared by CLI + WebUI (audit trail + replay)
- locking serialises CLI + WebUI installs (no more two-frontend race)
- WebUI's "current task" indicator now reflects CLI work too
- free `--detach` for fire-and-forget queueing
New: scripts/cli/task/cli_task_run.sh
cliTaskRun <cmd> [type] [app] [--detach]
Enqueues + follows; --detach prints the task id and exits 0.
cliTaskFollow <task_id>
`tail -F` the log + jq-poll the status; returns the task's exit_code.
Designed to be reused for `libreportal task log <id>` reattach later.
Trade-off: ~200-500ms latency before the first byte (write task file,
processor wakes, opens log, follower starts tailing). Negligible for
install/update/backup — fast commands (list/status/config get) still
run inline. The current branch only changes `app install`; uninstall +
update + backup can be moved on the same pattern once this lands clean.
Signed-off-by: librelad <librelad@digitalangels.vip>
Finishes the installApp refactor started in d941f59 (Wave A). Every app
whose <app>.sh was either pure boilerplate (Wave B) or boilerplate +
small custom logic (Wave C) now routes through the generic driver in
scripts/app/install/app_install.sh; bespoke logic moved to declarative
hooks in containers/<app>/scripts/<app>_install_hooks.sh.
Net: ~4,000 lines of duplicated 10-step sequence gone. From 31 per-app
.sh files (pre-Wave-A) down to 2 intentional keepers.
DELETED outright (pure boilerplate — driver replaces them identically):
jellyfin, mastodon, focalboard, ipinfo, speedtest, dashy, invidious,
nextcloud, ollama, vaultwarden, pihole
DELETED + hook-extracted (small bespoke step preserved in a hook):
bookstack, moneyapp, owncloud, trilium, searxng, gitea, headscale,
unbound, prometheus, grafana, gluetun, wireguard, jitsimeet, authelia,
traefik, adguard, onlyoffice
KEPT (intentional special cases):
crowdsec — host-app pattern (no docker compose, runs as apt+
systemd via installCrowdsecHost; uninstall/stop/
restart hooks already live in this file and are
invoked by dockerUninstall/Stop/RestartApp directly).
libreportal — WebUI bootstrap. Pre-compose image build + post-install
webuiLibrePortalUpdate + bootstrap-time suppression of
menuShowFinalMessages don't fit the generic flow.
Driver change — scripts/app/install/app_install.sh:
Moved monitoringToggleAppConfig "$app_name" "docker-compose.yml" from
the post-start integrations block into the install body at post-compose
(right after dockerComposeSetupFile, before docker-compose up). The
toggle edits the compose file on disk — running it after start meant
the container had already been brought up with the unmodified compose,
so the metrics endpoint wouldn't reflect CFG_<APP>_MONITORING until
the next restart. Matches the original ordering in every per-app .sh
that used to call it inline.
Hook surface (declare-f-gated, silent no-op when absent):
<slug>_install_pre before any install work
<slug>_install_post_setup after dockerConfigSetupToContainer
<slug>_install_post_compose after dockerComposeSetupFile (+ the
shared monitoring toggle on the compose)
<slug>_install_post_start after dockerComposeUpdateAndStartApp
<slug>_install_message_data echoes extra argv for menuShowFinalMessages
<slug>_install_post very last thing, after the final message
+ the existing _uninstall_pre/_post, _stop_post, _restart_post
Notable extractions:
bookstack — _install_post_start: probe :PORT_1/login until 200/302,
then `bookstack:create-admin` inside the container with
CFG_BOOKSTACK_ADMIN_{EMAIL,PASSWORD}; falls back to the
seeded admin@admin.com on timeout.
adguard — _install_post_start drives the wizard's HTTP API
(POST /control/install/configure) so the admin doesn't
click through five pages, then pins the admin bind back
to 0.0.0.0:3000 (matches the compose mapping) and health
checks. _install_message_data echoes user/password to
menuShowFinalMessages.
authelia — _install_pre requirements; _install_post_compose copies
configuration.yml + users_database.yml, substitutes
theme/domain/host, generates JWT/session/storage secrets,
toggles monitoring on configuration.yml; _install_post_start
argon2-hashes the admin password via the container, writes
users_database.yml, restarts; _install_post echoes creds.
traefik — _install_pre prompts for the LE email if CFG_TRAEFIK_EMAIL
is unset; _install_post_compose copies static + dynamic
configs, wires CFG_TRAEFIK_DASHBOARD_ACCESS (local-only /
domain-only / public), toggles monitoring on traefik.yml,
then traefikUpdateWhitelist + traefikSetupLoginCredentials.
wireguard — _install_pre host-conflict guard (/etc/wireguard/params);
_install_post_compose persists CFG_WIREGUARD_SUBNET,
resolves WG_HOST (domain+traefik → host_setup, else IP),
runs runAppCfg wireguard-ip-forward; _install_post_start
restarts after wg-easy installs its iptables rules.
jitsimeet — _install_post_setup downloads the tagged release zip from
GitHub; _install_post_compose mass-edits the .env and runs
gen-passwords.sh; _install_post_start rewrites nginx
default site to usedport1/2 + restart.
prometheus — _install_post_compose seeds prometheus.yml under
$containers_dir/prometheus/prometheus/; _install_post_start
sets 0777 on storage dirs so the container TSDB can write
regardless of host UID mapping.
grafana — _install_pre requirements; _install_post_start 0777 on
grafana_storage.
gluetun — _install_post_start refreshes the provider snapshot,
reattaches every routed app (the netns container ID is
stale after gluetun gets recreated), then prompts to
onboard any existing apps.
+ the smaller bookstack-shape extractions for owncloud (version scrape),
trilium / searxng (wait-for-first-boot-config), gitea (Prometheus
bearer token sync), headscale / unbound (config copy), moneyapp
(Auth.js AUTH_URL), onlyoffice (compose-resolved user/pass into the
final message).
Manifest + arrays regenerated. Verified end-to-end:
- bash -n on every hook file + the driver: clean
- Each hook file sources cleanly in a subshell, exposes only the
intended functions, flagged lazy-loadable (not eager)
- Smoke-stubbed install run for jellyfin (pure), nextcloud (pure),
bookstack (hooked), crowdsec (kept): correct dispatch in all cases —
deleted apps route to installApp, kept apps still hit their real
function
Signed-off-by: librelad <librelad@digitalangels.vip>
The 31 containers/<app>/<app>.sh files each defined install<App>() with
the SAME 10-step sequence — ~4,000 lines of duplicated boilerplate.
Replaces all that with one generic driver + hook surface.
scripts/app/install/app_install.sh:
installApp <slug> [config_variables]
— Dispatches on $<slug> (c/u/s/r/i) the same way the per-app .sh
files did. Same convention; dockerInstallApp's existing
`declare $app=i` callsite needs no change.
— Runs the standard sequence: dockerConfigSetupToContainer →
dockerComposeSetupFile → optional .env copy → fixPermissions →
dockerComposeUpdateAndStartApp → standard post-install steps
(appUpdateSpecifics, setupHeadscale, databaseInstallApp,
webuiContainerSetup, monitoring registration) → final message.
— Hooks (all declare-f-gated, silent no-op when absent):
<slug>_install_pre / _post_setup / _post_compose / _post_start
<slug>_install_message_data (echoes extra args for menu)
<slug>_install_post
<slug>_uninstall_pre / _post
<slug>_stop_post
<slug>_restart_post
Hooks live in containers/<app>/tools/<app>_tools.sh (auto-sourced
per the modular-per-app-tools convention).
function_install_app.sh:
When no install<App>() function exists, fall through to
`installApp <app_name>` instead of erroring. So an app with no .sh
at all becomes a zero-byte addition — drop in <app>.config +
docker-compose.yml + <app>.svg, done.
containers/linkding/linkding.sh:
Deleted (canary). Linkding's body was 100% standard sequence;
fallback handles it identically. Smoke-tested with stubbed helpers
— dispatcher fires, generic runs full flow, monitoring integration
+ final-message hook plumbing all intact.
Wave B (next): delete the .sh for every other 'pure-boilerplate' app
(~15 candidates per the survey). Wave C: extract custom logic from
the 7 fat apps into hooks before deleting their .sh.
Signed-off-by: librelad <librelad@digitalangels.vip>
`webuiSystemUpdate` was calling `webuiSystemMetrics` and getting
"command not found": the lazy-load manifest was missing it (and
`webuiSystemApps`), even though both are defined in
webui_system_metrics.sh.
Cause: the manifest scanner's awk-based depth tracker is line-based.
The first function in that file, `_metricsReadCpu`, contains an
embedded awk script:
_metricsReadCpu() {
awk '/^cpu /{ <-- the open brace runs to end-of-line, depth += 1
...
}' /proc/stat <-- the close brace is mid-line, NOT counted
} <-- depth-- but we started one too high
The increment rule fired for the `/^cpu /{` line (open brace at EOL),
the matching close on the `}' /proc/stat` line was not counted because
the existing depth>0 branch only decremented when stripped was EXACTLY
`}`. Result: depth ended at 1 instead of 0 after _metricsReadCpu, and
every subsequent funcname() header was treated as "still inside a
function" and silently dropped. Same pattern across the ~6 files the
lazy-loader memory called out as "false-positive eager".
Fix: in the depth>0 branch, count open AND close braces per line
symmetrically and apply the net delta. Drops the redundant "exactly
`}`" special-case rule, clamps depth at 0 as a sanity backstop. False
positives from braces inside string literals could under/over-count
locally, but the clamp + the next legitimate `funcname()` header at
depth-0 re-anchors the tracker.
Result on a full regen:
- 858 → 876 functions indexed (+18 previously-stranded)
- webuiSystemMetrics + webuiSystemApps now correctly autoloaded
- eager-file count: 9 → 11 (two files that genuinely have both
function defs and top-level side effects are now correctly seen
both as eager AND get their functions indexed — net win on
every axis)
Verified live: the WebUI updater that was failing with "command not
found" now prints "Updated system information..." cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
`libreportal app generate <name>` (and the menu's "g. Generate App" entry)
was broken three independent ways and incompatible with the per-app
architecture the project actually uses now:
1. Copies from $install_containers_dir/template/ which doesn't exist —
the only template/ in the tree was in scripts/unused/OLD_CONTAINERS/
and was never installed into the live tree. cp -r would just fail.
2. Every sed call used BSD/macOS syntax `sed -i '' -e …`. On Linux
(every distro this targets) the empty '' becomes a positional file
argument, so the substitutions never ran. 8 calls, all broken.
3. Even if it had run, the produced skeleton would have been a
pre-modular-tools / pre-per-port-subdomain app shape: no tools/,
no scripts/ subdir, HOST_NAME=test in the .config. Every active
containers/<app>/ today carries the modular layout the rest of the
framework expects.
Plus the recent cleanups (the prompt loop fix in 9ffc8e4, the per-port
subdomain refactor in 2e4f420) had been peeling pieces off it without
the root question — does the function still belong? — getting asked.
Delete the whole surface:
- scripts/app/app_generate.sh (157 lines, the function body)
- scripts/unused/OLD_CONTAINERS/template/ (the never-installed source
files appGenerate would have copied — stale enough to still carry
HOST_NAME=test, CFG_<X>_HOST_NAME, and 248 lines of compose template)
- menu entry "g. Generate App" + its dispatch in menu_main.sh
- "generate" case branch in cli_app_commands.sh
- `libreportal app generate` line in cli_app_header.sh
- The corresponding entries auto-drop from files_app.sh +
function_manifest.sh via regen.
New apps are added the way the catalog already grew — by hand-crafting
containers/<app>/{<app>.sh, <app>.config, docker-compose.yml,
tools/<app>.tools.json, scripts/<app>_*.sh}. Copying an existing app's
folder + renaming is the closest thing to a "generator" and it's a one-
command operation.
Net: -556 lines, no behaviour lost (the function never worked).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The Debian-shipped passt AppArmor profile (/etc/apparmor.d/usr.bin.passt)
denies the accesses pasta needs to plumb rootlesskit's netns:
- ptrace_read on the rootlesskit child to enter its user namespace
- read /run/user/<uid>/dockerd-rootless/netns (the netns file)
- read /proc/<pid>/net/{tcp,tcp6,udp,udp6} for implicit port forwarding
Without these the rootless docker daemon fails with:
pasta failed with exit code 1:
Couldn't open user namespace /proc/<pid>/ns/user: Permission denied
scripts/docker/install/rootless/rootless_apparmor.sh:
New installRootlessApparmorForPasta() — idempotent fixup.
1. Adds `include if exists <local/usr.bin.passt>` to the main profile
(one line; re-adding is a no-op via grep).
2. Writes /etc/apparmor.d/local/usr.bin.passt with the four rules
pasta needs. The /local/ pattern is the standard Debian AppArmor
hook for site-managed overrides — survives `apt upgrade passt`
because it's outside the package's managed paths.
3. Reloads via apparmor_parser -r.
Called from installDockerRootless after the override.conf write, gated
on $rootless_net == pasta. slirp4netns installs skip it.
This box was already manually patched while debugging the pasta swap —
the installer-side change makes it idempotent across reinstalls and
applies the same fix on any other host that installs rootless docker
with pasta as the net driver.
Signed-off-by: librelad <librelad@digitalangels.vip>
CrowdSec's host-side install (the agent + nftables bouncer the LibrePortal
Traefik plugin talks to) had stayed on blanket sudo throughout the rootless +
de-sudo hardening: `sudo apt-get install crowdsec`, `curl | sudo bash`,
`sudo sed -i /etc/crowdsec/config.yaml`, `sudo touch + sudo chmod /var/log/
crowdsec*.log`, `echo $key | sudo tee /etc/crowdsec/traefik_bouncer.key`,
plus `sudo cscli capi register / console enroll / bouncers add`. None of
those are in the scoped LP_HELPERS / LP_SYSTEM sudoers grant the manager
now holds, so any user who enabled crowdsec would have hit hard sudo
failures on every privileged step.
Follow the libreportal-appcfg / libreportal-bininstall pattern: one new
root-owned helper at /usr/local/lib/libreportal/libreportal-crowdsec
that does every privileged op behind a fixed action vocabulary with strict
argument validation. The manager calls in via runCrowdsec — the scoped
sudoers grants exactly one binary, the same trust boundary the other
helpers rely on.
Actions:
install apt repo + agent + firewall-bouncer + enable +
crowdsecurity/{linux,sshd} collections + reload
(idempotent — skips parts already in place)
services <verb> enable | disable | restart
capi <verb> register | unregister | status
console <verb> enroll <token> | disenroll | status
token format strictly validated
bouncer-traefik-init cscli register + write the manager-owned key file
atomically (returns EXISTS or GENERATED:<key>)
bouncer-priority bouncer yaml nftables priority → -100
(moved from libreportal-appcfg; one helper for
every crowdsec root op)
bind-lapi flip listen_uri to 0.0.0.0:8080 in config.yaml
prometheus <on…|off> flip the prometheus block (validated addr/port)
touch-host-logs create + chmod 0644 /var/log/crowdsec*.log so the
libreportal container can tail them
Wired in via:
- new sudoers Cmnd_Alias entry for the helper in LP_HELPERS
- new helper baked alongside the others by initRootHelpers
(replaces __SYSTEM_DIR__ / __CONTAINERS_DIR__ / __MANAGER__ at
install, with safe runtime fallbacks if unbaked)
- new runCrowdsec dispatch in scripts/docker/command/run_privileged.sh
containers/crowdsec/scripts/crowdsec_install_host.sh now drives the whole
flow through runCrowdsec — every `sudo …` is gone, the compose-toggle sed
uses runFileOp, and the security_crowdsec CFG mirror uses runInstallOp
(configs/ is manager-owned). Net: install script shrinks ~80 lines while
gaining a single auditable trust boundary. crowdsec_fix_priority.sh swung
over to runCrowdsec bouncer-priority too — the appcfg crowdsec_priority
action drops out cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Tree-wide audit (working tree + deployed install + every local/remote ref
+ every reachable commit + unreachable objects via git fsck) found zero
external callers. Existed dead since v0.1.0 — never wired in.
The function set DOMAINSUBNAME, TIMEZONE, DOCKER_NETWORK (all duplicates
of fills that happen elsewhere) plus the two unique-to-it CONFIGS_DIR_TAG
+ CONTAINERS_DIR_TAG. Those two are already wired directly into the
standard tag-fill block in dockerConfigSetupFileWithData (commit 521f08b),
so dropping the source file leaves no behavioural gap.
Also tighten the comment that explained why we inlined the two tags —
don't reference the function we're deleting in the same change. Describe
the current behaviour, not the history (per repo convention).
Regenerated the auto arrays + function_manifest.sh: the 3 stale entries
referencing this function drop out cleanly. files_cli.sh / files_config.sh
/ files_source.sh also rebuilt — no net content change beyond dropping
this one path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Containers used to be eager-loaded via `sourceScanFiles "containers"`
even under LP_LAZY=1 — sourcing all ~160 installer functions up front.
Phase 5 brings them into the autoload-stub mechanism.
generate_function_manifest.sh now scans BOTH scripts/ AND containers/
(maxdepth 3, matching sourceScanFiles' existing prune), with a per-entry
root selector so stub emission uses the right base directory:
scripts/peer/peer_add.sh → source "${install_scripts_dir}peer/peer_add.sh"
containers/linkding/linkding.sh → source "${install_containers_dir}linkding/linkding.sh"
New manifest exports:
LP_FN_MAP funcname → relpath (existing)
LP_FN_ROOT funcname → scripts|containers NEW
LP_EAGER_FILES "<root>:<relpath>" entries NEW format
~860 autoload stubs (was ~700; +160 from containers)
Loader changes (initilize_files.sh):
- Parses LP_EAGER_FILES entries as `root:path`, dispatches to the
right install_*_dir. Pre-Phase-5 entries without a colon default to
scripts (backwards-compatible).
- sourceScanFiles "containers" is skipped when LP_LAZY=1 AND
LP_FN_MAP is loaded (manifest-driven autoload covers it).
Eager mode and lazy-with-missing-manifest both still run the scan.
Measurement target: ~70 ms saved on top of Phase 4. Verified separately
in the commit message of the next deploy.
Signed-off-by: librelad <librelad@digitalangels.vip>
The previous run had 32 eager files; 24 of them were the auto-generated
files_*.sh arrays (only useful to the eager loader) + the manifest
itself (which the lazy loader sources explicitly). Eager-sourcing them
under lazy mode was pure overhead — ~55ms on the manifest alone (it was
being parsed twice, once via the explicit lazy-loader source and once
via the LP_EAGER_FILES loop).
Down to 8 eager files (the genuinely-side-effecting ones: setup_lock.sh,
the two crontab task processors, backup_db.sh, backup_files.sh,
docker swap_docker_type.sh, migrate_url_rewrite.sh, cli_debug_commands.sh).
The files_*.sh arrays are still sourced by the eager loader's existing
path — that's unchanged. Lazy mode just doesn't need them because it
never iterates files_libreportal_app[@] / files_libreportal_cli[@].
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/loading/initilize_files.sh gains an LP_LAZY=1 branch:
- Sources scripts/source/files/arrays/function_manifest.sh once. The
manifest defines LP_FN_MAP, LP_EAGER_FILES, AND ~700 autoload stubs
(precompiled by the generator — one parse cost vs evaluating 700
snippets at startup).
- Eager-sources every file listed in LP_EAGER_FILES (top-level side
effects: variable assignments, source calls, bare commands). These
can't safely be deferred — they'd skip the side effect, not just the
function definition.
- Skips the bulk loop that sources every files_to_source[@] entry.
Default behaviour (LP_LAZY unset or 0) is byte-identical to the previous
loader — every file gets eager-sourced up front. Long-running processes
(WebUI service, task processor) leave LP_LAZY unset because their first
call to anything wants the function already hot.
Each autoload stub looks like:
funcname() {
source "${install_scripts_dir}path/to/file.sh"
funcname "$@"
}
First call sources the real file, which redefines the function with the
real body; the stub's trailing `funcname "$@"` then calls the freshly-
defined real implementation. Sourcing the file also redefines stubs for
any sibling functions the same file declares, so they don't re-source.
Safety nets:
- Missing manifest → fall back to eager loading (`export LP_LAZY=0`).
No regression risk if someone enables LP_LAZY=1 on a stale install
whose regen never ran.
- LP_LOAD_TRACE=1 still works in lazy mode — it records the manifest
parse + each eager file (tagged LAZY-manifest / LAZY-EAGER) so Phase 4
can measure the actual saving.
No automatic flip yet — this commit only adds the path. Phase 4 will set
LP_LAZY=1 by default for the CLI entrypoint (and re-measure with the
trace tool from Phase 1).
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/files/generate_function_manifest.sh — scans every .sh in
scripts/ (skip-list matches generate_arrays.sh, plus excludes peer_shell.sh
which is a standalone forced-command target) and emits
scripts/source/files/arrays/function_manifest.sh:
declare -gA LP_FN_MAP=(
[acquireSingletonLock]="crontab/task/crontab_task_processor.sh"
[adoptDockerSubnet]="checks/requirements/check_docker_network.sh"
... # 698 entries
)
LP_EAGER_FILES=(
"backup/db/backup_db.sh"
"source/files/arrays/files_app.sh"
... # 32 entries (~7% of files)
)
The lazy loader (Phase 3) consumes LP_FN_MAP to install autoload stubs of
the form `name() { source "$LP_FN_MAP[name]"; name "$@"; }`. First call
sources the real file, which redefines the stub with the real body;
subsequent calls hit the real one. LP_EAGER_FILES enumerates files with
top-level side effects (variable assignments, source calls, bare commands
outside any function) — those MUST always source so the side effects fire.
Heuristic correctness, in order of importance:
- Function header detection requires EMPTY parens (`name()`), not just
`name(` — otherwise lines like `if (...)`, `for (...)`, `while (...)`
in embedded awk/perl get misread as bash function defs.
- Handles three function styles: `name() {` (same line), `name()\n{`
(LibrePortal convention), and one-liners `name() { body; }`.
- Tracks { } balance for inside-function depth, with the safe fallback that
ambiguous cases get marked eager (false positive = no behaviour change;
false negative would skip a needed source).
- Files containing embedded awk/perl with their own { } blocks (about 6 of
them: cli_debug_commands.sh, crontab_task_processor.sh, backup_db.sh,
backup_files.sh, etc.) get false-positive flagged eager — acceptable
because they just stay eager-loaded, matching today's behaviour.
- Collisions report to stderr (last-write wins, same as eager-load
semantics); no collisions found in the current tree.
Wiring:
- lpRegenArrays (`libreportal regen arrays`) now also runs the manifest
generator when the existing arrays need regen, keeping the two in sync.
- update.sh's quick-deploy regen step does the same after copying files
to the live install. Best-effort: failures don't abort because lazy
loading is opt-in (LP_LAZY=1) in Phase 3 and not the default yet.
Scanned: 454 files, indexed 698 function definitions, 32 eager (9 real,
23 auto-generated arrays + the manifest itself). 0 name collisions.
No behaviour change in this commit — the manifest is just data the loader
in Phase 3 will use. The default eager loading path is untouched.
Signed-off-by: librelad <librelad@digitalangels.vip>
Linkding has shipped without any Tools-tab actions since v0.1.0 — the only
artifact was scripts/menu/tools/manage_linkding.sh, a dead legacy CLI menu
referencing an `appLinkdingSetupUser` function that was never defined.
Build the real thing, mirroring bookstack's pattern (manifest + thin tool
wrappers + auth_adapter that drives the app's native admin shell):
containers/linkding/tools/linkding.tools.json — manifest, 5 tools
containers/linkding/tools/linkding_<id>.sh — one wrapper per tool
containers/linkding/scripts/linkding_auth.sh — Django shell driver
Tools (all category=users, so the WebUI's custom user-list panel and its
row-level 🔑 / 👑 / 🗑 buttons light up):
reset_password — set_password on an existing user (random if blank)
create_account — create_user / create_superuser
list_users — emits EZ_USER\t<username>\t<username>\t<role> rows
(linkding is username-primary, so username goes into
both display slots — keeps the panel click-through
identifier consistent with the other tools' fields)
delete_user — delete by username (destructive, confirm gated)
set_admin — toggle is_superuser + is_staff
Implementation runs entirely inside the linkding-service container via
`runFileOp docker exec ... python manage.py shell -c "<code>"`, reading
inputs through `-e` env vars so quoting stays safe. Django's default
get_user_model() User is used directly — passwords hash exactly the way
the web UI does, admin flags map to the same fields the UI reads.
Also drop the dead legacy stub (scripts/menu/tools/manage_linkding.sh)
and regenerate files_menu.sh so the source-scan no longer pulls it in.
Nothing referenced linkdingToolsMenu — verified by tree-wide grep.
Verified live on dev-ai (Debian 12, linkding installed, Django 5 + sqlite):
$ libreportal app tool linkding create_account 'username=alice|password=…|admin=true'
✓ Linkding user created — Username: alice — Password: …
$ libreportal app tool linkding list_users ''
EZ_USER alice alice admin
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Polish pass for the migration system. Two concrete additions; the live-mirror
and full drift-verify ideas from the original plan are intentionally
deferred — both need real-world test data to land correctly, and the kernel
already exposes everything they'd need.
Per-app migrate hooks (scripts/migrate/migrate_hooks.sh):
Apps can declare two optional functions in their tools.sh (already
auto-sourced per [[libreportal-modular-app-tools]]):
<app>_migrate_pre() — runs before stop+wipe
<app>_migrate_post() — runs after restart, before the user sees it
Each receives:
$1 = source identifier (peer name or backup-tag hostname)
$2 = transport ("restic" | "direct-ssh")
migrateRunHook() is now called from both migration apply paths:
- migrate_apply.sh (restic-mediated, shared backup channel)
- peer_pull.sh (direct-SSH, peer-shell stream)
Use cases: rotate federation keys after a Mastodon move, regenerate
OIDC client secrets, drop SaaS-style locks, fix hostname-baked configs
the URL-rewrite layer doesn't cover.
Hooks are optional — apps without them inherit the standard flow.
Failed hooks emit a non-fatal notice (the rest of the migrate still
reaches 'done') so a single bad hook can't strand an otherwise-working
app in stopped state.
Peer friendly-name overlay (Migrate tab):
Was deferred from Phase 2 because it required Phase 3's UI to feel
cohesive. BackupPage.refreshAll() now also fetches peers.json and builds
a hostname → peer-name lookup. renderMigrate() shows
'homelab (host: homelab.lan)'
for any backup-channel peer that matches the source host, and falls back
to the bare hostname when no peer is defined. Same data, friendlier UI.
Skipped (genuinely deferred, not just out of time):
- Live mirror / warm-standby (continuous one-way sync). Needs a scheduler
+ drift-state to track. Right place for it is a separate feature on top
of the existing kernel rather than bolted onto migrate.
- Drift-verify ("what would change if I migrated?"). Cheap to write but
needs a real cross-host pair to validate against — adding it untested
would just be theatre.
Signed-off-by: librelad <librelad@digitalangels.vip>
End-to-end direct-ssh-direct: two LibrePortal instances exchange pairing
tokens, each authorizes the other to call a locked-down peer-shell dispatcher
via SSH forced-command, then either side can pull live app data from the
other without needing a shared backup repo.
Push and Connect-via-relay are deferred — push is symmetric to pull (same
forced-command, opposite verb), and the relay variant waits for Connect to
actually exist (config_json + kind enum already future-proofed in Phase 2).
Key generation (peer_key.sh):
One ed25519 keypair per install at ~<manager>/.ssh/libreportal-peer{,.pub}.
Generated lazily on the first peer-related call. Used as our outbound
SSH identity AND as the pubkey other instances authorize.
Forced-command dispatcher (peer_shell.sh):
Standalone script, deployed by peerInstallShell() to
~<manager>/.local/bin/peer-shell. authorized_keys entries look like:
command="~/.local/bin/peer-shell <peer-name>",no-pty,no-port-forwarding,
no-X11-forwarding,no-agent-forwarding,no-user-rc ssh-ed25519 AAAA… peer:<name>
sshd hands us $SSH_ORIGINAL_COMMAND; we parse, whitelist the verb, and
refuse anything else. Verbs:
ping Liveness probe (JSON ok:true).
list-apps JSON {peer, apps:[{slug, size_kb}]}.
stream-app tar of containers_dir/<slug> to stdout (slug strictly
validated — lowercase alnum+dash; rejects path traversal).
Audit log appended to ~/.local/state/libreportal/peer-shell.log. Excluded
from the generated source arrays (would crash any sourcing shell on empty
SSH_ORIGINAL_COMMAND); generate_arrays.sh skip-list extended.
Pairing token (peer_pairing.sh):
Format: lp-peer|v1|<name>|<user>|<host>|<port>|<base64-pubkey>|<fingerprint>
Pipe-delimited because the SHA256 fingerprint and base64 pubkey both
contain ':'. peerPairingParse decodes + re-derives the fingerprint from
the actual key, refusing tokens with mismatched fingerprints (catches
truncation / tampering). peerPairingAccept:
1. Installs peer-shell (peerInstallShell).
2. Appends to authorized_keys with the lockdown options above.
3. Inserts a peers row (kind=direct-ssh-direct, config carries host,
port, user, fingerprint).
Symmetric — user runs accept on BOTH sides with the other's token to
enable bidirectional calls.
Outbound SSH (peer_remote.sh):
peerExec <name> <verb> [args] — looks up the peer's connection config and
ssh's in with the right key, BatchMode + ConnectTimeout + accept-new for
the host key. peerPing wraps it and updates peers.status + last_seen.
Pull-an-app (peer_pull.sh):
peerPullApp <peer> <app> [--no-pre-backup] [--keep-urls]
1. peerPing (refuse if unreachable).
2. migratePreBackupDestination (reuses the Phase 0 safety wrapper —
same restic-tagged pre-migrate snapshot as the backup-channel flow).
3. Stop + wipe destination's app folder.
4. peerExec stream-app | tar -x (pipefail; bails on partial transfers).
5. migrateApplyUrlRewrite + dockerComposeUpdateAndStartApp install
(URL repointing, idempotent install path).
6. dockerComposeUp + post-restore hooks.
Identical Stage-2..6 to migrateApplyApp; only the data source differs
(tar-over-SSH instead of restic-restore).
CLI (cli_peer_commands.sh + header):
libreportal peer token — emit this host's pairing token
libreportal peer pair <token> [name] — accept a token (override name)
libreportal peer apps <peer> — live peer-shell list-apps
libreportal peer pull <peer> <app> [--no-pre-backup] [--keep-urls]
WebUI (/peers):
Header gains 'Show my token' and 'Pair with token' buttons (both open
modals around the matching CLI verbs). Token modal warns the user that
the token is credentials. Pair modal accepts a free-form override name.
Direct-SSH peer cards gain a 'List apps' button that opens an inline
drawer showing the peer's live app inventory (via peer apps) with per-
app 'Pull' buttons. Pull modal has the same two safety toggles as the
Migrate tab (pre-backup ON, URL rewrite ON by default).
Backup-channel manual-add modal kept; direct-SSH must use the token flow.
Smoke-tested:
- All 16 peer-subsystem functions register without crashing the shell.
- peer-shell ping ⇒ {ok:true}; unknown-verb refused; path-traversal slug
refused; valid-slug streams.
- Token emit→parse round-trip preserves every field; garbage rejected
with not-a-token; v99 rejected with unsupported-version.
Signed-off-by: librelad <librelad@digitalangels.vip>
A peer is a named reference to another LibrePortal instance. Phase 2 only
implements kind=backup-channel (friendly label over a hostname that shows
up in a shared backup repo); direct-ssh-direct and direct-ssh-via-relay
(Connect's blind-relay) are reserved enum values for Phase 3.
DB schema (db_create_tables.sh):
CREATE TABLE peers (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT UNIQUE NOT NULL,
kind TEXT NOT NULL DEFAULT 'backup-channel',
config_json TEXT NOT NULL DEFAULT '{}',
status TEXT DEFAULT 'unknown',
last_seen TEXT,
created_at TEXT DEFAULT CURRENT_TIMESTAMP
);
+ indexes on name and kind.
config_json is kind-specific so new transports don't need a schema
migration. For backup-channel it carries {"hostname":"","loc_idx":N}.
Bash module (scripts/peer/):
peer_helpers.sh _peerDb, peerSqlEscape, peerValidateName/Kind.
peer_add.sh peerAdd <name> <kind> [k=v ...] → INSERT, refresh
generator. Rejects unimplemented kinds early so users
don't create dead-end peer records.
peer_remove.sh peerRemove <name> → DELETE.
peer_list.sh peerList → JSON array; peerGet, peerNameForHostname
(reverse-lookup for the migrate-tab overlay).
peer_check.sh peerCheckReachable, peerCheckAll. For backup-channel
'reachable' = at least one snapshot from that hostname
visible in (preferred|any enabled) location. Updates
status + last_seen so UI dots render without re-probing.
CLI (scripts/cli/commands/peer/):
libreportal peer list
libreportal peer get <name>
libreportal peer add <name> backup-channel hostname=<host> [loc_idx=<n>]
libreportal peer remove <name>
libreportal peer check [name]
Auto-routed by cli_initialize.sh's category-discovery.
WebUI data generator (scripts/webui/data/generators/peers/webui_peers.sh):
Emits data/peers/generated/peers.json with the peerList output and a
generated_at envelope. Hooked into webuiLibrePortalUpdate alongside the
backup generators.
Frontend:
- New top-level /peers route in spa.js (PeersPage class, peers-content.html).
- 'Peers' nav item in the topbar between Backups and the right-side controls.
- Add-peer modal with friendly-name + kind + hostname + preferred-location
selector (populated from the existing backup-locations data).
- Per-peer card with status dot, last-checked time, Check + Remove buttons.
- Phase 3 kinds appear in the kind dropdown as disabled options so users
can see what's coming.
Source-array wiring:
- generate_arrays.sh auto-created files_peer.sh from the new peer/ dir.
- cli_files.sh + app_files.sh include ${peer_scripts[@]} alphabetically.
- files_webui.sh auto-picked-up the new peers/ generator subfolder.
The migrate-tab friendly-name overlay (use peer names in /backup/migrate
when a peer record exists for a hostname) is intentionally deferred — it's
a 5-line frontend lookup once peers.json is loaded; cleaner to add after
Phase 3 ships its peer-detail view.
Signed-off-by: librelad <librelad@digitalangels.vip>
Phase 1 of the migration-system refresh. Surfaces Phase 0's kernel
(libreportal restore migrate ...) as a WebUI flow so users don't have
to drop to the CLI to pull an app from a peer's backups.
backend / data generator:
scripts/webui/data/generators/backup/webui_backup_migrate.sh
Walks every enabled backup location, lists every (other_host, app)
pair with snapshot count + latest id/date, and emits a single
destination summary block (installed apps, running apps, disk free)
so the frontend can compute collisions and warnings without per-row
API round-trips. Filters out our own hostname — we don't migrate to
ourselves. Output: data/backup/generated/migrate.json.
Hooked into the standard webuiLibrePortalUpdate refresh pipeline,
so 'libreportal regen webui' (and the periodic task-processor poll)
keep it fresh on their own.
frontend:
- New 'Migrate' sidebar tab on /backup, sits between Locations and
Configuration. Path-based URL: /backup/migrate.
- Per-source-host cards listing every available app, with snapshot
count + relative-time hint, collision dot when the app is already
installed here, and per-app + per-host migrate buttons.
- Confirm modal with two checkboxes matching the kernel's defaults:
[✓] Back up the destination's existing copy first (pre-migrate
backup; auto-disabled when there's nothing to back up)
[✓] Rewrite host-bound URLs to this host (URL rewrite
— uncheck only to keep source hostnames)
On confirm, runs 'libreportal restore migrate app/system …' via the
task system; opt-out checkboxes append --no-pre-backup / --keep-urls
only when the user un-ticks, matching the kernel's default-on flags.
- Empty state when no other hosts have visible backups, explaining
the shared-backup-location prerequisite.
The CLI dispatcher hooks (Phase 0) wire restore migrate app/system to
migrateApplyApp/migrateApplySystem, so the WebUI gets pre-backup safety,
URL rewrite, and structured progress (when --json-progress is set; not
needed here yet — the task system's log tail is enough for v1).
Signed-off-by: librelad <librelad@digitalangels.vip>
Phase 0 of the migration-system refresh. Replaces the 77-line
scripts/migrate/ with a properly-shaped kernel that Phase 1 (WebUI) and
Phase 3 (direct peer SSH) can both build on.
New module layout (6 files):
migrate_progress.sh — migrateEmit JSON-per-line helper; opt-in via
MIGRATE_JSON_PROGRESS=1, writes to fd 3 if open
(clean WebUI streaming channel) else stdout.
migrate_discover.sh — migrateDiscoverHosts / migrateDiscoverApps /
migrateDiscoverAppDetail (JSON {snapshots, latest_*}).
Old migrateDiscoverAppsForHost kept as back-compat.
migrate_preflight.sh — migratePreflight emits one JSON object with
snapshot{id,date}, destination{installed,running,
disk_free_kb}, collision{occurs,default_action,
pre_backup_default}, url_rewrite{default_action,
per_app_opt_out}, warnings[], errors[].
Exit 0 on usable preflight, 1 on hard error.
migrate_url_rewrite.sh— Host-bound CFG_<APP>_* fields (URL/HOST/DOMAIN/
DOMAIN_PREFIX/HOSTNAME/PUBLIC_URL) get rewritten
from the destination's install-template after
restore — so a moved app stops claiming the
source's hostnames. Per-app opt-out via
CFG_<APP>_MIGRATE_URL_REWRITE=false. All other
fields (DB passwords, API keys, prefs) carry
over from the source unchanged.
migrate_pre_backup.sh — migratePreBackupDestination takes a snapshot of
the destination's existing <app> (tagged
pre-migrate=<UTC timestamp>) before the wipe.
Default ON; opt-out with --no-pre-backup. Safety
net for the always-replace collision policy.
migrate_apply.sh — migrateApplyApp / migrateApplySystem. Parses
--no-pre-backup / --keep-urls / --json-progress
opts, runs preflight → pre-backup → restoreAppStart
(existing flow) → URL rewrite → re-deploy compose.
migrateApp / migrateSystem kept as shims so the
old CLI surface still works.
CLI dispatcher (cli_restore_commands.sh + cli_restore_header.sh):
Existing 'restore migrate app/system/discover' calls all still work.
New verbs:
restore migrate list <host> [loc_idx]
restore migrate preflight <host> <app> [loc_idx] ← JSON, for the WebUI
Design choices baked in (per the spec):
- Always-replace collision (no multi-install of an app), safety net is the
on-by-default pre-migrate backup.
- URL rewrite by host-bound suffix list, not per-field allowlist — works
out-of-the-box for new apps without extra config.
- migrateEmit fd-3 contract is what Phase 1's WebUI will stream; falls
back to stdout in interactive CLI so dev/debug just works.
- Transport-agnostic: nothing in this kernel knows whether the backup
location is local/SSH/S3/Connect — engineSnapshotsJson + engineBackupApp
do that, so Connect (the future blind-relay) plugs in as 'just another
location kind' with zero kernel changes.
Smoke-tested: all 13 public functions register; JSON emit produces correct
escaping (quoted strings vs bare numerics) and respects MIGRATE_JSON_PROGRESS.
Signed-off-by: librelad <librelad@digitalangels.vip>
The migrate/ helpers were either uncallable or no-ops:
- migrateAppsToNewNetwork + updateComposeFileNetwork: never called from
anywhere. The intended sed-on-compose subnet rewrite would also have
fought the tag system / network_resources DB.
- checkAppNetworkCompatibility: called from updateDockerNetworkConfig as
a gate, but never explicitly returns, so it's effectively always-true
and both branches do the same work. Pure noise.
- getInstalledApps: only used by the above.
- updateDockerNetworkConfig: collapses to a 2-line 'CFG := docker's
reported subnet' adoption — inlined into check_docker_network.sh as
adoptDockerSubnet(), which is what it actually does.
The legitimate 'subnet changed, refresh apps' path is already covered by
the idempotent per-app reinstall (dockerInstallApp ... reset_network=true
→ clears DB allocations → installer re-runs → ipUpdateComposeTags picks
fresh IPs from the current CFG_NETWORK_SUBNET). Migration (infrastructure
regen) vs restore (data) stays clean: reinstall regenerates compose+IPs,
restore lays data on top. No new pathway needed.
Files dropped:
scripts/docker/network/migrate/migrate_apps_to_new_network.sh
scripts/docker/network/migrate/migrate_check_app_network_compatibility.sh
scripts/docker/network/migrate/migrate_get_installed_apps.sh
scripts/docker/network/migrate/migrate_update_compose_file_network.sh
scripts/docker/network/migrate/migrate_update_docker_network_config.sh
Plus the now-empty migrate/ subdir; files_docker.sh regenerated to drop
the references.
Signed-off-by: librelad <librelad@digitalangels.vip>
If it's gluetun code, it lives with gluetun. Both functions in
scripts/config/tags/processors/tags_processor_network_mode.sh manipulate gluetun
markers / gluetun's compose, so move them into containers/gluetun/scripts/
gluetun_network.sh and rename to the per-app-hook convention:
tagsProcessorNetworkMode -> appNetworkApplyMode_gluetun
tagsProcessorGluetunForwardedPorts -> appNetworkRegisterPorts_gluetun
Central call sites are now provider-agnostic — no "gluetun" literal anywhere:
- docker_config_setup_data.sh: an app routing via CFG_<APP>_NETWORK=<provider>
triggers `appNetworkApplyMode_<provider>` + `appNetworkRegisterPorts_<provider>`
via declare -F, so any future gateway provider plugs in with no engine edits.
- uninstall_app.sh: loops every `appNetworkRegisterPorts_*` hook (each self-skips
when its provider isn't installed), so removing a routed app refreshes the
right provider with no provider name in central code.
Delete tags_processor_network_mode.sh; regenerate arrays. Verified with stubs:
default mode no-ops, gluetun-routed app fires both hooks, gluetun itself is
skipped, unknown provider is silently no-op, uninstall loop calls registerPorts.
Drive-by cleanup: 9 stale "${X_scripts[@]}" array references in app_files.sh /
cli_files.sh (gluetun + headscale from this session's moves, plus 7 pre-existing:
command/ssl/swapfile/ufw/ufwd/user — all from older refactors that left them
behind). Each expanded to nothing at runtime (harmless), but they're dead
misleading refs. Cleaned both files; every remaining array ref now points to a
real files_*.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
dockerCheckAllowedInstall was a one-app `case` whose only active caller was the
wireguard app itself — so inline its check (abort if a host WireGuard exists at
/etc/wireguard/params, which would collide on the wg kernel module + UDP 51820)
directly into containers/wireguard/wireguard.sh and delete
scripts/docker/app/checks/allowed_install.sh.
The protection is unchanged; wireguard is now fully self-contained and the last
app name leaves central install code. Regenerated arrays. (The only remaining
dockerCheckAllowedInstall references are in scripts/unused/ — retired apps,
never sourced.)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
(a) Docs: reserve tools/ scripts/ resources/ as LibrePortal folder names (apps must
not bind-mount to them); document resources/ as the home for nest-able data AND for
.sh payloads that execute on load (vs scripts/ for sourced functions); document the
backup model (what's captured vs reproducible).
(b) System-config backup so a bare-metal restore is self-sufficient — this is why
the system root is its own tree. New scripts/backup/system/backup_system.sh:
- backupSystemConfig snapshots <system>/configs (global settings, WebUI creds, and
the BACKUP-LOCATION creds — otherwise the keys to reach your own backups live only
on the box) to every enabled location. Lightweight static-dir snapshot — it does
NOT go through backupAppStart (no containers to quiesce / DBs to dump).
- restic adapter resticBackupSystemToLocation (tag system=config) + dispatcher
engineBackupSystem; restore via resticRestoreSystemLatest / engineRestoreSystemLatest
+ backupRestoreSystemConfig (restores to a STAGING dir — never auto-overwrites
live config).
- backupAllApps runs it after the app loop.
WebUI exclusion: backupAllApps skips the 'libreportal' app — its frontend + generated
JSON regenerate, and its only state (the login) is in the system config now captured
above. Nothing in its data dir warrants a snapshot.
Verified with stubs: app loop skips libreportal + invokes the system backup; the
system backup dispatches to both locations; backup/restore function names pair with
the dispatcher. NOTE: restic-only (the sole live engine adapter); end-to-end repo
round-trip still needs a live box before being relied on.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Move the whole central scripts/headscale/ tree into containers/headscale/, the
last app-specific dir living centrally:
- 11 sourced function files (incl. the former local/ remote/ subdirs) flattened
into containers/headscale/scripts/ — flat because the container scan is
maxdepth 3, so one subfolder level is the limit; basenames already encode the
local/remote distinction.
- tailscale.sh is a CONTAINER PAYLOAD (ends in a bare `install_tailscale` call,
runs apt/curl) — it must never be sourced into the manager, so it goes to
containers/headscale/resources/ (pruned by the scan), NOT scripts/. Verified
install_tailscale does not leak into the runtime after sourcing.
- Fix tailscaleInstallToContainer to copy the payload from its new resources/
path (it previously referenced ${install_scripts_dir}tailscale.sh, which never
matched the file's actual location) and drop the dead commented docker-cp line.
- Remove the now-moot headscale special-case from generate_arrays.sh; regenerate
(files_headscale.sh drops — headscale is fully container-scanned now).
All 11 functions source + define cleanly; callers resolve by name regardless of
location.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Move scripts/webui/data/generators/apps/webui_gluetun_providers.sh ->
containers/gluetun/scripts/gluetun_providers.sh and replace the gluetun-specific
gated call in webui_updater.sh with a generic per-app loop: an installed app may
define appWebuiRefresh_<app> (in its scripts/) for data it wants refreshed on
every WebUI update. gluetun provides appWebuiRefresh_gluetun (a thin wrapper over
webuiGenerateGluetunProviders).
- No gluetun-specific code remains in central WebUI code — it's a true drop-in.
- Install gate preserved + generalized: the loop iterates the manager-owned
install templates (listable) and tests each app's live compose directly (works
without list perm on the container-user data dir), so non-users never pay for it.
- webuiGenerateGluetunProviders keeps its name (still called by the installer and
the gluetun_refresh_providers tool); now sourced via the container scan.
- Regenerate arrays (generator drops out of files_webui).
Loop verified with stubs: only installed apps with a defined hook fire; apps
without a hook are skipped; nothing fires when nothing's installed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Replace the central app-name if-ladder in app_update_specifics.sh with a generic
dispatcher: each app ships containers/<app>/scripts/<app>_update_specifics.sh
defining appUpdateSpecifics_<app> (live-sourced by the container scan, dispatched
by `declare -F` — same pattern as tools). A hook may set shouldrestart=true. Apps
with no specifics ship no hook.
- Move the adguard/pihole (DNS updater), dashy (conf refresh), focalboard (nobody
ownership + restart), and libreportal (webui regen) branches to per-app hooks.
- Move scripts/gluetun/gluetun_route_apps.sh -> containers/gluetun/scripts/
(scripts/gluetun/ removed).
- Move scripts/install/install_crowdsec.sh -> containers/crowdsec/scripts/
crowdsec_install_host.sh; fix the path note in crowdsec.sh.
- Regenerate arrays (moved files drop out; the per-app files are container-scanned,
not arrayed).
Dispatch verified with stubs: adguard/pihole/dashy/focalboard/libreportal behave
identically to the old ladder (incl. shouldrestart propagation), apps without a
hook are a clean no-op. The CLI itself had no per-app branches — app-specific CLI
is already the (now fully modular) tools system.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Add `lpRegen` (scripts/webui/webui_regen.sh) — one entry point that rebuilds the
file-derived artifacts whose sources changed, so callers don't have to know which
generator owns what. Self-heal is a cheap `find -newer` mtime compare (no watcher
/ daemon): a stage runs only when a source is newer than its artifact, or --force.
- `libreportal regen [all|webui|arrays] [--force]` CLI command (new category).
- Task processor idle tick runs a throttled `regen webui` poll, so an app dropped
in out-of-band (drag-drop / marketplace) appears on its own — no manual command,
no inotify (works on the relocatable/external-drive roots where inotify can't).
- make_release.sh guards against shipping stale source arrays (regenerate; abort
if the committed tree was out of date), killing the "forgot generate_arrays" bug
class at the build boundary.
- Document the front door in DEVELOPMENT.md.
webui scope rebuilds from containers/<app>/{*.config,tools/*.tools.json}; arrays
scope from scripts/** (a dev/build concern — a no-op on a normal install). Gate
logic verified in a sandbox (clean/config-newer/tools-newer/force/missing).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Each app now carries everything under containers/<app>/: Tools-tab actions in
tools/ (declaration <app>.tools.json + function <app>_<tool_id>.sh) and logic
helpers in scripts/ (e.g. <app>_auth.sh). The container scan live-sources every
.sh under the app (maxdepth 3, prunes only resources/) and webui_tools.sh
auto-merges the .tools.json, so an app is a true drop-in — no central edit, no
array regen.
- Empty the central webui_tools.sh heredoc; all 34 tools across 11 apps now
come from per-app declarations (verified byte-identical to the old output).
- Retire the orphaned mattermost tool scripts to scripts/unused (there is no
containers/mattermost; its install fn already lived in unused).
- Update the dispatch comment/error path, the auth-adapter doc, and
DEVELOPMENT.md to the new convention.
- Regenerate static arrays (files_app.sh no longer lists app/containers/*).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Establish the self-contained tools convention and prove it on a core app:
- discovery now reads containers/<app>/tools/<app>.tools.json (the tools/ subfolder);
tool functions live at containers/<app>/tools/*.sh, auto-sourced by the container
scan (depth 3) — no scripts/app/ entry, no array regen.
- adguard migrated: its 2 Tools-tab actions (reset_password, apply_dns_updater) moved
to containers/adguard/tools/ + tools/adguard.tools.json, and dropped from the
central webui_tools.sh heredoc. adguard_auth.sh stays in scripts/app/ — it's a logic
helper, NOT a tool (the key distinction: only DECLARED tools move).
Central + per-app styles coexist (pihole etc. still central), so the remaining apps
can migrate one at a time with nothing breaking. Verified: heredoc valid sans adguard,
per-app merge re-adds adguard's 2 tools, scripts array dropped the moved fns.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
getlibreportal (downloads host) + weblibreportal (website) — including the website
Eleventy source and the publish tool functions — now live in the separate
LibrePortal-Infra repo (Webstar/LibrePortal-Infra). They're the project's own
outward-facing hosting, not something users install, so the base stays clean.
Removed from base: containers/{getlibreportal,weblibreportal}, the
scripts/app/containers/<app>/<app>_publish.sh tool functions, and their entries in
webui_tools.sh; regenerated the sourced-file arrays; dropped the dead .gitignore
docroot lines. scripts/release/make_release.sh stays here (it builds the base
release). docs/DEVELOPMENT.md now points publishing at LibrePortal-Infra.
LibrePortal-Infra overlays onto an install and picks up releases/catalogue from the
base tree — see its README.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Surface the publish step through the existing Tools system (apps-tools.json -> Tools
tab + 'libreportal app tool <app> publish'), so the docroot can be (re)built from
the WebUI instead of a manual cd + script.
- webui_tools.sh: declare a 'publish' tool (no inputs) for getlibreportal + weblibreportal.
- scripts/app/containers/getlibreportal/getlibreportal_publish.sh (appGetlibreportalPublish):
runs the host's publish.sh into the served data dir, as the container user (owns it).
- scripts/app/containers/weblibreportal/weblibreportal_publish.sh (appWeblibreportalPublish):
builds Eleventy as the manager (owns the install tree), then syncs the result into
the container-user-owned docroot — handling the build-vs-write owner split.
- Both guard for the build prerequisites (repo source / npm / dist) and fail with a
clear message; regenerated the sourced-file arrays.
Honest status: scaffolding only — wiring verified (dispatch names match, files sourced,
JSON valid) but the end-to-end tool RUN is untested, and it's build-box-only (needs the
repo checkout + npm + a built dist/). These hosting apps are dev-only and headed for a
separate repo; this just sets the automation up so it's ready to iterate on.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Adding scripts/release/ made generate_arrays emit files_release.sh, which would
source the build tool make_release.sh at runtime (it runs git archive at load —
wrong + harmful). Skip release/ in the folder scan (like system/ and unused/),
remove the orphaned files_release.sh, and regenerate the meta-list. fetch.sh stays
sourced; make_release is in no array.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/fetch.sh (sourced at runtime via files_source.sh):
- lpFetchRelease [ver]: resolve channel manifest -> download tarball -> VERIFY
sha256 (refuse on mismatch/absence) -> replace the install tree ( is
code-only now; configs/logs live in the separate system tree, so no backup
dance). Host/channel from LP_RELEASE_BASE_URL/CFG_RELEASE_BASE_URL + channel.
- lpFetchSource: dispatch release|git|local.
- lpVersionGt: numeric dotted semver compare (used by the updater + badge).
init.sh initGIT is now release-aware: the bootstrap (install.sh) stages+verifies
the code and sets LP_ALREADY_FETCHED=1 (skip re-fetch); a direct release run sources
fetch.sh; a bare /root reinstall is directed to install.sh. install.sh exports
LP_ALREADY_FETCHED + LP_RELEASE_BASE_URL on hand-off. validateUnattended already
accepts release (git-url is gated on git mode).
Config: CFG_INSTALL_MODE default -> release, + CFG_RELEASE_BASE_URL / CFG_RELEASE_CHANNEL
(add-only reconcile preserves existing installs' git/local mode).
Verified: lpVersionGt across cases; lpFetchRelease downloads+verifies+extracts a
clean tree against a local server. The updater + reset/reinstall release paths are
phase D.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
CFG_DOCKER_MANAGER_USER / installDockerManagerUser was a chrooted SFTP file-access
user — unrelated to the LibrePortal control-plane manager (sudo_user_name), and
the source of the 'two managers' confusion. It was permanently-off dead code: the
gate CFG_DOCKER_MANAGER_ENABLED and the CFG_DOCKER_MANAGER_USER/_PASS keys are
defined in no config template, so it never ran. Its SSH-key-management sibling
(unused/ssh_manager.sh) was already retired; admin host SSH access is handled by
the current /ssh page + scripts/ssh/host_access.sh.
Move install_user_manager.sh / uninstall_user_manager.sh / check_manager.sh to
scripts/unused/manager/ (recoverable, matches the graveyard convention — not
deleted, in case the SFTP-user idea is rebuilt cleanly later), drop the two call
sites (start_preinstall.sh, check_requirements.sh), regenerate the arrays.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Introduce scripts/source/paths.sh as the canonical path resolver for three
independently-relocatable roots:
LP_SYSTEM_DIR manager-owned control plane (configs/logs/install/db/ssl/ssh/migrate)
LP_CONTAINERS_DIR container-user-owned live app data
LP_BACKUPS_DIR container-user-owned backup repos (own mount-able)
Roots come from the environment when set (install bakes them; CLI/app inherit
from init.sh), else default to /libreportal-*. A transitional compat default
keeps EXISTING installs (legacy single /docker tree, by config marker) on /docker
until a deliberate reinstall, so deploying this never strands a running box.
- init.sh derives the same vars inline (self-contained for the bare /root/init.sh
reinstall case); paths.sh mirrors it for the standalone task/check processors,
which now self-locate their scripts dir and source it.
- Replace functional /docker literals with the derived vars across runtime,
install, backup, crontab, crowdsec/restic, headscale, and reinstall paths;
clean the inert '== /docker/containers/*' guard fallbacks to the variable form.
- backend: CONTAINERS_DIR now from LP_CONTAINERS_DIR (compose env, filled at
generation via a new CONTAINERS_DIR_TAG), legacy-safe default for un-recreated
containers.
- backup default path falls back to the backups root; exclude paths.sh from the
sourced-file arrays (bootstrap file, sourced explicitly).
The CLI-wrapper heredoc + root helpers still reference /docker; those get baked
in phase 3. No layout/ownership change yet (phase 2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Bring the remaining deferred subsystems under the scoped sudoers, and drop
the one that's redundant.
Backup engines + app configs -> root-owned helpers (same pattern as
ownership/dns/ssh/socket/svc):
- scripts/system/libreportal-bininstall: install <restic|kopia> — does the
whole pkg-manager/signed-download install itself for a fixed, validated
engine name (no blanket sudo apt-get/install). restic_install/kopia_install
call it.
- scripts/system/libreportal-appcfg: {adguard-auth <user> <bcrypt>|
crowdsec-priority|owncloud-config <public> <host> <ip> <public_ip>} —
faithful ports of the AdGuard yaml / CrowdSec bouncer / ownCloud config.php
rewrites, fixed paths + validated args. adguard_auth/crowdsec_fix_priority/
owncloud_setup_config call it.
- run_privileged: runBinInstall / runAppCfg; init.sh installs + allowlists both.
Retire standalone (host-level) WireGuard — it's a duplicate of the
containerized containers/wireguard app (+ headscale mesh), its slirp4netns
speed rationale is largely moot with a better rootless net backend / typical
WAN-bound throughput, and it was the heaviest host-root subsystem (apt +
sysctl + iptables + /etc/wireguard), the worst fit for the rootless/
least-privilege direction:
- moved scripts/wireguard/ + manage_wireguard.sh + check_wireguard.sh to
scripts/unused/; dropped the install-path call, the Tools menu 'w' entry,
and the requirement check; removed the half-built libreportal-wg helper.
- generate_arrays.sh now also skips system/ (root-owned helpers, never
sourced); arrays regenerated (files_wireguard.sh pruned).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Add webui_system_metrics.sh, run each minute from webuiSystemUpdate:
- whole-server snapshot (metrics.json): CPU% + load, memory + swap,
per-mount disk + inodes, network rx/tx rate, docker summary
- capped ring buffer (metrics_history.json, 24h default) for trend charts
- per-app docker stats grouped by compose project (metrics_apps.json)
plus a short per-app history (metrics_apps_history.json) for sparklines
CPU% and network rate use stateful deltas stashed beside the JSON; all
host metrics read from /proc and docker via runFileOp, so it works rootless.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The install/start paths and the switch reconcile managed /docker ownership
separately, so a fresh install produced different ownership than a post-switch
state — the root cause of the rootless 'touch: Permission denied' storm.
Consolidate onto the reconcile model:
- dockerContainerOwner(): single definition of the mode's container owner
(rooted -> manager, rootless -> config-authoritative docker install user).
- reconcileContainersTopOwnership(): owns + makes traversable the structural
containers/ top dir; now also run by the switch reconcile (previously only
the install pass set it, so a rootless->rooted switch left it stale).
- reconcileWebuiDirOwnership(): now uses dockerContainerOwner.
- reconcileDockerOwnership(): calls both helpers.
- fixFolderPermissions(): slimmed to the +x traversal bits; its ad-hoc
containers/ chown is now the shared helper.
- fixPermissionsBeforeStart(): drop changeRootOwnedFilesAndFolders (a
pre-de-sudo band-aid that only fixed root-owned files and ran contrary to
the don't-touch-third-party-data rule); reconcile the WebUI dir via the
shared helper instead. Delete the now-unused root_files_folders.sh and
regenerate the source arrays.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Single place that decides how a privileged op runs by Docker mode:
- runFileOp / runFileWrite: /docker data-plane ops — rooted uses sudo (identical
to today), rootless runs as the unprivileged install user (no root).
- runSystem: genuine system-admin ops, sudo in both modes, funnelled here so it
can later be confined to a scoped sudoers allowlist.
Call sites converted to these are byte-for-byte unchanged under rooted, so
existing/live boxes can't regress; rootless gets the de-privileged path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The SSH-access feature's files_ssh.sh array was never registered in
files_source.sh, leaving it unsourced and blocking the deploy auto-merge.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Reads files the backup user can't see from the host (container-owned, e.g.
Nextcloud's www-data data dir) by streaming them out THROUGH the container
(docker exec tar) — no host root, no host read perms, works rooted + rootless.
Extracts to staging as plain files so restic keeps full dedup + per-file
restore (not a piped tar blob); the live path is excluded from the snapshot.
Restore streams the staging copy back through a throwaway in-namespace
container that recreates the tree with the app's uid:gid.
Declared via a libreportal.backup.files compose label; Nextcloud (html, 33:33)
is the first to use it. Live capture failure falls back to stop-snapshot-start.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Fresh, on-demand inbound SSH-access management for the host (replaces the old
maze). scripts/ssh/host_access.sh manages the install user's authorized_keys —
add a pasted public key (validated), list, remove — and toggles sshd password
login behind a lockout guard (won't disable passwords with no key; won't drop
the last key while passwords are off; sshd -t before reload, with backup).
New 'ssh' CLI category (status/key-add/key-remove/password-auth/generate) and
a webuiGenerateSshAccess snapshot (data/ssh/access.json: user, password_auth,
authorized keys as type+fingerprint+comment — public only) wired into the
regen chain. Nothing runs automatically; only explicit admin actions change
anything. WebUI page next.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The old inbound-admin-SSH layer was effectively dead: gated on config flags
that don't exist (CFG_SSHKEY_*_ENABLED, CFG_REQUIREMENT_SSHREMOTE), its
authorized_keys installer was unwired, and its download path (sshdownload
container) was already retired. What remained reachable was either a no-op or
a lockout footgun (disable-passwords with no working key install).
Remove it whole: scripts/ssh/*, the four SSH requirement checks, the SSH tools
menu, the dead webui SSH populater, and the unused ssh DB inserts; drop their
calls from the start/requirements/menu flows. A fresh, WebUI-driven admin SSH
access feature replaces it next.
Also make generate_arrays.sh self-healing: prune files_*.sh whose source
folder no longer exists (cleared the now-stale files_ssh.sh + an orphan
files_api.sh) so removed areas don't linger in the sourced set.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
New script files are sourced from the committed files_*.sh arrays (built by
generate_arrays.sh), not a live tree scan — and quick deploys don't rerun
generate_arrays. So the schema generator added last commit was never loaded
live: webuiGenerateBackupSchema was undefined, breaking the webui_updater
backup chain at that step (skipping the passwords regen after it) and leaving
schema.json un-generated.
Regenerate the arrays so the file is registered; deploy now sources it and
'webui generate all' rebuilds schema.json on its own.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Adds a logical-dump path so apps with a database can be backed up with zero
downtime and full consistency, instead of stopping the container.
- backup_db.sh: dump each declared DB live (mysqldump --single-transaction /
pg_dump / sqlite3 .backup), exclude the raw data dir from the snapshot, and
replay the dump on restore (pre-start rehydrate for sqlite, post-start load
for server engines).
- Databases are declared via a 'libreportal.backup.db' compose label so the
metadata travels with the app in the snapshot.
- New 'auto' strategy (now the default): live where a DB is dumpable or the app
is marked live-safe, stop-snapshot-start otherwise. Explicit stop/pause/live
remain as overrides.
- restic/borg/kopia adapters honour an exclude list on the live path.
- Manifest records the resolved per-app strategy and dumped databases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Application backups were driven by one crontab entry per app, each offset by
id * CFG_BACKUP_CRONTAB_APP_INTERVAL minutes. That minute offset is written
straight into cron's 0-59 minute field, so past ~20 apps it overflowed into
an invalid entry that silently never fired, and the fixed spacing could not
serialize backups that ran longer than the gap.
Replace it with a single daily entry (`libreportal backup scheduled`) that
enqueues a backup task per enabled app. The existing systemd task processor
drains them serially — no minute overflow, real serialization, and backups
are now visible/cancellable in the Tasks UI. Per-app enable is read from
CFG_<APP>_BACKUP at schedule time instead of being mirrored into crontab.
Removes the stagger machinery (timing/setup/check/remove scripts), the
now-unused cron_jobs table + insert, and the CFG_BACKUP_CRONTAB_APP_INTERVAL
config knob and its WebUI field.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Propagate the ✓ Success / ✗ Error / ! Notice / ❯ Question glyphs (from markers.sh) through the rest of the pipeline: swap the inlined helpers in init.sh and generate_arrays.sh, and replace raw echo -e "${RED}ERROR:${NC}" calls with the isX helpers in config_check_missing.sh, check_success.sh, initilize_files.sh, and reset_git.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Swap the ### hash headers (isHeader) for a ╔═╗ ║ ╚═╝ double-line box and
wrap the LibrePortal logo in a matching 52-wide box. Build the rule with
printf-repeat and fixed pad widths instead of tr/${#} so multibyte box
chars stay aligned regardless of locale. Mirrors the credentials panel.
Applied to all three copies (markers.sh, init.sh, generate_arrays.sh).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
librelad