The Migrate tab carried two walls of explanation text — a 3-line hint
under the h2 ("Pulls a snapshot taken on another host…") and an even
longer empty-state paragraph ("Either no other LibrePortal has backed
up to a location this host can see, or this is the only host using its
locations…"). Both spelled out diagnosis the user can infer from the
empty list itself, and the tone didn't match the rest of the backup
page (cards elsewhere have a short title + a 4-6 word hint, with any
long explanation as a hover title attribute).
Three changes:
1. h2 down to "Cross-host migrate" with a small ℹ️ carrying the full
explanation as a title= tooltip — matches the existing tooltip
pattern in the Locations form (BACKUP_RETENTION_PRESET_META).
The short subtitle "Restore an app from another LibrePortal" stays
as backup-card-hint, mirroring "Per-app status / Latest backup per
app on this host" elsewhere on the page.
2. The empty state is now the standard `<div class="backup-empty-state">`
container (same shape Locations + Snapshots use), one trimmed line
("No backups from other hosts visible in any enabled location.
Add a shared backup location on both hosts to enable cross-host
migrate.") instead of two paragraphs.
3. Added an "Open Locations" CTA button inside the empty state — the
#1 next-step for a user staring at this empty list is to add a
shared location, which lives one tab over. New data-action
"go-to-locations" wired through the existing event-delegation
handler in backup-page.js calling switchTab('locations').
The renderMigrate JS still toggles #backup-migrate-empty.hidden — the
wrapper id is unchanged, only its inner markup tightened. No
behavioural change beyond the CTA + tab switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Existing installs were locked out of template-side description /
options / marker changes because reconciliation kept the user's whole
`CFG_KEY=value # comment` line verbatim. So new metadata like the
**DEV** marker I'm adding for the developer-mode feature wouldn't take
effect on already-deployed boxes — only fresh installs.
Updated reconcileConfigFile to split each line into value-part and
comment-part. User value is still sacred; the comment (title,
description, [options], **ADVANCED**/**DEV** markers) now comes from
the template. Field renames, label tweaks, marker additions/removals
shipped in a release reach existing installs on the next CLI
invocation (which runs the reconciler).
Specifically unblocks: the developer-mode WebUI feature (CFG_DEV_MODE
field gets added by the existing add-only path; CFG_INSTALL_MODE and
CFG_RELEASE_CHANNEL now pick up their new **DEV** markers and the
'Release - Stable' / 'Bleeding Edge' labels).
Signed-off-by: librelad <librelad@digitalangels.vip>
What this delivers (Stage 1+2 of the dev-mode feature):
1. New `**DEV**` marker for config fields. Mirrors the existing
`**ADVANCED**` pattern: stays in the description string, frontend
strips it for display, presence flips a 'hide unless dev mode is on'
behaviour. Implemented in ConfigUtils.cleanDescription /
isDevField / isDevModeOn and in ConfigShared._filterDevKeys, which
the two generateFieldsForCategory* helpers now call before rendering.
2. New CFG_DEV_MODE field in configs/general/general_install. Visible
under Advanced; defaults to false. The canonical place to toggle
dev mode (the WebUI easter egg writes to it, the auto-detector
writes to it, and users can flip it directly here too).
3. Marked CFG_INSTALL_MODE and CFG_RELEASE_CHANNEL with `**DEV**`.
Normal users no longer see either field — they install Release-
Stable and that's the whole story. Devs see both with the
user-facing labels you asked for:
CFG_INSTALL_MODE Release - Stable | Git clone | Local folder
CFG_RELEASE_CHANNEL Release - Stable | Release - Bleeding Edge
(CFG_INSTALL_MODE label for the release option also renamed to match.)
4. 10-click LibrePortal-logo easter egg in topbar.js:
- Counter on any .libreportal-logo click; idle-reset after 3 s
- Toast countdown from click 6 ('4 clicks away from being a developer…')
- At 10: toggles CFG_DEV_MODE via the standard config_update task
(same path the Config form uses); shows '🛠️ Developer mode
unlocked. Reload to see the extra options.'
- Re-using the same logo when dev mode is on toggles it back off
('… away from disabling developer mode') — symmetric, no separate UI
5. Auto-detect: on every WebUI load, if CFG_INSTALL_MODE is git or
local AND CFG_DEV_MODE is off, auto-flip to on with a one-time
toast 'Developer mode auto-enabled — you're on a git install.
Click the LibrePortal logo 10× to disable.' Stops dev-install
users getting locked out of the very options they need to manage
their install. Idempotent — runs once per page load; no-op if
already on or on release.
Disable surfaces: (a) CFG_DEV_MODE in Advanced on the Config form is
the canonical toggle; (b) 10 more logo clicks. A 3rd surface (a System
page banner) is deferred — those two cover the practical cases.
Signed-off-by: librelad <librelad@digitalangels.vip>
`libreportal app generate <name>` (and the menu's "g. Generate App" entry)
was broken three independent ways and incompatible with the per-app
architecture the project actually uses now:
1. Copies from $install_containers_dir/template/ which doesn't exist —
the only template/ in the tree was in scripts/unused/OLD_CONTAINERS/
and was never installed into the live tree. cp -r would just fail.
2. Every sed call used BSD/macOS syntax `sed -i '' -e …`. On Linux
(every distro this targets) the empty '' becomes a positional file
argument, so the substitutions never ran. 8 calls, all broken.
3. Even if it had run, the produced skeleton would have been a
pre-modular-tools / pre-per-port-subdomain app shape: no tools/,
no scripts/ subdir, HOST_NAME=test in the .config. Every active
containers/<app>/ today carries the modular layout the rest of the
framework expects.
Plus the recent cleanups (the prompt loop fix in 9ffc8e4, the per-port
subdomain refactor in 2e4f420) had been peeling pieces off it without
the root question — does the function still belong? — getting asked.
Delete the whole surface:
- scripts/app/app_generate.sh (157 lines, the function body)
- scripts/unused/OLD_CONTAINERS/template/ (the never-installed source
files appGenerate would have copied — stale enough to still carry
HOST_NAME=test, CFG_<X>_HOST_NAME, and 248 lines of compose template)
- menu entry "g. Generate App" + its dispatch in menu_main.sh
- "generate" case branch in cli_app_commands.sh
- `libreportal app generate` line in cli_app_header.sh
- The corresponding entries auto-drop from files_app.sh +
function_manifest.sh via regen.
New apps are added the way the catalog already grew — by hand-crafting
containers/<app>/{<app>.sh, <app>.config, docker-compose.yml,
tools/<app>.tools.json, scripts/<app>_*.sh}. Copying an existing app's
folder + renaming is the closest thing to a "generator" and it's a one-
command operation.
Net: -556 lines, no behaviour lost (the function never worked).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
initilize_files.sh's load-comment said "Phase 6 will tackle this with a
precompiled cache file" — but Phase 6 deliberately shipped a different
(simpler) win, and Phase 7 was reconsidered and rejected (the ~30 ms
saving doesn't justify the invalidation complexity across every CFG
write site). Rewrite the comment to describe current behaviour: the two
config scans run because bash can't lazy-load variables, and the
precompile was looked at and dropped.
app_generate.sh had a `read -p "" host_name` inside a `while true` with
no break — anyone who actually ran `libreportal app generate` would
have hung at the hostname prompt forever. The value was then fed into
a `sed 's/HOST_NAME=test/HOST_NAME='"$host_name"'/g'` against the new
app's .config file, but the post-2e4f420 template no longer carries
HOST_NAME=test (per-port subdomains in the PORT row drive routing now).
So the prompt was infinite-looping to gather a value that fed a no-op
sed. Drop both — the function loses no real capability since the
subdomain field is set per-port via the standard PORT row.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The installer (rootless_docker.sh:123) already defaulted CFG_ROOTLESS_NET
to pasta when unset — but the bundled configs/network/network_rootless
shipped CFG_ROOTLESS_NET=slirp4netns with a description warning about
the AppArmor caveat. That made the WebUI Config page surface slirp4netns
as the selected option even though the install script preferred pasta if
unset, and the warning told users they'd have to hand-relax the AppArmor
profile if they switched.
Both are now obsolete:
- CFG_ROOTLESS_NET=pasta is now the explicit default in the bundled
config (matches the installer's implicit default).
- Description drops the AppArmor manual-fix warning since the
installer applies the local override automatically
(installRootlessApparmorForPasta, shipped in the previous commit).
Dropdown order swapped too — pasta now top of the list as the
recommended option, slirp4netns kept as 'legacy fallback'.
The live install on this box already runs pasta (manually flipped
during debugging); the CFG file was synced to match so a future
rootless reinstall doesn't revert.
Signed-off-by: librelad <librelad@digitalangels.vip>
The Debian-shipped passt AppArmor profile (/etc/apparmor.d/usr.bin.passt)
denies the accesses pasta needs to plumb rootlesskit's netns:
- ptrace_read on the rootlesskit child to enter its user namespace
- read /run/user/<uid>/dockerd-rootless/netns (the netns file)
- read /proc/<pid>/net/{tcp,tcp6,udp,udp6} for implicit port forwarding
Without these the rootless docker daemon fails with:
pasta failed with exit code 1:
Couldn't open user namespace /proc/<pid>/ns/user: Permission denied
scripts/docker/install/rootless/rootless_apparmor.sh:
New installRootlessApparmorForPasta() — idempotent fixup.
1. Adds `include if exists <local/usr.bin.passt>` to the main profile
(one line; re-adding is a no-op via grep).
2. Writes /etc/apparmor.d/local/usr.bin.passt with the four rules
pasta needs. The /local/ pattern is the standard Debian AppArmor
hook for site-managed overrides — survives `apt upgrade passt`
because it's outside the package's managed paths.
3. Reloads via apparmor_parser -r.
Called from installDockerRootless after the override.conf write, gated
on $rootless_net == pasta. slirp4netns installs skip it.
This box was already manually patched while debugging the pasta swap —
the installer-side change makes it idempotent across reinstalls and
applies the same fix on any other host that installs rootless docker
with pasta as the net driver.
Signed-off-by: librelad <librelad@digitalangels.vip>
The 1h max-age set in Phase A caused a cache-vs-deploy mismatch when
Phase B refactored config-manager.js to lazy-load admin-overview.js et
al. The new index.html no longer eager-loads those scripts, but
browsers with the cached (pre-Phase-B) config-manager.js didn't do the
lazy-load either — so AdminOverview / AdminSystem / etc. were
undefined and the admin tools rendered 'failed to load' errors.
60s is the right balance: rapid in-session clicks skip the network
round-trip, but a deploy is visible within a minute. ETag-based 304s
still keep the per-request cost tiny when nothing changed.
Signed-off-by: librelad <librelad@digitalangels.vip>
Two webui-data generators wrote to a temp file via bare `cat > "$temp_file"`
then `runFileOp mv` to the final path. The temp file's path sits inside
$containers_dir/libreportal/frontend/data/<x>/generated/ — owned by the
dockerinstall user (the data plane). The generators run as the manager,
who can't open paths under that tree for write, so every WebUI update
hit:
webui_backup_migrate.sh: line 125: …/migrate.json.tmp.<pid>: Permission denied
mv: cannot stat '…/migrate.json.tmp.<pid>': No such file or directory
webui_peers.sh: line 23: …/peers.json.tmp.<pid>: Permission denied
mv: cannot stat '…/peers.json.tmp.<pid>': No such file or directory
Pipe the heredoc through `runFileWrite "$output_file"` instead — same
shape the 5 sibling generators in this dir (backup_app_status,
backup_locations, backup_passwords, backup_snapshots, backup_dashboard)
already use. runFileWrite routes the write via the install user that
owns the data tree, so the file lands on disk in one step (no temp +
mv dance needed). The unused `local temp_file=...` declarations dropped
out cleanly.
The trailing `runFileOp chmod 644 "$output_file"` stays — it's the only
guarantee the WebUI container (which reads these files RO) sees them as
world-readable regardless of dockerinstall's umask.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
initUpdateConfigOption (init.sh) and commandUpdateConfigOption (the CLI
wrapper heredoc) both rewrite CFG_<NAME>= lines with a sed s-command
using `|` as the delimiter. The escaping covered only `/` and `&` in
$escaped_value, and $comment_part wasn't escaped at all — so any line
whose comment contains a literal `|` blew up the substitution:
sed: -e expression #1, char 167: unknown option to `s'
The trigger in the install log is the CFG_INSTALL_MODE comment:
# Installation Mode - ... [release:Release tarball (recommended)|git:Git clone (dev)|local:Local folder (dev)]
Two sed errors in install-20260526-223006.log, both same line — once
from initUpdateConfigOption during the initial-values pass, once from
the CFG_INSTALL_MODE re-set later. The substitution silently failed
(line not rewritten) and the install continued.
Switch the delimiter to SOH (\x01). Text-based config values + comments
never contain that byte, so the delimiter never needs to be escaped.
Only `&` (whole-match insertion in the replacement) and `\` (escape
char) remain hazardous in the replacement field, and BOTH are now
neutralised in $escaped_value AND in $comment_part.
Verified against the actual offending line: the old form reproduces
`sed: unknown option to 's'` at char 165; the new form rewrites cleanly
with every `|` in the comment preserved.
Same fix applied to both functions — initUpdateConfigOption lives at
install-time, commandUpdateConfigOption is baked into the CLI wrapper
at /usr/local/lib/libreportal/libreportal; new installs pick up both
from this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The wall-of-dashes "✗ Error This PERMANENTLY removes EVERYTHING" listing
made the most consequential prompt in the project look like a routine
error log: same icon as a failed command, unaligned columns, no visual
grouping. Replaced with a structured block:
- Single red ⚠ "PERMANENT — there is no undo" callout (instead of the
✗ "Error" prefix, which semantically means a thing failed — this is
a pre-action warning).
- Four bold section headings (Filesystem / Users / System integration
/ Containers + binaries) so the reader can scan by category.
- Aligned %-34s path column with dim trailing descriptors — the eye
can sweep the left edge without re-anchoring per line.
- Green "Left in place:" reassurance lands at the end (same content as
before, just promoted from two isNotice lines into one styled line).
Pure-presentation change — no behavioural difference, same destroy list,
same DELETE LIBREPORTAL prompt. Verified the printf format renders
cleanly with the colour vars from variables.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
7 page-specific controllers were eager-loaded in index.html on every cold
visit, even when the user lands on /dashboard and never opens /backup,
/admin, etc. Moved them to lazy-load via spa.js's existing loadScript()
helper, fired from each route's handler on first navigation:
/js/components/backup/backup-page.js — handleBackup()
/js/components/backup/backup-app-card.js — handleBackup()
/js/components/ssh/ssh-page.js — config-manager ssh-access
/js/components/peers/peers-page.js — config-manager peers
/js/components/admin/admin-overview.js — config-manager overview
/js/components/admin/charts.js — config-manager overview
/js/components/admin/admin-system.js — config-manager system
config-manager.js gets a tiny `lazyLoad` helper that delegates to
window.spaClean.loadScript with a graceful fallback when the SPA hasn't
booted (legacy paths). loadScript is idempotent — subsequent visits to
the same route are no-ops, so we don't re-fetch after the first nav.
Cold-load impact on /dashboard (the most common landing):
Before: 25 sync <script> tags loading ~1.7 MB raw / ~430 KB gzipped
After: 18 sync <script> tags loading ~1.5 MB raw / ~380 KB gzipped
+ corresponding parse-cost reduction on the client (no longer parsing
backup-page.js + apps-related JS just to render the dashboard)
Page-specific JS still loads cleanly when the user navigates there — a
single extra network round-trip per route on first visit, then cached
for 1h (per Phase A's cache headers). Compression (Phase A) means the
deferred JS is ~75 % smaller on the wire than it would have been
pre-Phase-A.
Sister update to .../Scripts/update.sh: rsync now uses --delete so
file removals in the source tree (this commit deletes 7 script tags;
earlier commits deleted config-manager-old.js) propagate to the live
install. Excludes still protect frontend/data/.
Signed-off-by: librelad <librelad@digitalangels.vip>
Three WebUI cold-load wins:
1. DELETED containers/libreportal/frontend/js/components/config/config-manager-old.js
66 KB / 68189 bytes. Zero references anywhere in source or deployed
tree (confirmed via grep across containers/libreportal/). Pure dead
code from a previous refactor — removed.
2. ADDED `compression` middleware (defensive require)
Gzip-compresses JS/CSS/HTML/JSON responses. Typical ~70 % wire-size
reduction → the 1.7 MB cold-load drops to ~500 KB. New package.json
dependency; container's node_modules is baked into the image so the
require is wrapped in try/catch to degrade silently until the image
is next rebuilt (libreportal app install libreportal, or a full
deploy). Once active: free wire-size win on every response.
3. ADDED static cache headers via staticOptions on express.static
- JS/CSS/icons: Cache-Control: max-age=3600 + ETag
(1h browser cache, cheap 304 revalidation after)
- HTML files: Cache-Control: no-cache + ETag
(always revalidates so SPA shell updates land
immediately after a deploy; 304 if unchanged)
Repeat navigation in the same browser session skips ~25 script-tag
round-trips entirely.
Net effect once compression deploys:
- Cold load: 1.7 MB → ~500 KB on the wire (~70 % shrink)
- Warm load: 25 conditional requests → 0 (served from cache for 1h)
- Deploy lands: HTML revalidates immediately, JS/CSS picks up after 1h
or hard refresh
Phase B (defer non-critical scripts via SPA loadScript) and Phase C
(rebuild image / split the bind-mount story for node_modules) come
next; this commit is the safe Phase A foundation.
Signed-off-by: librelad <librelad@digitalangels.vip>
CrowdSec's host-side install (the agent + nftables bouncer the LibrePortal
Traefik plugin talks to) had stayed on blanket sudo throughout the rootless +
de-sudo hardening: `sudo apt-get install crowdsec`, `curl | sudo bash`,
`sudo sed -i /etc/crowdsec/config.yaml`, `sudo touch + sudo chmod /var/log/
crowdsec*.log`, `echo $key | sudo tee /etc/crowdsec/traefik_bouncer.key`,
plus `sudo cscli capi register / console enroll / bouncers add`. None of
those are in the scoped LP_HELPERS / LP_SYSTEM sudoers grant the manager
now holds, so any user who enabled crowdsec would have hit hard sudo
failures on every privileged step.
Follow the libreportal-appcfg / libreportal-bininstall pattern: one new
root-owned helper at /usr/local/lib/libreportal/libreportal-crowdsec
that does every privileged op behind a fixed action vocabulary with strict
argument validation. The manager calls in via runCrowdsec — the scoped
sudoers grants exactly one binary, the same trust boundary the other
helpers rely on.
Actions:
install apt repo + agent + firewall-bouncer + enable +
crowdsecurity/{linux,sshd} collections + reload
(idempotent — skips parts already in place)
services <verb> enable | disable | restart
capi <verb> register | unregister | status
console <verb> enroll <token> | disenroll | status
token format strictly validated
bouncer-traefik-init cscli register + write the manager-owned key file
atomically (returns EXISTS or GENERATED:<key>)
bouncer-priority bouncer yaml nftables priority → -100
(moved from libreportal-appcfg; one helper for
every crowdsec root op)
bind-lapi flip listen_uri to 0.0.0.0:8080 in config.yaml
prometheus <on…|off> flip the prometheus block (validated addr/port)
touch-host-logs create + chmod 0644 /var/log/crowdsec*.log so the
libreportal container can tail them
Wired in via:
- new sudoers Cmnd_Alias entry for the helper in LP_HELPERS
- new helper baked alongside the others by initRootHelpers
(replaces __SYSTEM_DIR__ / __CONTAINERS_DIR__ / __MANAGER__ at
install, with safe runtime fallbacks if unbaked)
- new runCrowdsec dispatch in scripts/docker/command/run_privileged.sh
containers/crowdsec/scripts/crowdsec_install_host.sh now drives the whole
flow through runCrowdsec — every `sudo …` is gone, the compose-toggle sed
uses runFileOp, and the security_crowdsec CFG mirror uses runInstallOp
(configs/ is manager-owned). Net: install script shrinks ~80 lines while
gaining a single auditable trust boundary. crowdsec_fix_priority.sh swung
over to runCrowdsec bouncer-priority too — the appcfg crowdsec_priority
action drops out cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
runBackupOp dropped privileges to $docker_install_user with `sudo -E`,
which preserves the CALLER's environment — including HOME. The caller is
the manager (libreportal), so restic-running-as-dockerinstall ended up
with HOME=/home/libreportal and tried to mkdir
`/home/libreportal/.cache/restic` for its cache. dockerinstall can't
write into libreportal's home, so every backup ran with:
unable to open cache: mkdir /home/libreportal/.cache/restic: permission denied
twice (once in backup, once in the verify-via-scratch-restore step), with
restic falling back to a no-cache run that's a few × slower than it
should be.
Add `-H` (sudo's "reset HOME to target user's home"). Now restic sees
HOME=/home/dockerinstall, creates ~/.cache/restic there (dockerinstall
owns its own home, no help needed), and the warning is gone. Confirmed
live: a `backup app create linkding` round-trip is silent on cache, and
the dir lands at /home/dockerinstall/.cache/restic, mode 0700, correctly
owned.
All restic/borg/kopia calls funnel through runBackupOp, so this single
character fix covers every backup-tool invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Three small bugs in the legacy git-update flow that all hung off the
same never-set variable:
1. backup_install_dir was referenced in 4 files (reset_git_backup,
install_git_backup, use_git_backup, config_git_check) but DEFINED
nowhere — never has been, in any branch or tag. Resolved to "", so
"$backup_install_dir/$backupFolder" became "/backup_<ts>" (filesystem
root, perm denied). Add it to libreportalDerivePaths beside the other
roots, point it at $backup_dir/install (a sibling of restic's per-
location subdirs at $backup_dir/<idx>), and add it to initFolders so
it exists on first install.
2. gitCleanInstallBackups' find expression was
find ... -mindepth 1 -type f ! -name '*.zip' -o -type d ! -name '*.zip' -exec rm -rf {} +
`-o` binds looser than the implicit -a, so the -exec only applied to
the second clause. That meant: every non-.zip DIR anywhere under the
tree got deleted; every non-.zip FILE got matched and ignored. Even
once $backup_install_dir resolved correctly the cleanup would've
wrecked unrelated dirs.
Collapsed to `-mindepth 1 -maxdepth 1 ! -name '*.zip' -exec rm -rf {} +`
— direct children of $backup_install_dir, kill everything that isn't
a zip, let -rf take care of the dirs. Synthetic-tree smoke test
confirms only the .zip files survive.
3. use_git_backup.sh had a typo'd doubled var:
copyFolders "$backup_install_dir$backup_install_dir/$backup_file_without_zip/" ...
Reduced to the single $backup_install_dir/$backup_file_without_zip/.
All three only fire in the manual-update path (libreportal update apply
under git/local install mode); the install-blocking path is unaffected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Tree-wide audit (working tree + deployed install + every local/remote ref
+ every reachable commit + unreachable objects via git fsck) found zero
external callers. Existed dead since v0.1.0 — never wired in.
The function set DOMAINSUBNAME, TIMEZONE, DOCKER_NETWORK (all duplicates
of fills that happen elsewhere) plus the two unique-to-it CONFIGS_DIR_TAG
+ CONTAINERS_DIR_TAG. Those two are already wired directly into the
standard tag-fill block in dockerConfigSetupFileWithData (commit 521f08b),
so dropping the source file leaves no behavioural gap.
Also tighten the comment that explained why we inlined the two tags —
don't reference the function we're deleting in the same change. Describe
the current behaviour, not the history (per repo convention).
Regenerated the auto arrays + function_manifest.sh: the 3 stale entries
referencing this function drop out cleanly. files_cli.sh / files_config.sh
/ files_source.sh also rebuilt — no net content change beyond dropping
this one path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The location_loader's find ran through runFileOp (sudo -u libreportal),
which forks a sudo shell purely for the find call. configs/backup/
locations/ is already manager-owned (not in the dockerinstall-owned
containers tree), so the sudo step adds ~50 ms of process-creation
overhead for zero security benefit.
Plain `find` now, with a tiny [[ ! -r || ! -x ]] guard that falls
back to runFileOp if someone relocates the dir to a non-traversable
location. Same observable behaviour, ~50 ms saved per CLI invocation.
This is the simpler half of Phase 6 — the libreportal_configs scan
itself was already plain-find (only ~11 ms total for 22 files). The
remaining costly scan is app_configs against /libreportal-containers/,
which legitimately needs sudo because dockerinstall owns that tree.
Precompiling its content is possible but adds invalidation complexity
(updateConfigOption writes happen at runtime) — deferred for now;
better return on simpler interventions.
Signed-off-by: librelad <librelad@digitalangels.vip>
The early loop in sourceInitilize() sources every .sh under source/files/
recursively — including the new arrays/function_manifest.sh, which now
carries ~860 autoload stub definitions (~50 ms parse cost). Even in
eager mode where lazy infrastructure is never touched, every invocation
was paying that cost up front.
The manifest is only needed in lazy mode, where it's sourced explicitly
at the top of the lazy branch. Excluding it from the early loop:
- Eager mode: drops the ~50 ms regression introduced by Phase 5.
- Lazy mode: unchanged — the explicit source still runs.
This brings eager back to the pre-Phase-5 baseline and lets the lazy
container-stub gain (skipping sourceScanFiles containers, ~70 ms) show
through as a real saving.
Signed-off-by: librelad <librelad@digitalangels.vip>
Containers used to be eager-loaded via `sourceScanFiles "containers"`
even under LP_LAZY=1 — sourcing all ~160 installer functions up front.
Phase 5 brings them into the autoload-stub mechanism.
generate_function_manifest.sh now scans BOTH scripts/ AND containers/
(maxdepth 3, matching sourceScanFiles' existing prune), with a per-entry
root selector so stub emission uses the right base directory:
scripts/peer/peer_add.sh → source "${install_scripts_dir}peer/peer_add.sh"
containers/linkding/linkding.sh → source "${install_containers_dir}linkding/linkding.sh"
New manifest exports:
LP_FN_MAP funcname → relpath (existing)
LP_FN_ROOT funcname → scripts|containers NEW
LP_EAGER_FILES "<root>:<relpath>" entries NEW format
~860 autoload stubs (was ~700; +160 from containers)
Loader changes (initilize_files.sh):
- Parses LP_EAGER_FILES entries as `root:path`, dispatches to the
right install_*_dir. Pre-Phase-5 entries without a colon default to
scripts (backwards-compatible).
- sourceScanFiles "containers" is skipped when LP_LAZY=1 AND
LP_FN_MAP is loaded (manifest-driven autoload covers it).
Eager mode and lazy-with-missing-manifest both still run the scan.
Measurement target: ~70 ms saved on top of Phase 4. Verified separately
in the commit message of the next deploy.
Signed-off-by: librelad <librelad@digitalangels.vip>
`commandReloadConfigs` (baked into /usr/local/lib/libreportal/libreportal) and
`initCheckConfigs` both iterate every category dir's contents and `source` each
entry, with only a string-suffix exclusion for `.category` markers — no
`-f` test. That worked when `configs/<category>/` held only flat files.
The new backup system parks per-location configs at
`configs/backup/locations/<idx>/location.config`, so `configs/backup/locations/`
is now a SUBDIRECTORY inside the backup category. Sourcing it tripped:
source: /libreportal-system/configs/backup/locations: is a directory
…surfacing whenever something triggered a drift-driven config reload (e.g.
during a `regen --force` or a release-mode re-fetch). The nested location
configs already have their own dedicated loader (`sourceBackupLocations`)
that handles the depth-3 walk; the category-level loop just needs to leave
that subtree alone.
Collapse both loops to the cleaner guard `initReloadConfigs` and
`commandUpdateConfigOption` already use:
if [ -f "$config_file" ] && [[ ! "$config_file" =~ \.category$ ]]; then
…which both excludes directories (the bug) and the `.category` markers in
one shot, and drops a small pile of `should_load`/`filename` boilerplate
along the way. Verified live on dev-ai (CLI tool dispatch now works
through a drift-triggered reload without exiting non-zero).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/load_sources.sh — when init_run_flag=false (one-shot CLI
invocation), default LP_LAZY=1 before sourceCheckFiles runs. Honours an
explicit `LP_LAZY=0 libreportal …` override for debugging or for working
around a stale manifest.
Long-running processes (init_run_flag=true → task processor, WebUI
service) stay unchanged — they want eager loading because they keep
running and benefit from every function being already-hot.
Measured on this box, 3 runs each:
EAGER (default before this commit):
real 0.91s / 1.12s / 0.94s avg ~0.99s wall
user 0.40s / 0.41s / 0.42s avg ~0.41s CPU
LAZY (new default for CLI):
real 0.63s / 0.65s / 0.66s avg ~0.65s wall
user 0.26s / 0.27s / 0.26s avg ~0.26s CPU
Wall: ~340ms saved per invocation (34%).
User: ~150ms saved per invocation (37%).
Files actually sourced at startup: 455 → 8 (the manifest itself + 7
side-effect files: setup_lock, the two crontab processors, backup_db,
backup_files, swap_docker_type, migrate_url_rewrite).
Safety nets:
- Missing manifest auto-falls-back to eager loading (init.sh check
in the lazy branch sets LP_LAZY=0 if function_manifest.sh is absent).
- Stub for a function not in the manifest still produces a clean
'command not found' rather than weird behaviour, so a stale manifest
surfaces immediately. `libreportal regen arrays` regenerates both
files_*.sh and function_manifest.sh.
Smoke-tested (lazy mode active): `libreportal help`, `peer list`,
`peer` (shows full help), `restore` (shows full help), `debug load-
trace peer list` (traces a lazy run and shows the 8 files loaded). All
output identical to eager mode.
Signed-off-by: librelad <librelad@digitalangels.vip>
The previous run had 32 eager files; 24 of them were the auto-generated
files_*.sh arrays (only useful to the eager loader) + the manifest
itself (which the lazy loader sources explicitly). Eager-sourcing them
under lazy mode was pure overhead — ~55ms on the manifest alone (it was
being parsed twice, once via the explicit lazy-loader source and once
via the LP_EAGER_FILES loop).
Down to 8 eager files (the genuinely-side-effecting ones: setup_lock.sh,
the two crontab task processors, backup_db.sh, backup_files.sh,
docker swap_docker_type.sh, migrate_url_rewrite.sh, cli_debug_commands.sh).
The files_*.sh arrays are still sourced by the eager loader's existing
path — that's unchanged. Lazy mode just doesn't need them because it
never iterates files_libreportal_app[@] / files_libreportal_cli[@].
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/loading/initilize_files.sh gains an LP_LAZY=1 branch:
- Sources scripts/source/files/arrays/function_manifest.sh once. The
manifest defines LP_FN_MAP, LP_EAGER_FILES, AND ~700 autoload stubs
(precompiled by the generator — one parse cost vs evaluating 700
snippets at startup).
- Eager-sources every file listed in LP_EAGER_FILES (top-level side
effects: variable assignments, source calls, bare commands). These
can't safely be deferred — they'd skip the side effect, not just the
function definition.
- Skips the bulk loop that sources every files_to_source[@] entry.
Default behaviour (LP_LAZY unset or 0) is byte-identical to the previous
loader — every file gets eager-sourced up front. Long-running processes
(WebUI service, task processor) leave LP_LAZY unset because their first
call to anything wants the function already hot.
Each autoload stub looks like:
funcname() {
source "${install_scripts_dir}path/to/file.sh"
funcname "$@"
}
First call sources the real file, which redefines the function with the
real body; the stub's trailing `funcname "$@"` then calls the freshly-
defined real implementation. Sourcing the file also redefines stubs for
any sibling functions the same file declares, so they don't re-source.
Safety nets:
- Missing manifest → fall back to eager loading (`export LP_LAZY=0`).
No regression risk if someone enables LP_LAZY=1 on a stale install
whose regen never ran.
- LP_LOAD_TRACE=1 still works in lazy mode — it records the manifest
parse + each eager file (tagged LAZY-manifest / LAZY-EAGER) so Phase 4
can measure the actual saving.
No automatic flip yet — this commit only adds the path. Phase 4 will set
LP_LAZY=1 by default for the CLI entrypoint (and re-measure with the
trace tool from Phase 1).
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/files/generate_function_manifest.sh — scans every .sh in
scripts/ (skip-list matches generate_arrays.sh, plus excludes peer_shell.sh
which is a standalone forced-command target) and emits
scripts/source/files/arrays/function_manifest.sh:
declare -gA LP_FN_MAP=(
[acquireSingletonLock]="crontab/task/crontab_task_processor.sh"
[adoptDockerSubnet]="checks/requirements/check_docker_network.sh"
... # 698 entries
)
LP_EAGER_FILES=(
"backup/db/backup_db.sh"
"source/files/arrays/files_app.sh"
... # 32 entries (~7% of files)
)
The lazy loader (Phase 3) consumes LP_FN_MAP to install autoload stubs of
the form `name() { source "$LP_FN_MAP[name]"; name "$@"; }`. First call
sources the real file, which redefines the stub with the real body;
subsequent calls hit the real one. LP_EAGER_FILES enumerates files with
top-level side effects (variable assignments, source calls, bare commands
outside any function) — those MUST always source so the side effects fire.
Heuristic correctness, in order of importance:
- Function header detection requires EMPTY parens (`name()`), not just
`name(` — otherwise lines like `if (...)`, `for (...)`, `while (...)`
in embedded awk/perl get misread as bash function defs.
- Handles three function styles: `name() {` (same line), `name()\n{`
(LibrePortal convention), and one-liners `name() { body; }`.
- Tracks { } balance for inside-function depth, with the safe fallback that
ambiguous cases get marked eager (false positive = no behaviour change;
false negative would skip a needed source).
- Files containing embedded awk/perl with their own { } blocks (about 6 of
them: cli_debug_commands.sh, crontab_task_processor.sh, backup_db.sh,
backup_files.sh, etc.) get false-positive flagged eager — acceptable
because they just stay eager-loaded, matching today's behaviour.
- Collisions report to stderr (last-write wins, same as eager-load
semantics); no collisions found in the current tree.
Wiring:
- lpRegenArrays (`libreportal regen arrays`) now also runs the manifest
generator when the existing arrays need regen, keeping the two in sync.
- update.sh's quick-deploy regen step does the same after copying files
to the live install. Best-effort: failures don't abort because lazy
loading is opt-in (LP_LAZY=1) in Phase 3 and not the default yet.
Scanned: 454 files, indexed 698 function definitions, 32 eager (9 real,
23 auto-generated arrays + the manifest itself). 0 name collisions.
No behaviour change in this commit — the manifest is just data the loader
in Phase 3 will use. The default eager loading path is untouched.
Signed-off-by: librelad <librelad@digitalangels.vip>
