`libreportal app generate <name>` (and the menu's "g. Generate App" entry)
was broken three independent ways and incompatible with the per-app
architecture the project actually uses now:
1. Copies from $install_containers_dir/template/ which doesn't exist —
the only template/ in the tree was in scripts/unused/OLD_CONTAINERS/
and was never installed into the live tree. cp -r would just fail.
2. Every sed call used BSD/macOS syntax `sed -i '' -e …`. On Linux
(every distro this targets) the empty '' becomes a positional file
argument, so the substitutions never ran. 8 calls, all broken.
3. Even if it had run, the produced skeleton would have been a
pre-modular-tools / pre-per-port-subdomain app shape: no tools/,
no scripts/ subdir, HOST_NAME=test in the .config. Every active
containers/<app>/ today carries the modular layout the rest of the
framework expects.
Plus the recent cleanups (the prompt loop fix in 9ffc8e4, the per-port
subdomain refactor in 2e4f420) had been peeling pieces off it without
the root question — does the function still belong? — getting asked.
Delete the whole surface:
- scripts/app/app_generate.sh (157 lines, the function body)
- scripts/unused/OLD_CONTAINERS/template/ (the never-installed source
files appGenerate would have copied — stale enough to still carry
HOST_NAME=test, CFG_<X>_HOST_NAME, and 248 lines of compose template)
- menu entry "g. Generate App" + its dispatch in menu_main.sh
- "generate" case branch in cli_app_commands.sh
- `libreportal app generate` line in cli_app_header.sh
- The corresponding entries auto-drop from files_app.sh +
function_manifest.sh via regen.
New apps are added the way the catalog already grew — by hand-crafting
containers/<app>/{<app>.sh, <app>.config, docker-compose.yml,
tools/<app>.tools.json, scripts/<app>_*.sh}. Copying an existing app's
folder + renaming is the closest thing to a "generator" and it's a one-
command operation.
Net: -556 lines, no behaviour lost (the function never worked).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
CrowdSec's host-side install (the agent + nftables bouncer the LibrePortal
Traefik plugin talks to) had stayed on blanket sudo throughout the rootless +
de-sudo hardening: `sudo apt-get install crowdsec`, `curl | sudo bash`,
`sudo sed -i /etc/crowdsec/config.yaml`, `sudo touch + sudo chmod /var/log/
crowdsec*.log`, `echo $key | sudo tee /etc/crowdsec/traefik_bouncer.key`,
plus `sudo cscli capi register / console enroll / bouncers add`. None of
those are in the scoped LP_HELPERS / LP_SYSTEM sudoers grant the manager
now holds, so any user who enabled crowdsec would have hit hard sudo
failures on every privileged step.
Follow the libreportal-appcfg / libreportal-bininstall pattern: one new
root-owned helper at /usr/local/lib/libreportal/libreportal-crowdsec
that does every privileged op behind a fixed action vocabulary with strict
argument validation. The manager calls in via runCrowdsec — the scoped
sudoers grants exactly one binary, the same trust boundary the other
helpers rely on.
Actions:
install apt repo + agent + firewall-bouncer + enable +
crowdsecurity/{linux,sshd} collections + reload
(idempotent — skips parts already in place)
services <verb> enable | disable | restart
capi <verb> register | unregister | status
console <verb> enroll <token> | disenroll | status
token format strictly validated
bouncer-traefik-init cscli register + write the manager-owned key file
atomically (returns EXISTS or GENERATED:<key>)
bouncer-priority bouncer yaml nftables priority → -100
(moved from libreportal-appcfg; one helper for
every crowdsec root op)
bind-lapi flip listen_uri to 0.0.0.0:8080 in config.yaml
prometheus <on…|off> flip the prometheus block (validated addr/port)
touch-host-logs create + chmod 0644 /var/log/crowdsec*.log so the
libreportal container can tail them
Wired in via:
- new sudoers Cmnd_Alias entry for the helper in LP_HELPERS
- new helper baked alongside the others by initRootHelpers
(replaces __SYSTEM_DIR__ / __CONTAINERS_DIR__ / __MANAGER__ at
install, with safe runtime fallbacks if unbaked)
- new runCrowdsec dispatch in scripts/docker/command/run_privileged.sh
containers/crowdsec/scripts/crowdsec_install_host.sh now drives the whole
flow through runCrowdsec — every `sudo …` is gone, the compose-toggle sed
uses runFileOp, and the security_crowdsec CFG mirror uses runInstallOp
(configs/ is manager-owned). Net: install script shrinks ~80 lines while
gaining a single auditable trust boundary. crowdsec_fix_priority.sh swung
over to runCrowdsec bouncer-priority too — the appcfg crowdsec_priority
action drops out cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Tree-wide audit (working tree + deployed install + every local/remote ref
+ every reachable commit + unreachable objects via git fsck) found zero
external callers. Existed dead since v0.1.0 — never wired in.
The function set DOMAINSUBNAME, TIMEZONE, DOCKER_NETWORK (all duplicates
of fills that happen elsewhere) plus the two unique-to-it CONFIGS_DIR_TAG
+ CONTAINERS_DIR_TAG. Those two are already wired directly into the
standard tag-fill block in dockerConfigSetupFileWithData (commit 521f08b),
so dropping the source file leaves no behavioural gap.
Also tighten the comment that explained why we inlined the two tags —
don't reference the function we're deleting in the same change. Describe
the current behaviour, not the history (per repo convention).
Regenerated the auto arrays + function_manifest.sh: the 3 stale entries
referencing this function drop out cleanly. files_cli.sh / files_config.sh
/ files_source.sh also rebuilt — no net content change beyond dropping
this one path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Containers used to be eager-loaded via `sourceScanFiles "containers"`
even under LP_LAZY=1 — sourcing all ~160 installer functions up front.
Phase 5 brings them into the autoload-stub mechanism.
generate_function_manifest.sh now scans BOTH scripts/ AND containers/
(maxdepth 3, matching sourceScanFiles' existing prune), with a per-entry
root selector so stub emission uses the right base directory:
scripts/peer/peer_add.sh → source "${install_scripts_dir}peer/peer_add.sh"
containers/linkding/linkding.sh → source "${install_containers_dir}linkding/linkding.sh"
New manifest exports:
LP_FN_MAP funcname → relpath (existing)
LP_FN_ROOT funcname → scripts|containers NEW
LP_EAGER_FILES "<root>:<relpath>" entries NEW format
~860 autoload stubs (was ~700; +160 from containers)
Loader changes (initilize_files.sh):
- Parses LP_EAGER_FILES entries as `root:path`, dispatches to the
right install_*_dir. Pre-Phase-5 entries without a colon default to
scripts (backwards-compatible).
- sourceScanFiles "containers" is skipped when LP_LAZY=1 AND
LP_FN_MAP is loaded (manifest-driven autoload covers it).
Eager mode and lazy-with-missing-manifest both still run the scan.
Measurement target: ~70 ms saved on top of Phase 4. Verified separately
in the commit message of the next deploy.
Signed-off-by: librelad <librelad@digitalangels.vip>
The previous run had 32 eager files; 24 of them were the auto-generated
files_*.sh arrays (only useful to the eager loader) + the manifest
itself (which the lazy loader sources explicitly). Eager-sourcing them
under lazy mode was pure overhead — ~55ms on the manifest alone (it was
being parsed twice, once via the explicit lazy-loader source and once
via the LP_EAGER_FILES loop).
Down to 8 eager files (the genuinely-side-effecting ones: setup_lock.sh,
the two crontab task processors, backup_db.sh, backup_files.sh,
docker swap_docker_type.sh, migrate_url_rewrite.sh, cli_debug_commands.sh).
The files_*.sh arrays are still sourced by the eager loader's existing
path — that's unchanged. Lazy mode just doesn't need them because it
never iterates files_libreportal_app[@] / files_libreportal_cli[@].
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/loading/initilize_files.sh gains an LP_LAZY=1 branch:
- Sources scripts/source/files/arrays/function_manifest.sh once. The
manifest defines LP_FN_MAP, LP_EAGER_FILES, AND ~700 autoload stubs
(precompiled by the generator — one parse cost vs evaluating 700
snippets at startup).
- Eager-sources every file listed in LP_EAGER_FILES (top-level side
effects: variable assignments, source calls, bare commands). These
can't safely be deferred — they'd skip the side effect, not just the
function definition.
- Skips the bulk loop that sources every files_to_source[@] entry.
Default behaviour (LP_LAZY unset or 0) is byte-identical to the previous
loader — every file gets eager-sourced up front. Long-running processes
(WebUI service, task processor) leave LP_LAZY unset because their first
call to anything wants the function already hot.
Each autoload stub looks like:
funcname() {
source "${install_scripts_dir}path/to/file.sh"
funcname "$@"
}
First call sources the real file, which redefines the function with the
real body; the stub's trailing `funcname "$@"` then calls the freshly-
defined real implementation. Sourcing the file also redefines stubs for
any sibling functions the same file declares, so they don't re-source.
Safety nets:
- Missing manifest → fall back to eager loading (`export LP_LAZY=0`).
No regression risk if someone enables LP_LAZY=1 on a stale install
whose regen never ran.
- LP_LOAD_TRACE=1 still works in lazy mode — it records the manifest
parse + each eager file (tagged LAZY-manifest / LAZY-EAGER) so Phase 4
can measure the actual saving.
No automatic flip yet — this commit only adds the path. Phase 4 will set
LP_LAZY=1 by default for the CLI entrypoint (and re-measure with the
trace tool from Phase 1).
Signed-off-by: librelad <librelad@digitalangels.vip>
scripts/source/files/generate_function_manifest.sh — scans every .sh in
scripts/ (skip-list matches generate_arrays.sh, plus excludes peer_shell.sh
which is a standalone forced-command target) and emits
scripts/source/files/arrays/function_manifest.sh:
declare -gA LP_FN_MAP=(
[acquireSingletonLock]="crontab/task/crontab_task_processor.sh"
[adoptDockerSubnet]="checks/requirements/check_docker_network.sh"
... # 698 entries
)
LP_EAGER_FILES=(
"backup/db/backup_db.sh"
"source/files/arrays/files_app.sh"
... # 32 entries (~7% of files)
)
The lazy loader (Phase 3) consumes LP_FN_MAP to install autoload stubs of
the form `name() { source "$LP_FN_MAP[name]"; name "$@"; }`. First call
sources the real file, which redefines the stub with the real body;
subsequent calls hit the real one. LP_EAGER_FILES enumerates files with
top-level side effects (variable assignments, source calls, bare commands
outside any function) — those MUST always source so the side effects fire.
Heuristic correctness, in order of importance:
- Function header detection requires EMPTY parens (`name()`), not just
`name(` — otherwise lines like `if (...)`, `for (...)`, `while (...)`
in embedded awk/perl get misread as bash function defs.
- Handles three function styles: `name() {` (same line), `name()\n{`
(LibrePortal convention), and one-liners `name() { body; }`.
- Tracks { } balance for inside-function depth, with the safe fallback that
ambiguous cases get marked eager (false positive = no behaviour change;
false negative would skip a needed source).
- Files containing embedded awk/perl with their own { } blocks (about 6 of
them: cli_debug_commands.sh, crontab_task_processor.sh, backup_db.sh,
backup_files.sh, etc.) get false-positive flagged eager — acceptable
because they just stay eager-loaded, matching today's behaviour.
- Collisions report to stderr (last-write wins, same as eager-load
semantics); no collisions found in the current tree.
Wiring:
- lpRegenArrays (`libreportal regen arrays`) now also runs the manifest
generator when the existing arrays need regen, keeping the two in sync.
- update.sh's quick-deploy regen step does the same after copying files
to the live install. Best-effort: failures don't abort because lazy
loading is opt-in (LP_LAZY=1) in Phase 3 and not the default yet.
Scanned: 454 files, indexed 698 function definitions, 32 eager (9 real,
23 auto-generated arrays + the manifest itself). 0 name collisions.
No behaviour change in this commit — the manifest is just data the loader
in Phase 3 will use. The default eager loading path is untouched.
Signed-off-by: librelad <librelad@digitalangels.vip>
librelad