Both found in a user's console log.
1. ReferenceError: setupMobileMenu is not defined (dashboard.js:98)
core/topbar/js/mobile-menu.js defines that global, and index.html
never loaded it. dashboard.js called it unguarded as the FIRST line
of setupEventListeners, so dashboard init threw every page load and
took loadInstalledApps() with it — and the burger menu was dead on
mobile. system-loader already guarded its own call with a typeof
check, which is why this survived unnoticed.
Loads the script (before dashboard.js) and guards the call, so
optional nav chrome can never take down the page below it again.
2. Endless 404s on /api/tasks/<id> for tasks that no longer exist
queue.json is append-only from the enqueue side and nothing ever
pruned it, so any task file removed afterwards left an id the WebUI
re-fetched forever, one 404 per poll per orphan. Adds
cleanupOrphanQueueEntries to the idle housekeeping pass: entries with
no task file are dropped and logged. Self-heals existing strays.
(Provoked by my own clean-up of two test tasks earlier in this
session, but the gap is real and predates it.)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The missing piece of hands-off updates/backups: when a task fails while
nobody has the WebUI open, LibrePortal now says so — email (via the
existing Mail settings), ntfy, Gotify, Discord, Slack, Telegram, or
Pushover, configured under Settings → Notifications.
One hook, everywhere: the task processor reports every terminal task to
`libreportal notify task <id>` (detached, never load-bearing — hard curl
timeouts, failures ignored). The POLICY lives in the notify command, not
the daemon: CFG_NOTIFY_EVENTS = failures (default) | all | off, and
cancelled tasks never notify. Failure copy is task-aware — a failed
update says the app was already rolled back and won't be retried, so the
reader knows the box is safe before opening the WebUI.
`libreportal notify test` sends to every enabled channel with per-channel
results. Verified against a local mock endpoint: all webhook payloads,
JSON escaping (quotes/newlines), the events policy, and fail-fast on
dead endpoints (8ms, exit nonzero).
The v0.1.0 per-app NOTIFY_* field-mapping scaffolding (never wired to a
sender) stays as-is; this global channel is the system it was waiting on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An offline trivy install crash-looped (server FATALs when it can't fetch the
vuln DB), and on rootless docker the restart storm churned the shared network's
port-forwarder until the WebUI's own published host port was torn down — the
WebUI stayed healthy INSIDE its container but was unreachable from the host, with
nothing detecting or healing it.
Three fixes, in the house self-healing style (mirrors the network-drift trio):
1. Control-plane health checker wired into the existing task-processor idle poll
(maybeRegenPoll), no new daemon. dockerHealthScan (read-only) detects daemon
down, a WebUI running-but-host-port-unreachable (the port-forward corruption),
and crash-looping containers. webuiSystemHealthCheck writes
frontend/data/system/health_status.json + self-dispatches a heal — the user
can't click a button on a dead WebUI, so the poll drives the fix. Frontend
health-notifier surfaces a topbar badge + dashboard banner + details panel.
2. Failure cap, enforced centrally by dockerHealthHeal (task-gated): stops
crash-loopers (removing the churn), restarts the WebUI to re-publish a lost
port forward, and — only if that fails — recycles the rootless daemon and
restarts the core container. Caps every app immediately, no template churn.
3. Trivy no longer crash-loops offline: the server runs in a shell retry-loop so
the container stays Up and quietly retries on a backoff instead of exiting
FATAL. Verified: container stays Up across repeated DB-download failures.
Core WebUI compose gains restart: unless-stopped so it self-recovers after a
reboot / daemon recycle instead of staying down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Root cause of the "task loops eternally" bug: the lazy-autoload stub was
fn() { source "$file"; fn "$@"; }
If the source fails — the real case: an app-install/deploy rsync briefly
removes-then-replaces a generator file while a setup task calls it — `fn` is
never redefined, so `fn "$@"` re-invokes the *stub*, which sources the (still
missing) file, which re-invokes the stub… A single setup finalize recursed
12,050 levels, flooding the task log and taking 82s before it happened to
recover when the file reappeared. A permanently-missing file would never
recover.
Fix (root cause): drop the stub before sourcing —
fn() { unset -f fn; source "$file"; fn "$@"; }
so a failed source degrades to one "command not found" (rc 127) instead of
unbounded recursion. Regenerated function_manifest.sh (975 stubs, reformat
only — no function-set change).
Failsafes on the task processor (defence in depth, per request):
- FUNCNEST cap (TASK_FUNCNEST_MAX, default 1000) inside the task's eval
subshell — any runaway recursion now aborts in milliseconds instead of
spamming the log until the stack/disk gives out.
- Wall-clock cap (TASK_MAX_RUNTIME_SECS, default 7200s, 0=off) — the heartbeat
watcher TERM→KILLs a task's process group once exceeded and marks it failed
(distinct from a user cancel via a .timeout marker). Generous so real long
installs/backups/migrations finish.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Holding the singleton flock at startup proves no other processor is alive
to heartbeat or complete anything, so every task still marked running is
a corpse from a killed predecessor. Recover them all before the first
dispatch (recoverOrphans now takes an 'all' mode) instead of waiting out
the 60s heartbeat-staleness window — which used to leave a dead task
showing 'running' alongside the genuinely-running next task for a minute
whenever the service was restarted mid-task (e.g. by the deploy chain
during initial setup). The idle-loop pass keeps the stale-only gate.
refactor(dashboard): slim the storage card back to chart + percentage
The disk card was only ever meant to be the donut and the % figure; drop
the Apps/Docker/Other/Free legend rows and signal the deeper view with a
corner expand glyph instead (the System page's chart-expand icon) — the
card already opens /admin/system/storage on click.
Signed-off-by: librelad <librelad@digitalangels.vip>
Replace the click-to-scan-only flow with a self-throttled auto-scan that
rides the existing task-processor idle poll (the same shape as the
network-drift check — no new daemon, unit, or endpoint):
- 'libreportal updater check auto' gates on the age of the generated
updates.json vs CFG_UPDATER_SCAN_INTERVAL (minutes, default 30,
0 disables); a fresh file makes the 60s tick a single stat() + return.
Manual checks and post-update rescans reset the clock for free, and a
missing file means the first scan runs ~a minute after install.
- Eligible signed hotfixes keep flowing through artifactApplyAuto, which
only enqueues ordinary tasks — mutations stay on the task path.
- Open updater surfaces (standalone /updater and the fleet Overview's
headless UpdaterPage) follow along with a 60s static-JSON re-read that
repaints only when a generated_at stamp changed; timer released via
dispose() on unmount, ticks skipped while hidden.
- Empty states now say the first scan happens automatically; Check now
stays as the immediate manual override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Closes the gap behind the vpn-recreate bug: when the shared network is
recreated with a different /24, every app's stored static IP is left
outside it and adoptDockerSubnet only realigns CFG, not the apps.
- networkScanConflicts (network_conflicts.sh): read-only scan diffing each
active network_resources IP against docker's real subnet (via ipInSubnet).
Per-service routing-aware — skips gateway-routed services whose ipv4 is
commented out in the deployed compose, so gluetun apps don't false-positive.
Distinguishes 'daemon down' (benign) from 'network missing' (real).
- webuiSystemNetworkCheck (webui_system_network.sh): self-throttled generator
that writes frontend/data/system/network_status.json (modelled on
verify_status.json). Wired into webuiSystemUpdate AND run unconditionally
every ~60s from the task-processor poll (regen webui is mtime-gated and
would never fire on drift, which touches no source file).
- networkHealConflicts (network_heal.sh) + 'libreportal system network
check|heal [app]': the heal adopts docker's subnet in-process, then re-IPs
stranded apps with reset_network=ip (ports preserved), gluetun first.
Mutating path runs only through the task system (dual-mode, like update
apply); read-only check runs inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
maybeRegenPoll truncates $REGEN_POLL_STAMP (.regen_poll_at) to throttle the
self-heal 'regen webui' poll, but the stamp lives in the docker-install-owned
TASK_DIR — the manager-run processor can't write there, so the truncate
EACCES'd every poll (swallowed by || true). The stamp never updated, so the
throttle read last=0 forever and 'regen webui' ran on every idle tick (and
spammed the journal ~16x/min).
Fix: pre-create the stamp world-writable in setupTaskDir, exactly like the
lock file and FIFO already are (runFileOp install -m 666). Truncate then
lands, the mtime advances, and the poll throttles to REGEN_POLL_INTERVAL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The task processor is a systemd-service daemon, not a cron job — move it out
of the misleadingly-named scripts/crontab/task/ to scripts/task/.
To stop the systemd unit from baking the processor's in-tree path (the footprint
coupling that forces a reinstall on every reorg), the unit now ExecStarts the
stable wrapper: /usr/local/bin/libreportal __task-processor. start.sh intercepts
that early (after paths.sh, before the heavy load), exports install_scripts_dir,
and exec's the processor with start_script. Future moves/renames need only the
one hand-off updated + a regen — no footprint bump.
- git mv scripts/crontab/task -> scripts/task (filenames kept; cron-watchdog grep
+ function names unchanged)
- libreportal-svc: ExecStart -> stable wrapper launcher
- start.sh: __task-processor internal launcher (export install_scripts_dir; exec)
- crontab_task_processor.sh: fix self-location ../.. -> .. for the new 1-level
depth (latent bug the move would otherwise have introduced)
- regen files_*/function_manifest; add task_scripts to the app/cli aggregates
- footprint_version 3 -> 4 (root-owned svc unit changed -> needs a root reinstall)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
