Prometheus kept being found stopped after boots, always Exited(0),
always alone. The journal settles it: both stops sit seconds before a
host shutdown boundary — container stopped 05:45:22, boot ended
05:45:30; stopped 04:41:59, boot ended 04:42:05. This is a laptop-class
host that gets shut down, and under ROOTLESS docker the containers are
ordinary processes in the user session, torn down by systemd in
parallel with dockerd's own exit.
That parallelism is the race. An app that handles SIGTERM promptly
exits while dockerd is still alive to record "stopped" — and
unless-stopped then means what it says: not restarted at the next
boot. Apps that exit slower, or die only when dockerd does, are
recorded as running and come back. Prometheus loses reliably because it
is the best-behaved process on the box ("See you next time!"), but
which app loses is a scheduling accident — changing Prometheus's
restart policy would treat the sample, not the race.
So an @reboot crontab entry now waits for the rootless daemon (up to
five minutes, then gives up rather than hang) and `compose up -d`s
every installed app via the existing dockerComposeUpAllApps. Idempotent:
running apps see no diff, stopped ones start, ordering is compose's
problem. Registered through crontabRefresh like the other entries, and
installed on this box.
The accepted trade, stated rather than hidden: an app deliberately
stopped before a reboot comes back after it. On a self-hosting box "the
fleet is up after boot" is the promise unless-stopped was already trying
to make; a stop that must survive reboots is what uninstall is for.
Verified by direct execution: daemon answered immediately, all
installed apps reconciled, running containers untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
'local result=$(cmd)' resets $? to 0 (the local builtin's own exit), so the
following checkSuccess always saw success regardless of cmd's real exit — the
mechanism that masked the de-sudo write failures. Split declaration from
assignment ('local result; result=$(cmd)') across all 235 active-code sites
(84 files) so the command's exit reaches checkSuccess. No behaviour change
beyond $? now being accurate (no set -e in runtime code; multi-line
assignments transform safely).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The task processor is a systemd-service daemon, not a cron job — move it out
of the misleadingly-named scripts/crontab/task/ to scripts/task/.
To stop the systemd unit from baking the processor's in-tree path (the footprint
coupling that forces a reinstall on every reorg), the unit now ExecStarts the
stable wrapper: /usr/local/bin/libreportal __task-processor. start.sh intercepts
that early (after paths.sh, before the heavy load), exports install_scripts_dir,
and exec's the processor with start_script. Future moves/renames need only the
one hand-off updated + a regen — no footprint bump.
- git mv scripts/crontab/task -> scripts/task (filenames kept; cron-watchdog grep
+ function names unchanged)
- libreportal-svc: ExecStart -> stable wrapper launcher
- start.sh: __task-processor internal launcher (export install_scripts_dir; exec)
- crontab_task_processor.sh: fix self-location ../.. -> .. for the new 1-level
depth (latent bug the move would otherwise have introduced)
- regen files_*/function_manifest; add task_scripts to the app/cli aggregates
- footprint_version 3 -> 4 (root-owned svc unit changed -> needs a root reinstall)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Extends the install-routing spike (e5273a4) to every long-running CLI
command, so CLI and WebUI now share one execution path everywhere:
app install ← already done
app uninstall
app start / stop / restart / up / down / reload
app backup
app restore
update apply
backup app create (matches `app backup` — same end target)
Each handler now has the same shape:
if [[ "$LIBREPORTAL_TASK_EXEC" == "1" ]]; then
<inline call> # processor's recursive invocation
else
cliTaskRun "<cmd>" <type> <app> # user invocation: enqueue + follow
fi
Processor change — crontab_task_processor.sh:
Adds `export LIBREPORTAL_TASK_EXEC=1` next to LIBREPORTAL_NONINTERACTIVE.
Universal bypass: every task command the processor runs (CLI-queued OR
pre-existing WebUI-queued like `libreportal app install adguard`)
inherits the env var, so the inline branch fires and we never
re-enqueue. This also lets us drop the env-var prefix the install spike
was baking into the command string (e5273a4) — cleaner task files +
one place to think about the bypass.
`backup app schedule` (the cron-driven path that already enqueues via
createTaskFile in backup_app_schedule.sh) is left alone — different
entry point, different runtime context, already correctly task-routed.
Why route the fast ones too (start/stop/restart/up/down):
Consistency beats the ~1s task-roundtrip latency for a CLI button.
Locking now serialises a CLI `app stop foo` against a WebUI restart of
the same app; the audit trail covers every state change. Cheap to
revert any individually if the latency turns out to bother someone.
Validated live earlier with `libreportal app install dashy` — task file
written, processor dispatched, follower streamed install live, exit 0
propagated. Same machinery now powers the other 9 handlers.
Signed-off-by: librelad <librelad@digitalangels.vip>
Add `lpRegen` (scripts/webui/webui_regen.sh) — one entry point that rebuilds the
file-derived artifacts whose sources changed, so callers don't have to know which
generator owns what. Self-heal is a cheap `find -newer` mtime compare (no watcher
/ daemon): a stage runs only when a source is newer than its artifact, or --force.
- `libreportal regen [all|webui|arrays] [--force]` CLI command (new category).
- Task processor idle tick runs a throttled `regen webui` poll, so an app dropped
in out-of-band (drag-drop / marketplace) appears on its own — no manual command,
no inotify (works on the relocatable/external-drive roots where inotify can't).
- make_release.sh guards against shipping stale source arrays (regenerate; abort
if the committed tree was out of date), killing the "forgot generate_arrays" bug
class at the build boundary.
- Document the front door in DEVELOPMENT.md.
webui scope rebuilds from containers/<app>/{*.config,tools/*.tools.json}; arrays
scope from scripts/** (a dev/build concern — a no-op on a normal install). Gate
logic verified in a sandbox (clean/config-newer/tools-newer/force/missing).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Introduce scripts/source/paths.sh as the canonical path resolver for three
independently-relocatable roots:
LP_SYSTEM_DIR manager-owned control plane (configs/logs/install/db/ssl/ssh/migrate)
LP_CONTAINERS_DIR container-user-owned live app data
LP_BACKUPS_DIR container-user-owned backup repos (own mount-able)
Roots come from the environment when set (install bakes them; CLI/app inherit
from init.sh), else default to /libreportal-*. A transitional compat default
keeps EXISTING installs (legacy single /docker tree, by config marker) on /docker
until a deliberate reinstall, so deploying this never strands a running box.
- init.sh derives the same vars inline (self-contained for the bare /root/init.sh
reinstall case); paths.sh mirrors it for the standalone task/check processors,
which now self-locate their scripts dir and source it.
- Replace functional /docker literals with the derived vars across runtime,
install, backup, crontab, crowdsec/restic, headscale, and reinstall paths;
clean the inert '== /docker/containers/*' guard fallbacks to the variable form.
- backend: CONTAINERS_DIR now from LP_CONTAINERS_DIR (compose env, filled at
generation via a new CONTAINERS_DIR_TAG), legacy-safe default for un-recreated
containers.
- backup default path falls back to the backups root; exclude paths.sh from the
sourced-file arrays (bootstrap file, sourced explicitly).
The CLI-wrapper heredoc + root helpers still reference /docker; those get baked
in phase 3. No layout/ownership change yet (phase 2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The scoped sudoers grants the manager (root) and (dockerinstall) but NOT
(itself), so the many 'sudo -u $sudo_user_name <cmd>' calls (crontab,
git/update, reinstall, swapfile, …) failed with 'a password is required'
once per CLI command. runAsManager runs the command plainly when already
the manager (the runtime case) and only sudo -u's when root (install
time), so it's correct in both contexts and needs no sudoers self-grant.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Under Model A the runtime runs as the manager, so establishing the
/docker ownership model needs root. Granting the manager a blanket
'sudo chown'/'sudo chmod' in the scoped sudoers would be root-equivalent
(chown /etc/sudoers, ...). Introduce a self-contained, root-owned helper
that performs only a FIXED set of reconciles on FIXED LibrePortal paths,
with owners derived from config + a baked manager name (never the caller)
and a strictly-validated app-name argument.
- scripts/system/libreportal-ownership: the helper (actions: reconcile,
traversal, containers-top, app-perms, webui, taskdir, app-data-nobody)
- run_privileged: runOwnership wrapper (sudo the installed helper; run the
bundled copy directly when already root mid-install)
- init.sh: installOwnershipHelper bakes the manager name and installs it
root:root 0755 to /usr/local/sbin (manager can't modify it)
- libreportal_folders/app_folder/app_update_specifics/task processor:
delegate the ownership chowns to runOwnership instead of runSystem chown
This removes chown/chmod-on-/docker from the runtime sudo surface, a
prerequisite for a non-root-equivalent scoped sudoers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
- copy_build_context: rsync/cp/rm -> runFileOp (writes the deployed tree AS the
container owner with --no-owner); drop the now-redundant runSystem chown.
- setup_lock: .setup_complete is in the docker-install-owned frontend/data ->
runFileOp touch/chmod/rm (drop the chown).
- tags_processor_docker_installation 'user:' enable + update_compose_yml
jail.local -> runFileOp (deployed compose/config under containers).
- crontab_clear: clear the manager's own crontab via runInstallOp.
- reinstall: cp init.sh to /root -> runSystem (genuine root path).
- create_successful_run_file: drop the pointless sudo echo -> runInstallWrite to
/docker/run.txt.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Model A prototype (run start.sh AS the manager, escalate only via helpers):
- check_root.sh: accept the manager user, not root-only (init.sh keeps its own
install-time root check).
- init.sh: guard the top-level root-check + installer entrypoint with
BASH_SOURCE!=$0 so it runs ONLY when init.sh is executed directly; when
start.sh sources it as the manager the entrypoint (and its root check) no
longer fires.
Also: convert bare daemon-touching 'docker' calls (no helper -> hit the
nonexistent /var/run socket in rootless) to runFileOp docker across
app_status, app_health_*, network_prune, ip_is_available, check_docker_network,
backup_db (db dumps) and crontab_check_processor. cd&&compose rooted-branches
and 'docker compose --version' checks left as-is (rooted-only / no daemon).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Foundation for a scoped sudoers: route every genuine system-admin command
(systemctl/ufw/ufw-docker/nft/apt/apt-get/pacman/sysctl/useradd/usermod/
service/wg/wg-quick/cscli/loginctl) through runSystem instead of raw sudo
across 28 active scripts. runSystem is 'sudo "$@"' so this is byte-identical
in every mode (safe on live installs) — it just collects all real-root use at
one chokepoint that will define the eventual /etc/sudoers.d allowlist.
Also: revert a crowdsec advice message the sweep wrongly rewrote (the admin
types sudo, not runSystem), and give crontab_check_processor.sh the same
startup bootstrap as the task processor — it runs standalone via cron and
already used runFileOp/runFileWrite (undefined there), so it was silently
broken; now it sources the helpers + docker-type config.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
The dir-ownership chown used runFileOp (the unprivileged dir owner), which
can't reclaim files a prior run left root/manager-owned — leaving a root-owned
task_processor.log the daemon then couldn't append to. Use runSystem (root) so
ownership is actually established.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
systemd launches the processor standalone, so it never sourced the LibrePortal
function library — runFileOp/runFileWrite were 'command not found' at runtime,
so it couldn't write its log, create its lock (flock died on a bad fd), or
update task status. Every task stayed queued and looped forever, and the setup
'finalize' never ran.
Source the privilege helpers (run_privileged.sh, docker_run_install.sh,
check_install_type.sh) + read the docker-type config at startup so runFileOp
knows rooted vs rootless. Also create the lock and per-task log via runFileOp
(world-writable) so the manager-user processor can open/append them in the
docker-install-owned task dir.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
updateTaskFields wrote its temp with a plain 'jq … > "$tmp"' shell redirect,
which runs as the processor's own user (the manager). But TASK_DIR is owned by
the docker install user and the manager can't create files in it, so the
redirect failed and the status write silently no-op'd — every task stayed
'queued', got reprocessed in an endless loop, and follow-on tasks (e.g. the
setup 'finalize' after 'config') never ran. The fix mirrors writeAtomic:
capture jq's output, write the temp through runFileWrite (the privileged
helper), then chmod + atomic mv.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
All operate on /docker data-plane (DB at $docker_dir, compose files,
task dir /docker/.../frontend/data/tasks): sqlite3/find/sed/mkdir/chmod/
chown/mv/rm/mkfifo/truncate/install/tee -> runFileOp/runFileWrite. The
two systemctl enable/start calls in the check processor -> runSystem.
Dropped spurious sudo on text-only echo/grep/date in db_app_scan.
Byte-identical in rooted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
On installs migrated from EasyDocker the spool file
/var/spool/cron/crontabs/<user> can be left owned by a defunct UID. The
sticky bit on the spool directory then blocks the current install user from
replacing it, so every `crontab -` write failed with
"rename: Operation not permitted" while the scripts still printed success.
crontabClear now removes the crontab as root (`crontab -u <user> -r`), which
bypasses the sticky bit and clears the stale file; the setup steps recreate
it owned by the install user, so the next crontab refresh self-heals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
Application backups were driven by one crontab entry per app, each offset by
id * CFG_BACKUP_CRONTAB_APP_INTERVAL minutes. That minute offset is written
straight into cron's 0-59 minute field, so past ~20 apps it overflowed into
an invalid entry that silently never fired, and the fixed spacing could not
serialize backups that ran longer than the gap.
Replace it with a single daily entry (`libreportal backup scheduled`) that
enqueues a backup task per enabled app. The existing systemd task processor
drains them serially — no minute overflow, real serialization, and backups
are now visible/cancellable in the Tasks UI. Per-app enable is read from
CFG_<APP>_BACKUP at schedule time instead of being mirrored into crontab.
Removes the stagger machinery (timing/setup/check/remove scripts), the
now-unused cron_jobs table + insert, and the CFG_BACKUP_CRONTAB_APP_INTERVAL
config knob and its WebUI field.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
A free, open, self-hosted app platform (GNU AGPLv3): one-click app deploys,
Traefik reverse proxy with automatic SSL, rootless Docker support, gluetun
VPN routing, and a web dashboard to manage it all.
Free & open forever to self-host; optional paid hosted services fund it.
See PROMISE.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: librelad <librelad@digitalangels.vip>
