0e98988fcb
Prometheus kept being found stopped after boots, always Exited(0),
always alone. The journal settles it: both stops sit seconds before a
host shutdown boundary — container stopped 05:45:22, boot ended
05:45:30; stopped 04:41:59, boot ended 04:42:05. This is a laptop-class
host that gets shut down, and under ROOTLESS docker the containers are
ordinary processes in the user session, torn down by systemd in
parallel with dockerd's own exit.
That parallelism is the race. An app that handles SIGTERM promptly
exits while dockerd is still alive to record "stopped" — and
unless-stopped then means what it says: not restarted at the next
boot. Apps that exit slower, or die only when dockerd does, are
recorded as running and come back. Prometheus loses reliably because it
is the best-behaved process on the box ("See you next time!"), but
which app loses is a scheduling accident — changing Prometheus's
restart policy would treat the sample, not the race.
So an @reboot crontab entry now waits for the rootless daemon (up to
five minutes, then gives up rather than hang) and `compose up -d`s
every installed app via the existing dockerComposeUpAllApps. Idempotent:
running apps see no diff, stopped ones start, ordering is compose's
problem. Registered through crontabRefresh like the other entries, and
installed on this box.
The accepted trade, stated rather than hidden: an app deliberately
stopped before a reboot comes back after it. On a self-hosting box "the
fleet is up after boot" is the promise unless-stopped was already trying
to make; a stop that must survive reboots is what uninstall is for.
Verified by direct execution: daemon answered immediately, all
installed apps reconciled, running containers untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
81 lines
4.2 KiB
Bash
81 lines
4.2 KiB
Bash
#!/bin/bash
|
|
|
|
# Boot-time app reconcile — bring every installed app's containers back up.
|
|
# ---------------------------------------------------------------------------
|
|
# WHY THIS EXISTS. Every app runs with restart:unless-stopped, and on a machine
|
|
# that shuts down (this is a laptop-class host, not a rack server) that policy
|
|
# holds a race it can lose. Under ROOTLESS docker the containers are ordinary
|
|
# processes in the user's session, and at shutdown systemd tears that session
|
|
# down in parallel with dockerd's own exit. An app that handles SIGTERM
|
|
# promptly — Prometheus is the best-behaved process on the box — exits while
|
|
# dockerd is still alive to record "stopped", and unless-stopped then means
|
|
# exactly what it says: not restarted at the next boot. Apps that exit slower,
|
|
# or only die when dockerd itself does, are recorded as running and come back.
|
|
#
|
|
# Observed twice, same victim both times: host shutdown 05:45:30, Prometheus
|
|
# stopped 05:45:22; host shutdown 04:42:05, Prometheus stopped 04:41:59. Eight
|
|
# and six seconds ahead of the teardown — first over the line, only loser.
|
|
# Which app loses is a scheduling accident; the graceful ones are simply the
|
|
# most likely, so "fix Prometheus's policy" would treat the sample, not the
|
|
# race.
|
|
#
|
|
# So at boot, once the rootless daemon answers, `compose up -d` every installed
|
|
# app. That is idempotent: running apps are untouched (compose sees no diff),
|
|
# stopped ones start, and dependency order is compose's problem, not ours.
|
|
#
|
|
# The one behavioural trade: an app someone deliberately stopped BEFORE
|
|
# rebooting comes back after the reboot. That is accepted — on a self-hosting
|
|
# box "the fleet is up after boot" is the promise the restart policy was
|
|
# already trying to make, and a stop that must survive reboots is what
|
|
# uninstall (or disabling the app) is for.
|
|
|
|
script_boot_flag="$1"
|
|
|
|
# Only run when executed directly, not when sourced (mirrors the task
|
|
# processor's guard so sourcing this file for its functions stays side-effect
|
|
# free).
|
|
if [[ "$script_boot_flag" == "start_script" ]]; then
|
|
|
|
# --- Bootstrap: cron runs this standalone, same dance as the task processor --
|
|
LP_SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)"
|
|
LP_SCRIPTS="${install_scripts_dir:-$(cd "$LP_SELF_DIR/../../.." 2>/dev/null && pwd)/scripts/}"
|
|
[[ -f "${LP_SCRIPTS}source/paths.sh" ]] && source "${LP_SCRIPTS}source/paths.sh"
|
|
LP_SCRIPTS="${install_scripts_dir:-$LP_SCRIPTS}"
|
|
LP_DOCKER_CFG="${configs_dir:-/libreportal-system/configs/}general/general_docker_install"
|
|
[[ -f "$LP_DOCKER_CFG" ]] && \
|
|
eval "$(grep -E '^CFG_DOCKER_INSTALL_(TYPE|USER)=' "$LP_DOCKER_CFG" | sed 's/[[:space:]]*#.*//')"
|
|
: "${sudo_user_name:=libreportal}"
|
|
: "${containers_dir:=/libreportal-containers/}"
|
|
: "${docker_dir:=/libreportal-system}"
|
|
for _lp_f in docker/command/run_privileged.sh \
|
|
docker/command/docker_run_install.sh \
|
|
checks/requirements/check_install_type.sh \
|
|
source/files/arrays/function_manifest.sh; do
|
|
[[ -f "${LP_SCRIPTS}${_lp_f}" ]] && source "${LP_SCRIPTS}${_lp_f}"
|
|
done
|
|
command -v resolveDockerInstallUser >/dev/null 2>&1 && resolveDockerInstallUser
|
|
|
|
# Minimal logging shims when the full CLI helpers are not loaded.
|
|
command -v isNotice >/dev/null 2>&1 || isNotice() { echo "[boot-reconcile] $*"; }
|
|
command -v isSuccessful >/dev/null 2>&1 || isSuccessful() { echo "[boot-reconcile] $*"; }
|
|
command -v isError >/dev/null 2>&1 || isError() { echo "[boot-reconcile] $*" >&2; }
|
|
|
|
# --- Wait for the rootless daemon; it starts with the user session and can be
|
|
# a while behind cron's @reboot. Give up after 5 minutes rather than hang a
|
|
# boot-scoped job forever — the next manual `libreportal start` still works.
|
|
_lp_waited=0
|
|
until dockerCommandRun "docker info" >/dev/null 2>&1; do
|
|
sleep 5
|
|
_lp_waited=$(( _lp_waited + 5 ))
|
|
if (( _lp_waited >= 300 )); then
|
|
isError "Docker daemon not up after ${_lp_waited}s — skipping boot app reconcile."
|
|
exit 0
|
|
fi
|
|
done
|
|
|
|
isNotice "Docker up after ~${_lp_waited}s — reconciling installed apps."
|
|
declare -F dockerComposeUpAllApps >/dev/null 2>&1 && dockerComposeUpAllApps >> "${docker_dir}/logs/boot_reconcile.log" 2>&1
|
|
isSuccessful "Boot app reconcile finished."
|
|
|
|
fi
|