grafana restored and then died with "attempt to write a readonly database",
repeatedly. Its database is recorded in the snapshot as 231543:1002 and landed
as 1002:1002 — the owner was lost, so grafana, running as 231543, could not
write it at mode 0640.
Restore runs as the backup user with no CAP_CHOWN, so it reinstates ownership
inside a user namespace. The prefix was
unshare --map-root-user --map-users=SUB:SUB:N --map-groups=SUB:SUB:N
and unshare accepts ONE range per option, so the backup user's own GID was never
mapped — while app data is written as <container-uid>:<backup-user>. The group
half of every such chown referred to an unmapped id, lchown returned EINVAL, and
the file kept the restoring user's ownership. restic reports those as "ignoring
error ..." and still exits 0, so nothing failed: 1626 of one 13-app restore's
2086 failed chowns were grafana's, under a restore that reported success.
restic-userns-exec uses newuidmap/newgidmap, which write the multi-range maps
unshare cannot express:
uid: 0 <- caller inner root, or caps are dropped at exec
SUB.. <- SUB.. identity, so restic can name the stored uid
gid: caller <- caller identity: the group half of app-data chowns
SUB.. <- SUB.. identity
The caller's own UID is deliberately not identity-mapped — that slot is spent on
inner root — and a file stored as <caller>:<caller> lands owned by the caller
anyway, because that is who inner root is outside. So the one case this cannot
map is the one case needing no mapping. `unshare --map-auto --map-current-user`
is not a shortcut: it maps the subuid range to low inner ids while restic needs
identity. Tested.
Measured live, restoring grafana: failed chowns 1626 -> 12 (the 12 being the
caller's own files, correct), grafana.db back to 231543:1002, grafana up and
writing. Falls back to running the command plainly when there is no subuid
range, no newuidmap, or the namespace will not start.
scripts/dev/lp-userns-ownership-test pins all three ownership cases; verified
the old prefix fails it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
167 lines
6.9 KiB
Bash
167 lines
6.9 KiB
Bash
#!/bin/bash
|
|
|
|
# Build the `unshare` prefix that lets a NON-ROOT restic recreate the container
|
|
# uids a snapshot recorded.
|
|
#
|
|
# Why this is needed: backups run as the docker install user (runBackupOp — the
|
|
# backup engine never gets root). A non-root restic cannot chown a restored file
|
|
# to anyone else, so every file came back owned by that user. For LibrePortal's
|
|
# own files that is correct; for the ones a CONTAINER owns it is fatal. Under
|
|
# rootless, a container process running as uid N appears on the host as
|
|
# subuid_start + N - 1 (prometheus' nobody -> 296605, postgres -> 231141), and an
|
|
# app whose data dir is no longer owned by its own uid does not start:
|
|
# prometheus dies on "open data/queries.active: permission denied", and postgres
|
|
# refuses outright unless its data dir is 0700 and its own. Restores therefore
|
|
# handed back apps that could not boot.
|
|
#
|
|
# The fix needs no new privilege. The docker install user already owns a subuid
|
|
# range (that is what makes rootless work), so it may enter a user namespace in
|
|
# which it is root and those subuids are mappable. Mapping them to THEMSELVES
|
|
# means an id recorded in the snapshot is written back as the same host id.
|
|
#
|
|
# Files recorded as the docker install user's own uid are the one gap: that uid
|
|
# is outside the subuid range and is already consumed by the inner-root mapping,
|
|
# so restic's lchown for them fails with EINVAL. It is harmless — restic runs as
|
|
# inner root, which IS that user on the host, so those files already land with
|
|
# exactly the right owner. resticRestoreErrorsAreBenign below is what keeps that
|
|
# from being reported as a failed restore.
|
|
_resticUsernsPrefix()
|
|
{
|
|
local usr="${docker_install_user:-dockerinstall}"
|
|
|
|
# The mapping itself lives in restic-userns-exec, because it needs three id
|
|
# ranges at once and `unshare` takes one per option — see that file. The
|
|
# checks here only decide whether to reach for it at all; it re-checks and
|
|
# falls back to running the command plainly if anything is missing.
|
|
local helper="${install_scripts_dir%/}/backup/engine/restic-userns-exec"
|
|
[[ -r "$helper" ]] || return 0
|
|
command -v unshare >/dev/null 2>&1 || return 0
|
|
command -v newuidmap >/dev/null 2>&1 || return 0
|
|
command -v newgidmap >/dev/null 2>&1 || return 0
|
|
|
|
# No subuid range (rooted mode, or a hand-rolled account) — nothing to map,
|
|
# so leave the call exactly as it was rather than guess.
|
|
grep -q "^${usr}:" /etc/subuid 2>/dev/null || return 0
|
|
grep -q "^${usr}:" /etc/subgid 2>/dev/null || return 0
|
|
|
|
printf '%s\n' bash "$helper"
|
|
}
|
|
|
|
# True when every error restic reported is the expected "cannot map the backup
|
|
# user's own uid" one described above. Anything else — a missing pack, a full
|
|
# disk, a permission problem on the target — must still fail the restore.
|
|
resticRestoreErrorsAreBenign()
|
|
{
|
|
local out="$1"
|
|
local bad
|
|
# Every line restic prints for a failed ownership set, minus the benign form.
|
|
bad=$(printf '%s\n' "$out" | grep -E "^ignoring error for " \
|
|
| grep -vE "lchown .*: (invalid argument|operation not permitted)$")
|
|
[[ -z "$bad" ]]
|
|
}
|
|
|
|
resticRestoreSnapshot()
|
|
{
|
|
local idx="$1"
|
|
local snapshot_id="$2"
|
|
local target_dir="$3"
|
|
local include_path="$4"
|
|
|
|
if [[ -z "$snapshot_id" || -z "$target_dir" ]]; then
|
|
isError "resticRestoreSnapshot requires snapshot_id and target_dir"
|
|
return 1
|
|
fi
|
|
|
|
resticEnvExport "$idx" || return 1
|
|
|
|
runFileOp mkdir -p "$target_dir"
|
|
|
|
local args=(restore "$snapshot_id" --target "$target_dir")
|
|
[[ -n "$include_path" ]] && args+=(--include "$include_path")
|
|
|
|
isNotice "Restoring ${snapshot_id:0:8} from $(resticLocationName "$idx") → $target_dir"
|
|
|
|
local ns_prefix=()
|
|
mapfile -t ns_prefix < <(_resticUsernsPrefix)
|
|
|
|
# Output is captured (not streamed) so the benign-error check below can read
|
|
# it; it is echoed straight back afterwards, so the operator sees the same
|
|
# restic report as before.
|
|
local out rc
|
|
out=$(runBackupOp "${ns_prefix[@]}" restic "${args[@]}" 2>&1)
|
|
rc=$?
|
|
printf '%s\n' "$out"
|
|
|
|
# restic exits non-zero for un-mappable-uid lchowns even though the file
|
|
# CONTENTS landed. Forgive only that case.
|
|
#
|
|
# With the namespace mapping fixed (see restic-userns-exec), the only id
|
|
# that is still unmappable is the restoring user's own — its slot is spent
|
|
# on inner root — and a file stored as <caller>:<caller> lands owned by the
|
|
# caller regardless, because that is who inner root is on the outside. So
|
|
# these really are correct, which is what the message used to claim before
|
|
# the mapping worked and container-owned data was quietly losing its owner.
|
|
#
|
|
# Still report the count: if this number is large the mapping has stopped
|
|
# working again, and the symptom is an app that cannot write its own data.
|
|
if [[ $rc -ne 0 && ${#ns_prefix[@]} -gt 0 ]] && resticRestoreErrorsAreBenign "$out"; then
|
|
local _lch
|
|
_lch=$(printf '%s\n' "$out" | grep -cE "^ignoring error for .*lchown ")
|
|
isNotice "Ownership warnings on ${_lch} file(s) owned by ${docker_install_user:-the backup user} — expected; they are restored correctly."
|
|
rc=0
|
|
fi
|
|
|
|
resticEnvUnset
|
|
return $rc
|
|
}
|
|
|
|
resticRestoreAppLatest()
|
|
{
|
|
local idx="$1"
|
|
local app_name="$2"
|
|
local target_dir="$3"
|
|
local host="${4:-$CFG_INSTALL_NAME}"
|
|
|
|
local snapshot_id
|
|
snapshot_id=$(resticSnapshotLatestId "$idx" "$app_name" "$host")
|
|
|
|
if [[ -z "$snapshot_id" ]]; then
|
|
isError "No snapshot found in $(resticLocationName "$idx") for app=$app_name host=$host"
|
|
return 1
|
|
fi
|
|
|
|
# Prefer the path the SNAPSHOT records over this host's layout: they differ
|
|
# whenever the snapshot came from a host with a different --containers-dir,
|
|
# or from a different storage location, and an include filter that matches
|
|
# nothing restores nothing without saying so.
|
|
local include_path=""
|
|
if declare -f storageSnapshotSourcePath >/dev/null 2>&1; then
|
|
include_path=$(storageSnapshotSourcePath "$idx" "$snapshot_id" "$app_name" 2>/dev/null) || include_path=""
|
|
fi
|
|
[[ -z "$include_path" ]] && include_path="$(appDir "$app_name")"
|
|
resticRestoreSnapshot "$idx" "$snapshot_id" "$target_dir" "$include_path"
|
|
}
|
|
|
|
resticRestoreSystemLatest()
|
|
{
|
|
local idx="$1"
|
|
local target_dir="$2"
|
|
local host="${3:-$CFG_INSTALL_NAME}"
|
|
|
|
resticEnvExport "$idx" || return 1
|
|
local snapshot_id
|
|
snapshot_id=$(runBackupOp restic snapshots \
|
|
--tag "system=config" --host "$host" \
|
|
--latest 1 --json --no-lock 2>/dev/null | \
|
|
grep -o '"short_id":"[^"]*"' | head -1 | cut -d'"' -f4)
|
|
resticEnvUnset
|
|
|
|
if [[ -z "$snapshot_id" ]]; then
|
|
isError "No system-config snapshot found in $(resticLocationName "$idx") for host=$host"
|
|
return 1
|
|
fi
|
|
|
|
# Whole-snapshot restore (the snapshot is just the config tree) into staging.
|
|
resticRestoreSnapshot "$idx" "$snapshot_id" "$target_dir"
|
|
}
|