Webstar/LibrePortal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backup: restore ownership through a multi-range userns map

Browse Source 
grafana restored and then died with "attempt to write a readonly database",
repeatedly. Its database is recorded in the snapshot as 231543:1002 and landed
as 1002:1002 — the owner was lost, so grafana, running as 231543, could not
write it at mode 0640.

Restore runs as the backup user with no CAP_CHOWN, so it reinstates ownership
inside a user namespace. The prefix was

    unshare --map-root-user --map-users=SUB:SUB:N --map-groups=SUB:SUB:N

and unshare accepts ONE range per option, so the backup user's own GID was never
mapped — while app data is written as <container-uid>:<backup-user>. The group
half of every such chown referred to an unmapped id, lchown returned EINVAL, and
the file kept the restoring user's ownership. restic reports those as "ignoring
error ..." and still exits 0, so nothing failed: 1626 of one 13-app restore's
2086 failed chowns were grafana's, under a restore that reported success.

restic-userns-exec uses newuidmap/newgidmap, which write the multi-range maps
unshare cannot express:

    uid:  0       <- caller     inner root, or caps are dropped at exec
          SUB..   <- SUB..      identity, so restic can name the stored uid
    gid:  caller  <- caller     identity: the group half of app-data chowns
          SUB..   <- SUB..      identity

The caller's own UID is deliberately not identity-mapped — that slot is spent on
inner root — and a file stored as <caller>:<caller> lands owned by the caller
anyway, because that is who inner root is outside. So the one case this cannot
map is the one case needing no mapping. `unshare --map-auto --map-current-user`
is not a shortcut: it maps the subuid range to low inner ids while restic needs
identity. Tested.

Measured live, restoring grafana: failed chowns 1626 -> 12 (the 12 being the
caller's own files, correct), grafana.db back to 231543:1002, grafana up and
writing. Falls back to running the command plainly when there is no subuid
range, no newuidmap, or the namespace will not start.

scripts/dev/lp-userns-ownership-test pins all three ownership cases; verified
the old prefix fails it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
librelad 2026-08-27 17:04:12 +01:00
parent 1fa36abb63
commit 17316b7fe2
4 changed files with 241 additions and 71 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

84
docs/roadmap/first-run-restore.md
View File

@ -200,63 +200,69 @@ stoat's livekit publishes a fixed UDP range (50000–50100) that it advertises t
clients and so cannot be re-rolled, and a desktop's `kdeconnectd` held 50016. clients and so cannot be re-rolled, and a desktop's `kdeconnectd` held 50016.
Worth a fixed-range preflight check of its own; see §9. Worth a fixed-range preflight check of its own; see §9.
### 3.5 — Open: ownership is not actually being reinstated ### 3.5 — Ownership was not actually being reinstated
Found by the same run and **not fixed** — it is a different subsystem (the `grafana` restored and then died with *"attempt to write a readonly database"*,
backup engine's ownership handling), it predates all of the above, and the repeatedly. The snapshot records:
sensible fixes are security-relevant enough to want a decision first.
`grafana` restored and then died with *"attempt to write a readonly database"*.
The snapshot records:
``` ```
-rw-r----- 231543 1002 /libreportal-containers/grafana/grafana_storage/grafana.db -rw-r----- 231543 1002 /libreportal-containers/grafana/grafana_storage/grafana.db
``` ```
and what landed on disk was `1002:1002`. The owner was lost, so grafana — and what landed was `1002:1002`. The owner was lost, so grafana — running as
running as 231543 — cannot write its own database at mode 0640. 231543 — could not write its own database at mode 0640.
The cause is exact. `_resticUsernsPrefix` runs restic under Restore runs as the backup user, which has no `CAP_CHOWN`, so it reinstates
ownership through a user namespace. The prefix was:
``` ```
unshare --map-root-user --map-users=231072:231072:65536 --map-groups=231072:231072:65536 unshare --map-root-user --map-users=231072:231072:65536 --map-groups=231072:231072:65536
``` ```
and `unshare` accepts **one range per option**. So the backup user's own GID and `unshare` accepts **one range per option**. So the backup user's own GID was
(1002) is never mapped, while LibrePortal writes app data as never mapped — while LibrePortal writes app data as `<container-uid>:<backup-user>`.
`<container-uid>:<backup-user>`. Verified directly: The group half of every such chown referred to an unmapped id, `lchown` returned
EINVAL, and the file kept the restoring user's ownership. Measured directly:
| target | result | | target | old prefix | now |
|---|---| |---|---|---|
| `231543:231543` (both in the subuid range) | ✅ applied | | `231543:1002` (app data) | ❌ → `1002:1002` | ✅ `231543:1002` |
| `231543:1002` (the real case) | ❌ EINVAL → falls back to `1002:1002` | | `1002:1002` (LibrePortal's own) | ❌ → `1002:1002` (right anyway) | ❌ → `1002:1002` (right anyway) |
| `1002:1002` (LibrePortal's own files) | ❌ EINVAL → `1002:1002`, which is correct anyway | | `231072:231072` (container root) | ✅ | ✅ |
1626 of the 2086 failed chowns in one 13-app restore were grafana's. Same shape as the rest of §3: restic reports the misses as `ignoring error …`
and **still exits 0**, and the caller forgave them as *"expected, they are
already owned correctly"*. 1626 of one 13-app restore's 2086 failed chowns were
grafana's.
`resticRestoreErrorsAreBenign` forgave all of them and reported *"expected, they **Fixed** with `scripts/backup/engine/restic-userns-exec`, which uses
are already owned correctly"* — true only for the third row. That message is now `newuidmap`/`newgidmap` — those write multi-range maps, which is exactly what
honest about what was not reinstated, which is the only part fixed here. `unshare` cannot express:
Ways out, none free: ```
uid: 0 <- caller inner root, or capabilities are dropped at exec
SUB.. <- SUB.. identity, so restic can name the stored uid
gid: caller <- caller identity — the group half of app-data chowns
SUB.. <- SUB.. identity
```
1. **`newuidmap`/`newgidmap`** can write multi-range maps (`1002 1002 1` plus The caller's own UID is deliberately not identity-mapped: that slot is spent on
`231072 231072 65536`), which is exactly what is needed — but it means inner root. A file stored as `<caller>:<caller>` therefore fails its chown and
forking, mapping from outside, then continuing, rather than a one-line lands owned by the caller anyway, because that is who inner root is on the
prefix. `unshare --map-auto --map-current-user` does **not** work: it maps outside — so the one case this cannot map is the one case that needs no mapping.
the subuid range to low inner ids (container-style, `0 → 231072`), while
restic needs identity.
2. **Restore as real root via a helper**, the way `app-adopt` already moves
trees, with the target path validated against the storage registry. Simplest
and most reliable; widens what root does on the manager's say-so.
3. **Reapply ownership afterwards** from `restic ls -l`, via a root helper.
Contained, but a second pass over every file.
Recommendation: (1) if it can be kept small, else (2) with the same path `unshare --map-auto --map-current-user` does **not** work as a shortcut: it maps
validation `app-adopt` uses. Worth noting that apps whose data is captured into the subuid range to low inner ids (container-style, `0 → 231072`) while restic
`.lp-backup/files` are rehydrated "via container" as the right uid and so needs identity. Tested.
survive regardless — which is why bookstack, matrix, mattermost and nextcloud
came back healthy and grafana did not. Measured on the live install, restoring grafana: failed chowns **1626 → 12**
(the 12 being the caller's own files, which are correct), `grafana.db` back to
`231543:1002`, and grafana up and writing. `scripts/dev/lp-userns-ownership-test`
pins all three rows of the table above.
Anything unexpected — no subuid range, no `newuidmap`, a namespace that will not
start — falls back to running the command plainly, which is what happened before
any of this existed.
## 4. The password problem, stated plainly ## 4. The password problem, stated plainly

105
scripts/backup/engine/restic-userns-exec Executable file
View File

@ -0,0 +1,105 @@
#!/bin/bash
# Run a command in a user namespace whose id maps let it restore file ownership.
#
# restic-userns-exec restic restore <id> --target …
#
# Restore runs as the backup user, which has no CAP_CHOWN, so it cannot put back
# the uid a file had when it was backed up. A user namespace solves that — but
# only if every id involved is mapped, and LibrePortal needs three mappings at
# once while `unshare` accepts exactly ONE range per option.
#
# That limit is why the previous one-line prefix silently did not work:
#
# unshare --map-root-user --map-users=SUB:SUB:N --map-groups=SUB:SUB:N
#
# mapped root and the subuid range, but never the backup user's own GID. App
# data is written as <container-uid>:<backup-user>, so the group half of every
# chown referred to an unmapped id, lchown returned EINVAL, and the file kept
# the restoring user's ownership. Grafana's database came back 1002:1002 instead
# of 231543:1002 and the app died with "attempt to write a readonly database" —
# under a restore that reported success.
#
# newuidmap/newgidmap can write multi-range maps, so the maps become:
#
# uid: 0 <- caller (inner root: without this, capabilities are
# dropped at exec and chown is not permitted)
# SUB.. <- SUB.. (identity, so restic can ask for the stored
# container uid by its real number)
# gid: caller <- caller (identity: the group half of app-data chowns)
# SUB.. <- SUB.. (identity)
#
# The caller's own UID is deliberately NOT identity-mapped: it is spent on inner
# root. A file stored as <caller>:<caller> therefore fails its chown — and lands
# owned by the caller anyway, because that is who inner root is outside. So the
# one case this cannot map is the one case that needs no mapping.
#
# Anything unexpected — no subuid range, no newuidmap, a namespace that will not
# start — falls back to running the command plainly, which is what happened
# before this existed.
set -u
(( $# )) || { echo "restic-userns-exec: no command given" >&2; exit 2; }
TMPDIR_MADE=""
_plain() { [[ -n "$TMPDIR_MADE" ]] && rm -rf "$TMPDIR_MADE"; exec "$@"; }
for _bin in unshare newuidmap newgidmap; do
command -v "$_bin" >/dev/null 2>&1 || _plain "$@"
done
_usr=$(id -un 2>/dev/null) || _plain "$@"
_u=$(id -u); _g=$(id -g)
_uline=$(grep "^${_usr}:" /etc/subuid 2>/dev/null | head -1)
_gline=$(grep "^${_usr}:" /etc/subgid 2>/dev/null | head -1)
[[ -n "$_uline" && -n "$_gline" ]] || _plain "$@"
_ustart="${_uline#*:}"; _ustart="${_ustart%%:*}"; _ucount="${_uline##*:}"
_gstart="${_gline#*:}"; _gstart="${_gstart%%:*}"; _gcount="${_gline##*:}"
[[ "$_ustart" =~ ^[0-9]+$ && "$_ucount" =~ ^[0-9]+$ ]] || _plain "$@"
[[ "$_gstart" =~ ^[0-9]+$ && "$_gcount" =~ ^[0-9]+$ ]] || _plain "$@"
(( _ucount > 0 && _gcount > 0 )) || _plain "$@"
# The caller's own id has to sit outside its sub-range, or the two entries would
# overlap on the outer side and the kernel rejects the whole map.
(( _u < _ustart || _u >= _ustart + _ucount )) || _plain "$@"
(( _g < _gstart || _g >= _gstart + _gcount )) || _plain "$@"
TMPDIR_MADE=$(mktemp -d "${TMPDIR:-/tmp}/lp-userns.XXXXXX") || { TMPDIR_MADE=""; _plain "$@"; }
_fifo="$TMPDIR_MADE/gate"
mkfifo -m 600 "$_fifo" 2>/dev/null || _plain "$@"
# Opened here and inherited there, rather than opened by path in the child:
# until the map is written the child's uid is unmapped, so to the filesystem it
# is nobody and could not open its own gate.
exec 9<>"$_fifo"
unshare --user bash -c 'read -r _ <&9; exec "$@"' _ "$@" &
_child=$!
# The namespace exists before unshare execs, but only just — retry briefly
# rather than assume. A write that succeeds cannot be repeated, so stop there.
_mapped=0
for _ in $(seq 1 100); do
kill -0 "$_child" 2>/dev/null || break
if newuidmap "$_child" 0 "$_u" 1 "$_ustart" "$_ustart" "$_ucount" 2>/dev/null; then
_mapped=1; break
fi
sleep 0.02
done
if (( _mapped )); then
newgidmap "$_child" "$_g" "$_g" 1 "$_gstart" "$_gstart" "$_gcount" 2>/dev/null || _mapped=0
fi
if (( ! _mapped )); then
kill "$_child" 2>/dev/null
wait "$_child" 2>/dev/null
_plain "$@"
fi
printf 'go\n' >&9
wait "$_child"
_rc=$?
rm -rf "$TMPDIR_MADE"
exit "$_rc"

55
scripts/backup/engine/restic_restore.sh
View File

@ -28,23 +28,23 @@
_resticUsernsPrefix() _resticUsernsPrefix()
{ {
local usr="${docker_install_user:-dockerinstall}" local usr="${docker_install_user:-dockerinstall}"
command -v unshare >/dev/null 2>&1 || return 0
local uline gline ustart ucount gstart gcount # The mapping itself lives in restic-userns-exec, because it needs three id
uline=$(grep "^${usr}:" /etc/subuid 2>/dev/null | head -1) # ranges at once and `unshare` takes one per option — see that file. The
gline=$(grep "^${usr}:" /etc/subgid 2>/dev/null | head -1) # checks here only decide whether to reach for it at all; it re-checks and
# falls back to running the command plainly if anything is missing.
local helper="${install_scripts_dir%/}/backup/engine/restic-userns-exec"
[[ -r "$helper" ]] || return 0
command -v unshare >/dev/null 2>&1 || return 0
command -v newuidmap >/dev/null 2>&1 || return 0
command -v newgidmap >/dev/null 2>&1 || return 0
# No subuid range (rooted mode, or a hand-rolled account) — nothing to map, # No subuid range (rooted mode, or a hand-rolled account) — nothing to map,
# so leave the call exactly as it was rather than guess. # so leave the call exactly as it was rather than guess.
[[ -n "$uline" && -n "$gline" ]] || return 0 grep -q "^${usr}:" /etc/subuid 2>/dev/null || return 0
grep -q "^${usr}:" /etc/subgid 2>/dev/null || return 0
ustart="${uline#*:}"; ustart="${ustart%%:*}"; ucount="${uline##*:}" printf '%s\n' bash "$helper"
gstart="${gline#*:}"; gstart="${gstart%%:*}"; gcount="${gline##*:}"
[[ "$ustart" =~ ^[0-9]+$ && "$ucount" =~ ^[0-9]+$ ]] || return 0
[[ "$gstart" =~ ^[0-9]+$ && "$gcount" =~ ^[0-9]+$ ]] || return 0
printf '%s\n' unshare --map-root-user \
"--map-users=${ustart}:${ustart}:${ucount}" \
"--map-groups=${gstart}:${gstart}:${gcount}"
} }
# True when every error restic reported is the expected "cannot map the backup # True when every error restic reported is the expected "cannot map the backup
@ -93,30 +93,21 @@ resticRestoreSnapshot()
printf '%s\n' "$out" printf '%s\n' "$out"
# restic exits non-zero for un-mappable-uid lchowns even though the file # restic exits non-zero for un-mappable-uid lchowns even though the file
# CONTENTS landed. Forgive only that case — but do not pretend it is # CONTENTS landed. Forgive only that case.
# nothing, which is what this used to do.
# #
# It said "expected, they are already owned correctly". That holds for # With the namespace mapping fixed (see restic-userns-exec), the only id
# LibrePortal's own files (owner = the backup user, which is who restic runs # that is still unmappable is the restoring user's own — its slot is spent
# as anyway). It does NOT hold for container-owned data, and the namespace # on inner root — and a file stored as <caller>:<caller> lands owned by the
# this runs in cannot currently map those: unshare takes a single range per # caller regardless, because that is who inner root is on the outside. So
# option, so _resticUsernsPrefix maps the subuid range and root but not the # these really are correct, which is what the message used to claim before
# backup user's own GID — and LibrePortal writes app data as # the mapping worked and container-owned data was quietly losing its owner.
# <container-uid>:<backup-user>. Every such chown fails and the file falls
# back to <backup-user>:<backup-user>.
# #
# Observed: grafana's grafana.db is recorded in the snapshot as 231543:1002 # Still report the count: if this number is large the mapping has stopped
# and restored as 1002:1002. At mode 0640 the grafana process — running as # working again, and the symptom is an app that cannot write its own data.
# 231543 — then cannot write it, and the app dies with "attempt to write a
# readonly database". The restore reported success.
#
# So: still do not fail the restore (the data is there and some apps are
# rehydrated by other means), but say plainly what was not reinstated.
if [[ $rc -ne 0 && ${#ns_prefix[@]} -gt 0 ]] && resticRestoreErrorsAreBenign "$out"; then if [[ $rc -ne 0 && ${#ns_prefix[@]} -gt 0 ]] && resticRestoreErrorsAreBenign "$out"; then
local _lch local _lch
_lch=$(printf '%s\n' "$out" | grep -cE "^ignoring error for .*lchown ") _lch=$(printf '%s\n' "$out" | grep -cE "^ignoring error for .*lchown ")
isNotice "Restore could not reinstate ownership on ${_lch} file(s); they now belong to ${docker_install_user:-the backup user}." isNotice "Ownership warnings on ${_lch} file(s) owned by ${docker_install_user:-the backup user} — expected; they are restored correctly."
isNotice "LibrePortal's own files are correct that way. Container-owned data is NOT — an app may fail to write (e.g. a read-only database). Check the app after it starts."
rc=0 rc=0
fi fi

68
scripts/dev/lp-userns-ownership-test Executable file
View File

@ -0,0 +1,68 @@
#!/bin/bash
# Can a restore actually put back the uid a file had when it was backed up?
#
# scripts/dev/lp-userns-ownership-test # as the backup user
# sudo -u dockerinstall -H scripts/dev/lp-userns-ownership-test
#
# Restore runs unprivileged, so it reinstates ownership through a user namespace
# (scripts/backup/engine/restic-userns-exec). The previous one-line `unshare`
# prefix mapped root and the subuid range but not the backup user's own GID —
# and app data is stored as <container-uid>:<backup-user>, so the group half of
# every chown was unmapped, lchown returned EINVAL, and the file silently kept
# the restoring user's ownership.
#
# Nothing failed. restic reported the misses as "ignoring error …" and exited 0,
# and the caller forgave them as expected. The only symptom was grafana coming
# back as 1002:1002 instead of 231543:1002 and dying with "attempt to write a
# readonly database".
#
# So the case that matters is the middle one: a container uid with the backup
# user as its group.
REPO="$(cd "$(dirname "$0")/../.." && pwd)"
HELPER="$REPO/scripts/backup/engine/restic-userns-exec"
fail=0
chk(){ if [[ "$2" == "$3" ]]; then echo " ok $1"; else echo " FAIL $1: got '$2' want '$3'"; fail=1; fi; }
[[ -r "$HELPER" ]] || { echo " FAIL helper not found at $HELPER"; exit 1; }
for b in unshare newuidmap newgidmap; do
command -v "$b" >/dev/null 2>&1 || { echo " SKIP $b not installed"; exit 0; }
done
usr=$(id -un); u=$(id -u); g=$(id -g)
uline=$(grep "^${usr}:" /etc/subuid 2>/dev/null | head -1)
[[ -n "$uline" ]] || { echo " SKIP no subuid range for $usr"; exit 0; }
sub="${uline#*:}"; sub="${sub%%:*}"
CUID=$(( sub + 471 )) # a container uid, the way rootless docker maps one
BASE=$(mktemp -d "${TMPDIR:-/tmp}/lp-userns-test-XXXXXX") || exit 1
trap 'rm -rf "$BASE"' EXIT
echo "--- $usr ($u:$g), subuid base $sub, container uid $CUID ---"
# Ask the helper to set the three ownerships a restore actually encounters.
bash "$HELPER" bash -c '
: > "$1/app"; : > "$1/own"; : > "$1/croot"
chown '"$CUID"':'"$g"' "$1/app" 2>/dev/null
chown '"$u"':'"$g"' "$1/own" 2>/dev/null
chown '"$sub"':'"$sub"' "$1/croot" 2>/dev/null
exit 0' _ "$BASE" >/dev/null 2>&1
echo "--- container-owned app data (the grafana case) ---"
chk "uid:gid" "$(stat -c '%u:%g' "$BASE/app")" "$CUID:$g"
echo "--- LibrePortal's own files ---"
# The caller's uid is spent on inner root, so this chown cannot succeed — and
# does not need to: inner root IS the caller on the outside.
chk "uid:gid" "$(stat -c '%u:%g' "$BASE/own")" "$u:$g"
echo "--- container root ---"
chk "uid:gid" "$(stat -c '%u:%g' "$BASE/croot")" "$sub:$sub"
echo "--- exit status and stdout are passed through ---"
out=$(bash "$HELPER" bash -c 'echo hello; exit 7' 2>/dev/null); rc=$?
chk "stdout" "$out" "hello"
chk "status" "$rc" "7"
echo ""
if (( fail )); then echo "FAILED"; exit 1; fi
echo "All ownership-mapping checks passed."