Third occurrence of the same defect, in a third directory. config-adopt gave a file the destination did not already have 0640 — "these can hold secrets, so default closed" — but the config tree is 0755 and the backup account reads all of it. One 0640 file makes restic report permission denied, write an INCOMPLETE snapshot and exit 3, so the entire system-config backup is reported failed, and a first-run restore has nothing to restore from. A new file now inherits from a sibling in the same directory, matching whatever that tree's convention is rather than having a mode picked for it. Since this has now happened three times for three unrelated reasons, the test asserts the class rather than the instances: no file anywhere under configs/ may be unreadable by the backup account. Mutation-checked — chmod 0640 on any one config makes it fail. Also repaired the files the earlier buggy adoption runs clamped on this install, and fixed lp-storage-step-test's hardcoded step index, which the new Start step had shifted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
648 lines
36 KiB
Markdown
648 lines
36 KiB
Markdown
# LibrePortal — First-run: New Install or Restore (Roadmap / Proposal)
|
||
|
||
**Status:** Phases 1–4 **built**. · **Audience:** us, future-self · **Scope:** make "I'm rebuilding my server" a first-class path at first run, not a CLI expedition · **Origin:** "on the first install/setup we need 2 option blocks (New Install and Restore from Backup)" (2026-08-27)
|
||
|
||
---
|
||
|
||
## 0. The one idea
|
||
|
||
> **The wizard's first question becomes "is this a new server, or a replacement for one?"** Everything else follows from the answer.
|
||
|
||
Today a rebuild means knowing that `libreportal backup location add`, `libreportal restore first-run discover` and `libreportal restore first-run bulk` exist, in that order. That is a fine CLI story and a terrible first-run story — and rebuilding after a disk dies is exactly when someone is least able to go reading docs.
|
||
|
||
## 1. How much of this already exists
|
||
|
||
More than it looks. The restore path is built; what is missing is the front door.
|
||
|
||
| Piece | Status |
|
||
|---|---|
|
||
| Connect a backup repo (local / sftp / rest / s3 / b2 / gs / azure / rclone) | ✅ `locationAdd` |
|
||
| Read a repo without touching live state | ✅ `restoreFirstRunDiscover`, `migrateDiscoverHosts/Apps/AppDetail` |
|
||
| Restore the system config (settings + credentials) into staging | ✅ `backupRestoreSystemConfig`, `restore system` |
|
||
| Restore many apps in one go from another host | ✅ `restoreFirstRunBulk`, `restore first-run bulk` |
|
||
| Rewrite host-bound `CFG_*` (URL/HOST/DOMAIN) to this machine | ✅ `migrateUrlRewrite` |
|
||
| Preflight a migrate before committing | ✅ `migratePreflight` |
|
||
| Safety snapshot of the destination before overwriting | ✅ `migratePreBackupDestination` |
|
||
| Restore an app whose path differs from this host's | ✅ `storageRestoreAppTo` (§9 of storage-locations) |
|
||
| Snapshot records where the app lived | ✅ manifest `storage` block |
|
||
| **The wizard branch, and the reconciliation screen** | ❌ this document |
|
||
|
||
So this is mostly **assembly plus one genuinely new screen**, not new plumbing.
|
||
|
||
## 2. What the user sees
|
||
|
||
**Step 0 becomes two blocks**, before Experience:
|
||
|
||
```
|
||
┌───────────────────────────┐ ┌───────────────────────────┐
|
||
│ 🌱 New install │ │ ♻️ Restore from backup │
|
||
│ Set this machine up │ │ Rebuild a server from │
|
||
│ from scratch. │ │ an existing backup. │
|
||
└───────────────────────────┘ └───────────────────────────┘
|
||
```
|
||
|
||
**New install** → the wizard exactly as it is now.
|
||
|
||
**Restore** → Identity is skipped (it comes from the backup), and the flow becomes:
|
||
|
||
1. **Where is your backup?** — the existing backup-location fields, rendered from config metadata like the Locations page already does.
|
||
2. **Unlock it** — the repository password. See §4; this is the step that decides whether a restore is possible at all.
|
||
3. **What's in there?** — hosts found, then apps per host, with sizes and snapshot dates.
|
||
4. **What will change on this machine** — the reconciliation screen. §3. The new part.
|
||
5. **Restore** — system config first, then the chosen apps, with the task list the wizard already renders.
|
||
|
||
## 3. Reconciliation — a preflight report, not a screen
|
||
|
||
Originally written as a WebUI screen. The restore runs in the **installer**
|
||
(§2), so it becomes a report printed before anything is written, plus automatic
|
||
resolution where the answer is not a judgement call.
|
||
|
||
A backup describes a machine that no longer exists. The failure mode worth
|
||
designing against is re-deciding those facts silently.
|
||
|
||
**It is answerable before restoring** because every app's snapshot carries its
|
||
own `.libreportal-manifest.json`, and `engineDumpFile` can pull a single file
|
||
out of a snapshot without restoring it. So the installer can read all 13
|
||
manifests, compare them to this machine, and print a verdict per app.
|
||
|
||
| What differs | Detected from | Resolution |
|
||
|---|---|---|
|
||
| App no longer shipped by this version | template missing under `install/containers/` | **skip**, and say so — restoring an app whose template is gone produces an unstartable directory |
|
||
| Data larger than the target drive | manifest `size_bytes` vs `df` | **skip that app**, not the whole restore |
|
||
| Storage location named in the manifest doesn't exist here | manifest `storage.location` vs the registry | fall back to the default location, and say which app moved where |
|
||
| Old absolute path ≠ where it goes here | manifest `storage.path` | already handled — `storageRestoreAppTo` stages and moves |
|
||
| Domains point at the old host | — | already handled — `migrateUrlRewrite` |
|
||
| Ports / IPs already taken | port allocation tables | already handled by the install pipeline; reallocated on start |
|
||
|
||
Output is a verdict list, then one confirmation:
|
||
|
||
```
|
||
Checking 13 apps against this machine…
|
||
|
||
✓ bookstack 2.1G restores as-is
|
||
✓ nextcloud 44G -> bigdisk (its old location "ssd" is not on this machine)
|
||
✗ obsolete-app 120M skipped — this version no longer ships it
|
||
✗ jellyfin 1.2T skipped — needs 1.2T, /mnt/data has 400G free
|
||
|
||
11 will restore, 2 skipped.
|
||
Continue? [Y/n]:
|
||
```
|
||
|
||
### 3.1 — Two silent no-ops, and what they have in common
|
||
|
||
Both were found by running the installer's restore path end to end against a
|
||
real repository, not by reading the code — which is the point of doing that.
|
||
|
||
1. **The manifest is pretty-printed.** `"size_bytes": 123` carries a space, so
|
||
a `"key":[0-9]*` pattern missed it. Size came back empty, and the fit check
|
||
is guarded by `[[ -n "$size_bytes" ]]` — so it was skipped for every app.
|
||
2. **The manifest was never read at all.** `storageSnapshotSourcePath` passed a
|
||
snapshot id into an app-tag filter (see storage-locations §9), so it returned
|
||
1 every time and the preflight fell back to a `?` size.
|
||
|
||
The shared shape: **a check whose failure mode is to not run.** Neither printed
|
||
an error, and the report they produced — thirteen green ticks — is exactly what
|
||
a healthy run looks like. The only visible tell was the `?` in a column nobody
|
||
had a reason to distrust.
|
||
|
||
So the regression test (`scripts/dev/lp-preflight-test`) asserts the *negative*
|
||
cases: an app too big for the disk must be refused, an app this version no
|
||
longer ships must be refused, an app whose location is gone must be marked as
|
||
moved. Reintroducing either historical bug fails it.
|
||
|
||
### 3.2 — And a third: four apps of thirteen, reported as success
|
||
|
||
With the preflight finally reading manifests, the run got as far as restoring —
|
||
and restored 4 of 13, printing *"First-run restore complete — 4 apps restored"*
|
||
with exit 0.
|
||
|
||
The app list crosses two chokepoints, and fixing the first had hidden the
|
||
second:
|
||
|
||
1. the CLI dispatcher calls handlers with **no** arguments, so `"$@"` and
|
||
`shift` inside one operate on an empty list. `LP_CLI_ARGS` was added for this.
|
||
2. but `LP_CLI_ARGS` was built from `start.sh`'s `"$@"` — and the root wrapper
|
||
invokes `start.sh` with exactly **nine hardcoded positional slots**. The
|
||
array could never hold more than nine entries, so `${LP_CLI_ARGS[@]:5}` gave
|
||
at most four app names.
|
||
|
||
Same shape as §3.1 once more: the truncation had no failure path. Four apps
|
||
restored perfectly, and the success line counted the list it was handed.
|
||
|
||
Three changes, because one would not have been enough:
|
||
|
||
- the wrapper forwards the real argv after the nine slots (which stay, since
|
||
every dispatcher reads them and unset ones must keep arriving as `"empty"`);
|
||
`start.sh` reads it back as `"${@:10}"`. `footprint_version` 7 → 8.
|
||
- `restoreFirstRunBulk` with no list is a **whole-host restore**: it discovers
|
||
the host's apps and re-applies the preflight itself, because the installer's
|
||
report ran in a different process and its decision was otherwise lost — a
|
||
skipped app would have been restored anyway. The installer now passes no
|
||
list, so there is nothing to truncate.
|
||
- it counts what actually landed and returns non-zero naming the failures.
|
||
|
||
`scripts/dev/lp-cli-argv-test` builds stubs from the real lines in `init.sh` and
|
||
`start.sh` and pushes thirteen app names through them.
|
||
|
||
### 3.3 — Staging: created by one principal, written by another
|
||
|
||
The worst of the set, because it sat on the step everything else depends on.
|
||
|
||
`restore system` printed
|
||
|
||
```
|
||
✓ Success System config restored to: /libreportal-system/restore/system-config
|
||
```
|
||
|
||
for a directory that did not exist. Nothing had been written.
|
||
|
||
Restore stages through `$SYSTEM_DIR`, which the **manager** owns — but the thing
|
||
writing into the staging tree is restic, and `runBackupOp` runs it as the
|
||
**container user**. Both call sites created the directory as the wrong
|
||
principal, in opposite directions:
|
||
|
||
| Call site | Created by | Written by | Result |
|
||
|---|---|---|---|
|
||
| `backupRestoreSystemConfig` | `runFileOp` → container user | container user | `mkdir` denied on the 0751 manager-owned `restore_dir`, unchecked |
|
||
| `storageRestoreAppTo` | `runInstallOp` → manager | container user | restic could not create anything beneath it |
|
||
|
||
And restic reports a permission denial as `ignoring error …` and **still exits
|
||
0**, so the caller's success check was satisfied either way. Same family as
|
||
§3.1: the check had no failure path.
|
||
|
||
Root has to bridge that, the way `webui-bind` already bridges the mirror case.
|
||
`libreportal-ownership` gains `restore-stage <path>` (creates it `cowner:MANAGER`
|
||
0750 — owner writes, manager traverses to confirm and review) and
|
||
`restore-unstage <path>` (removes it: neither principal can, since the manager
|
||
cannot delete the container user's files inside and the container user cannot
|
||
unlink the entry from the manager's directory, so staging simply accumulated).
|
||
Both confine the path to a single component directly under the restore/migrate
|
||
area. `footprint_version` 8 → 9.
|
||
|
||
`backupRestoreSystemConfig` now also verifies the tree landed **as the user that
|
||
wrote it**, since the manager cannot read inside its own staging directory.
|
||
|
||
### 3.4 — What the run finally proved
|
||
|
||
With all of the above fixed, on a live machine:
|
||
|
||
- 13 of 13 apps restored from the repository and came up healthy, databases
|
||
included (`bookstack`, `matrix` ×3, `mattermost` ×2, `nextcloud` ×4,
|
||
`rocketchat` ×2, `stoat` ×10, …)
|
||
- `restore system` staged 57 real files
|
||
- the **relocation** branch of `storageRestoreAppTo` ran for the first time:
|
||
`speedtest`, moved to a second storage location, restored from a snapshot
|
||
taken at `/libreportal-containers/speedtest` into `/libreportal-alt/speedtest`
|
||
via stage-and-move, with the staging tree cleaned up afterwards
|
||
|
||
One genuine environment collision remains and is **not** a LibrePortal defect:
|
||
stoat's livekit publishes a fixed UDP range (50000–50100) that it advertises to
|
||
clients and so cannot be re-rolled, and a desktop's `kdeconnectd` held 50016.
|
||
Worth a fixed-range preflight check of its own; see §9.
|
||
|
||
### 3.5 — Ownership was not actually being reinstated
|
||
|
||
`grafana` restored and then died with *"attempt to write a readonly database"*,
|
||
repeatedly. The snapshot records:
|
||
|
||
```
|
||
-rw-r----- 231543 1002 /libreportal-containers/grafana/grafana_storage/grafana.db
|
||
```
|
||
|
||
and what landed was `1002:1002`. The owner was lost, so grafana — running as
|
||
231543 — could not write its own database at mode 0640.
|
||
|
||
Restore runs as the backup user, which has no `CAP_CHOWN`, so it reinstates
|
||
ownership through a user namespace. The prefix was:
|
||
|
||
```
|
||
unshare --map-root-user --map-users=231072:231072:65536 --map-groups=231072:231072:65536
|
||
```
|
||
|
||
and `unshare` accepts **one range per option**. So the backup user's own GID was
|
||
never mapped — while LibrePortal writes app data as `<container-uid>:<backup-user>`.
|
||
The group half of every such chown referred to an unmapped id, `lchown` returned
|
||
EINVAL, and the file kept the restoring user's ownership. Measured directly:
|
||
|
||
| target | old prefix | now |
|
||
|---|---|---|
|
||
| `231543:1002` (app data) | ❌ → `1002:1002` | ✅ `231543:1002` |
|
||
| `1002:1002` (LibrePortal's own) | ❌ → `1002:1002` (right anyway) | ❌ → `1002:1002` (right anyway) |
|
||
| `231072:231072` (container root) | ✅ | ✅ |
|
||
|
||
Same shape as the rest of §3: restic reports the misses as `ignoring error …`
|
||
and **still exits 0**, and the caller forgave them as *"expected, they are
|
||
already owned correctly"*. 1626 of one 13-app restore's 2086 failed chowns were
|
||
grafana's.
|
||
|
||
**Fixed** with `scripts/backup/engine/restic-userns-exec`, which uses
|
||
`newuidmap`/`newgidmap` — those write multi-range maps, which is exactly what
|
||
`unshare` cannot express:
|
||
|
||
```
|
||
uid: 0 <- caller inner root, or capabilities are dropped at exec
|
||
SUB.. <- SUB.. identity, so restic can name the stored uid
|
||
gid: caller <- caller identity — the group half of app-data chowns
|
||
SUB.. <- SUB.. identity
|
||
```
|
||
|
||
The caller's own UID is deliberately not identity-mapped: that slot is spent on
|
||
inner root. A file stored as `<caller>:<caller>` therefore fails its chown and
|
||
lands owned by the caller anyway, because that is who inner root is on the
|
||
outside — so the one case this cannot map is the one case that needs no mapping.
|
||
|
||
`unshare --map-auto --map-current-user` does **not** work as a shortcut: it maps
|
||
the subuid range to low inner ids (container-style, `0 → 231072`) while restic
|
||
needs identity. Tested.
|
||
|
||
Measured on the live install, restoring grafana: failed chowns **1626 → 12**
|
||
(the 12 being the caller's own files, which are correct), `grafana.db` back to
|
||
`231543:1002`, and grafana up and writing. `scripts/dev/lp-userns-ownership-test`
|
||
pins all three rows of the table above.
|
||
|
||
Anything unexpected — no subuid range, no `newuidmap`, a namespace that will not
|
||
start — falls back to running the command plainly, which is what happened before
|
||
any of this existed.
|
||
|
||
### 3.6 — Open: a fresh install generates new secrets, the backup carries the old ones
|
||
|
||
Found by restoring a real 13-app repository onto a purpose-built install
|
||
(storage-locations §12.5, case 4). 11 of 13 apps came back working. The two that
|
||
did not failed the same way for two different reasons, and the reason is
|
||
structural rather than a bug in either app.
|
||
|
||
**A first-run restore generates fresh credentials, then restores state that
|
||
encodes the old ones.** Install randomises every `CFG_*_PASSWORD`; the backup
|
||
contains a machine that used different ones. Wherever a secret is written into
|
||
something the restore then lays down, the two disagree:
|
||
|
||
| app | what disagreed | result |
|
||
|---|---|---|
|
||
| `matrix` | the install hook writes `homeserver.yaml` from `CFG_MATRIX_DB_PASSWORD_1` at step 3, and step 6 restores the snapshot *over* it | `password authentication failed for user "synapse"` — postgres was initialised with the new secret, the restored config file holds the old one |
|
||
| `stoat` | `data/rabbit/mnesia` restores raw, and rabbit only honours `RABBITMQ_DEFAULT_PASS` on an **empty** data dir | `ACCESS_REFUSED` — four dependent services exit 101 |
|
||
|
||
Apps with a **dump descriptor** are unaffected, and that is the tell: for those,
|
||
`restoreDbRehydratePreStart` clears the data dir so the engine initialises with
|
||
the current secret, then replays the dump into it. Everything else — a config
|
||
file inside the app dir, a broker's own user database — keeps the old value.
|
||
|
||
Three ways out:
|
||
|
||
1. **Apply the restored system config on first run** instead of only staging it.
|
||
The backup's `configs/` holds the original secrets, so adopting them makes
|
||
every restored app consistent by construction. `backupRestoreSystemConfig`
|
||
deliberately stages rather than applies — *"recovering creds/settings is a
|
||
review-then-copy step, never an automatic blast over a running control
|
||
plane"* — and that is right for a **running** box and wrong for a **fresh**
|
||
one. The code currently cannot tell those two situations apart. This is §9.1
|
||
restated with a concrete failure behind it, and it carries a real
|
||
consequence: it also restores the WebUI login, so the user signs in to the
|
||
new machine with the old password. Probably expected; must be said out loud.
|
||
2. **Re-run app config generation after the data restore.** Fixes matrix. Does
|
||
nothing for stoat, where the secret lives inside restored *service state*
|
||
rather than a generated file — so it is not sufficient on its own.
|
||
3. **Reset the credential in the restored service** (`ALTER ROLE`,
|
||
`rabbitmqctl change_password`). Per-engine, fragile, and needs a hook per
|
||
backing service.
|
||
|
||
Recommendation: **(1)**, gated on first run specifically — the flag already
|
||
exists as `init_mode=restore`. (2) is a reasonable belt-and-braces addition; (3)
|
||
is a last resort for services whose state cannot be re-initialised.
|
||
|
||
Not implemented here: it changes what a restore does with credentials, which is
|
||
a decision rather than a defect.
|
||
|
||
### 3.7 — The restore that restored nothing
|
||
|
||
Phases 1–4 were marked built, and the installer's restore path ran end to end
|
||
and printed "Settings restored". It had never restored a setting.
|
||
|
||
`backupRestoreSystemConfig` only **stages**. That is deliberate and correct in
|
||
general — overwriting the config of a running control plane is not something to
|
||
do automatically — but nothing in the tree ever adopted the staged result. The
|
||
backup locations, the domains, the logins all landed in
|
||
`$restore_dir/system-config` and stayed there, and the one line of output a
|
||
person actually reads said the opposite.
|
||
|
||
So adoption is now its own step (`restore adopt`), allowed only where the
|
||
original caution does not apply: a machine with nothing on it yet.
|
||
|
||
**What is not adopted matters as much as what is.** A backup describes a
|
||
machine that no longer exists, and some of what it says is about that machine
|
||
rather than about the user: `general_docker_install` (the container account and
|
||
its generated password, made by *this* install), `network_ports` (re-rolled per
|
||
install), `network_docker` / `network_rootless` (this box's hardware and
|
||
kernel), and `storage/locations` — the old machine's drives. App placement is
|
||
already reconciled per app from the snapshot manifests (§3); adopting a
|
||
registry of drives this box does not have would make every one of those lookups
|
||
resolve to a phantom.
|
||
|
||
`backup/locations/` **is** adopted, as a subtree rather than a filename, since
|
||
the index is part of the path. That one is the whole point: it holds every
|
||
repository and its credentials, and "one password you remember unlocks the
|
||
rest" is the promise a first-run restore makes.
|
||
|
||
#### The guard, and the shape it failed in
|
||
|
||
Adoption overwrites live config, so the first-run check is the only thing
|
||
between "restore onto a blank box" and "overwrite a working install".
|
||
|
||
The first version globbed the containers directory directly. The manager can
|
||
*traverse* that directory without being able to *list* it, so the glob came
|
||
back as the literal `*`, the loop skipped it as not-a-directory, and the
|
||
function fell out of the bottom returning "yes, first run" — on a machine with
|
||
three apps on it. It adopted over a live install in testing before the guard
|
||
was rewritten to fail closed, asking the container user for the listing and
|
||
treating an unreadable directory as "in use" rather than "empty".
|
||
|
||
That is the same shape as §3.1's silent no-ops, and it is worth naming again:
|
||
**a check whose failure mode is to not run is indistinguishable from a check
|
||
that passed.**
|
||
|
||
### 3.8 — Two config modes, inverted
|
||
|
||
Found while testing the above, because a first-run restore cannot restore from
|
||
a system snapshot that was never successfully taken.
|
||
|
||
**Storage location configs were `0640`.** They hold a name, a path and free
|
||
text — nothing secret. The backup runs as the container user, which could not
|
||
read them, so restic reported `permission denied`, wrote an **INCOMPLETE**
|
||
snapshot and exited 3. Every system-config backup failed the moment a second
|
||
storage location existed. Now `0644`, and the test asserts they contain no
|
||
secrets so that mode stays defensible.
|
||
|
||
**Backup location configs were `0644`.** They hold
|
||
`CFG_BACKUP_LOC_<n>_PASSWORD` — the key to every backup the user has — and
|
||
`nobody` could demonstrably read them. The mode cannot simply be tightened: the
|
||
backup genuinely has to read the credentials it is about to use. So the
|
||
*directory* carries the restriction instead (`config-secure`:
|
||
manager:container, `0750`). Both accounts that belong still get in; nothing
|
||
else can traverse, whatever the modes inside say.
|
||
|
||
A third bug came out of fixing that one. `config-adopt` created missing parent
|
||
directories and clamped them to `manager:manager 0750` — including ones that
|
||
already existed, which closed `configs/backup` to the container user and broke
|
||
the very credential read the directory fix had just preserved. A copy has no
|
||
business re-permissioning directories it merely passes through.
|
||
|
||
### 3.9 — Domain reconciliation
|
||
|
||
The adopted config carries the domains the **old** machine served, and DNS
|
||
still points wherever it pointed. Nothing checked this, so the first sign of
|
||
trouble was Traefik failing to get a certificate long after the installer said
|
||
it had finished.
|
||
|
||
`restore domains` reports one verdict per domain, and the installer offers to
|
||
drop the strays. It is a report, never a refusal: a domain that does not
|
||
resolve here is a perfectly normal state five minutes into a rebuild.
|
||
|
||
Three verdicts, not two. `setupCheckDomainPointsHere` falls back to
|
||
`hostname -I` when the public-IP lookup fails, which is fine for its own
|
||
purposes and wrong here — comparing a public A record against a private
|
||
`10.x` address makes every correctly-pointed domain look misconfigured, and
|
||
this is explicitly a LAN/VPN-first product where that lookup failing is
|
||
ordinary. So an unverifiable domain reports **unknown** and is never offered
|
||
for deletion; only a domain that demonstrably resolves elsewhere is.
|
||
|
||
Two parsing bugs worth recording, both caught by running it rather than reading
|
||
it. Config values carry a trailing comment column, so every domain arrived with
|
||
an essay attached and no lookup could match. And `updateConfigOption` writes an
|
||
empty value as a literal `""`, so nine cleared slots read back as nine
|
||
two-character domains and were reported as nine failures.
|
||
|
||
### 3.10 — Reading a repository without restoring it
|
||
|
||
`restore inspect <idx> [host]` answers "what would a restore from here bring?"
|
||
without writing anything: which machines' backups are in the repository, which
|
||
apps and how big, and — the part that is least obvious to get at — **which
|
||
domains**. Those live in the system-config snapshot, so `engineDumpFile` pulls
|
||
`network/network_domains` straight out of it, the same way the preflight pulls
|
||
an app's manifest. Knowing "this backup hands you six domains, four of which
|
||
point somewhere else" before committing is the difference between a rebuild and
|
||
a surprise.
|
||
|
||
`restore connect <base64-json>` is the WebUI's entry point: it creates the
|
||
location from a payload, redeems the repository password from the single-use
|
||
secret channel, and inspects. It deliberately does **not** call
|
||
`engineInitLocation` — every other path that creates a location initialises it
|
||
because it is about to write there; this one is pointed at a repository that
|
||
already exists and is only going to be read.
|
||
|
||
This is what unblocks §4. `app_portable.sh` records that a `.lpapp` can live in
|
||
the WebUI *because* it is unencrypted and no password has to cross from the
|
||
browser to the host — and that the repository restore therefore could not. The
|
||
`secret:<ref>` channel is that missing piece, so the constraint no longer holds.
|
||
|
||
A wrong password is the ordinary case here and the user simply tries again, so
|
||
a failed connect removes the location it just created. Without that, every
|
||
retry left another half-configured destination behind and the Backup page grew
|
||
a column of identical dead entries.
|
||
|
||
### 3.11 — Three more found by using it
|
||
|
||
**`locationRemove` never worked.** It removed with `runFileOp` — the container
|
||
user — but `configs/` is manager-owned, so the unlink was always denied. The
|
||
result was never checked, and `isSuccessful` printed regardless, so a location
|
||
"removed" from the WebUI came straight back on the next listing. Now
|
||
`runInstallOp`, and the directory is checked before claiming anything.
|
||
|
||
**`webuiSecretSweep` had no callers.** Written for exactly this and never
|
||
wired in, so a flow the user abandoned — closed the tab, hit a validation
|
||
error, never pressed Save — left its repository password on disk indefinitely.
|
||
The sweep now runs in the `/api/setup/secret` route before each write, which
|
||
ties it to the one event guaranteed to happen whenever secrets are being made.
|
||
|
||
**Adoption took the WebUI down.** `config-adopt` chowned every adopted file to
|
||
`manager:manager 0640`. `webui_logins` is bind-mounted into the WebUI
|
||
container, which then could not read its own credentials file: the container
|
||
died with exit 137 and *no log line at all*, which is a genuinely hard failure
|
||
to read. It also clamped every parent directory it passed through, closing
|
||
`configs/webui` and `configs/backup` to the container user.
|
||
|
||
The fix is a principle rather than a special case: **a restore replaces the
|
||
content of a config file and nothing else.** The live install already knows who
|
||
is allowed to read each one. Adoption preserves the destination's existing
|
||
ownership and mode and never re-permissions a directory it merely passes
|
||
through.
|
||
|
||
For a file this install did not have, the first attempt picked `0640` — "these
|
||
can hold secrets, so default closed". That is the storage-location bug again,
|
||
one directory over: the config tree is `0755`, the backup account reads all of
|
||
it, and a single `0640` file makes restic write an INCOMPLETE snapshot and
|
||
report the whole run as failed. A new file now inherits from a sibling in the
|
||
same directory, so it matches whatever the tree's convention is rather than
|
||
having a mode chosen for it.
|
||
|
||
That is three occurrences of the same defect in three directories for three
|
||
different reasons, so the test now asserts the *class*: **no file anywhere
|
||
under `configs/` may be unreadable by the backup account.** That one line would
|
||
have caught all three.
|
||
|
||
## 3.12 — The WebUI branch
|
||
|
||
§2 described the wizard's first question becoming "is this a new server, or a
|
||
replacement for one?", and it now is. `Start` asks, and the answer selects one
|
||
of **two disjoint step sets**:
|
||
|
||
```
|
||
new Start -> Experience -> Identity -> Domains -> Storage
|
||
-> Backups -> Import -> Recommended -> (Metrics)
|
||
restore Start -> Source -> Contents -> Rebuild
|
||
```
|
||
|
||
Disjoint on purpose. A restore is never asked for an install name, domains or
|
||
an app list, because the backup answers all three — asking would invite someone
|
||
to type an answer that is about to be written over, which is worse than not
|
||
asking. The test asserts non-overlap in *both* directions rather than only that
|
||
the restore steps appear.
|
||
|
||
**Source** collects the repository the same way the Backup page does, minus
|
||
everything that only means something for a place you WRITE to: no retention, no
|
||
schedule, no enable toggle. The password leaves through the one-shot
|
||
`secret:<ref>` channel and is cleared from the DOM; the test asserts the value
|
||
never appears in the payload, since that payload reaches a task command line
|
||
and tasks are recorded world-readable.
|
||
|
||
**Contents** is §3's reconciliation, rendered. Apps with sizes, and the domains
|
||
with a verdict each — checked through the same `/api/setup/dns-check` the
|
||
Domains step uses rather than adding a second way to ask the question — plus
|
||
the offer to leave the strays out until DNS is repointed.
|
||
|
||
**Rebuild** hands over to `restore rebuild`, which is the installer's order
|
||
with the same reasoning: settings first (they carry every other repository's
|
||
credentials), then domains, then apps with no explicit list so `bulk` discovers
|
||
and re-preflights them itself.
|
||
|
||
### The index that moved
|
||
|
||
Inserting `Start` shifted every step index by one, and `validateStep` was a
|
||
chain of `idx === 1 … idx === 6` with a comment already explaining which
|
||
earlier insertions had moved them. It is keyed on the step NAME now.
|
||
|
||
`lp-storage-step-test` had the same pin and did not survive the change: it
|
||
called `validateStep(3)` for Storage, which had become Domains, and reported
|
||
that nothing blocked. That reads exactly like validation being broken — and if
|
||
the four assertions had happened to be less specific it would have read like
|
||
everything passing instead. Tests look their step up by name too now.
|
||
|
||
## 4. The password problem, stated plainly
|
||
|
||
**An encrypted repository cannot be opened with anything inside itself.** `CFG_BACKUP_LOC_<idx>_PASSWORD` lives in the system config — which is *inside the backup*. So on a fresh machine the user must supply the repository password by hand. There is no way around this and it is not a bug; it is what encryption means.
|
||
|
||
That single fact drives two requirements:
|
||
|
||
- The restore step must ask for it early and say what it is, because a user who never wrote it down has no backup, and finding that out at step 2 is better than at step 5.
|
||
- Everywhere we generate that password we must push harder than a comment. The location config already says *"back up offline!"*; the WebUI should show it once, prominently, at creation — treated like a recovery key, because that is what it is.
|
||
|
||
Once the repo opens, ordering is already correct in the CLI and should be preserved: **system config first** (it carries every *other* location's credentials, so one password unlocks the rest), then apps.
|
||
|
||
### 4.1 — Where does the typed password actually travel? (blocker for phase 2)
|
||
|
||
Found while building phase 2, and it needs a decision before the restore branch
|
||
can be written, because it is a security trade-off rather than an implementation
|
||
detail.
|
||
|
||
The WebUI cannot run restic. So a password the user types in the browser has to
|
||
reach the host somehow, and the two existing channels both leak it:
|
||
|
||
| Channel | Problem |
|
||
|---|---|
|
||
| Task command string (what the Backup page already does for this exact field, via `config_update CFG_BACKUP_LOC_<n>_PASSWORD=…`) | lands in the task JSON under `frontend/data/tasks/`, which is **0644** so the manager can read it — i.e. world-readable — and is visible in `ps` while the task runs |
|
||
| A file in `frontend/data/tasks/` | the container writes as `dockerinstall`; the manager runs as `libreportal`. At 0640 the manager cannot read it (verified), and 0644 is world-readable again |
|
||
|
||
Note the first row is **existing behaviour**, not something this feature would
|
||
introduce: editing a backup location's password on the Backup page already
|
||
sends it that way. So this is a product-wide finding that phase 2 happens to
|
||
surface, and the restore case is the sharpest version of it — that password is
|
||
the key to every backup the user has.
|
||
|
||
Three ways out, roughly in order of effort:
|
||
|
||
1. **A one-shot secret drop.** The ownership helper already knows how to make a
|
||
path readable across exactly this boundary (`_webui_bind_access` chowns
|
||
`MANAGER:cowner` 0640 so the container can read manager-owned files). The
|
||
reverse needs the same treatment: a root-helper-created directory owned
|
||
`cowner:MANAGER` 0730, into which the container drops a 0640 file the manager
|
||
reads once and unlinks.
|
||
2. **Never persist it.** Hold the password only in the task processor's memory
|
||
for the life of the restore; write it into the location config only after the
|
||
system-config restore lands (and reconcile with what the backup contained).
|
||
3. **Accept the existing channel** for consistency, and fix it product-wide
|
||
later — cheapest now, and no worse than what shipping code already does, but
|
||
it does mean a restore password sits world-readable in a task file until that
|
||
task is pruned.
|
||
|
||
Recommendation: **(1)**, and apply it to the Backup page's password field at the
|
||
same time. It is a small, well-scoped addition to a helper that already exists
|
||
for the mirror-image case, and it fixes a live weakness rather than only
|
||
avoiding a new one.
|
||
|
||
## 5. "Set up a backup server if you don't have one"
|
||
|
||
Same components, other direction. After a **New install**, offer: *"Where should your backups go?"* — the same location fields, then `engineInit` and a first `backup system`. That closes a real gap: today backups exist but nothing prompts you to configure them, so the people most likely to need a restore are the least likely to have one.
|
||
|
||
Worth doing as its own wizard step even without the restore branch.
|
||
|
||
## 6. On "upload the backup file" — why the format differs
|
||
|
||
Worth being precise, because the mental model doesn't match the engines. restic, borg and kopia back up to a **repository** — a directory or a remote — not a single file. There is nothing to upload. The equivalents are:
|
||
|
||
- **local** — a path on a plugged-in disk. "Navigate to it" is right, and a directory picker is the natural UI.
|
||
- **sftp / s3 / b2 / …** — credentials, which the existing fields already collect.
|
||
|
||
If a genuine single-file import is wanted, that is a **different feature**: a portable per-app export (`tar` of the app dir + manifest, optionally encrypted) that could be handed around and imported. Cheap to build on the manifest that already exists, but it is not what the backup engines produce and shouldn't be conflated with them.
|
||
|
||
## 7. Portable export — the single file people actually mean
|
||
|
||
§6 explains why "upload the backup file" does not match a restic repository.
|
||
But the underlying want is real and worth serving directly: **one file, one
|
||
app, hand it around**.
|
||
|
||
libreportal app export <app> [file] # -> <app>-<date>.lpapp
|
||
libreportal app import <file>
|
||
|
||
The format is deliberately boring: a gzipped tar of the app directory with its
|
||
`.libreportal-manifest.json` at the root. That manifest already records the
|
||
compose hash, images, volumes, size, databases and storage location, so import
|
||
gets the same reconciliation as §3 for free.
|
||
|
||
This also gives the installer a third answer to "where is your backup?" — a
|
||
`.lpapp` file — which is exactly the "navigate to the backup file" flow that
|
||
prompted this document.
|
||
|
||
Not a replacement for the backup engines: no deduplication, no history, no
|
||
encryption unless the user encrypts it themselves. It is a courier format —
|
||
moving one app between machines, or keeping a copy of something before a risky
|
||
change — and the docs should say so plainly so nobody uses it as their backup.
|
||
|
||
## 8. Phasing
|
||
|
||
| Phase | Deliverable |
|
||
|---|---|
|
||
| **1** ✅ | Backup destination step in the WebUI wizard (§5) — the *new setup* half |
|
||
| **2** ✅ | Two installer paths: New setup / Restore from backup, through connect → discover → system config → apps. *The system-config half only staged until §3.7; it now adopts.* |
|
||
| **3** ✅ | Preflight reconciliation report in the installer (§3) |
|
||
| **4** ✅ | `app export` / `app import` (§7). The installer's `.lpapp` option is still open — see §9.5 |
|
||
| **5** ✅ | The WebUI branch: New install / Restore from backup, through source → contents → rebuild (§3.12) |
|
||
|
||
## 9. Open questions
|
||
|
||
1. **Does the restore branch also restore the system config's *identity*** — install name, domains, WebUI credentials? Restoring the WebUI login means the user logs into the new box with the old password, which is probably what they expect, but it is a surprise if not stated.
|
||
2. **Partial restore of a host** — pick apps individually (already supported by `restoreFirstRunBulk`'s signature) or all-or-nothing at first run?
|
||
3. **What if the backup is newer than this LibrePortal version?** The manifest records the commit; refusing is safer than guessing, but it strands someone whose only copy is newer.
|
||
5. **Should the installer's restore path accept a `.lpapp` too?** `app import`
|
||
exists, so the third answer to "where is your backup?" is a small addition —
|
||
but a single app file is a thin thing to rebuild a *server* from, and
|
||
offering it beside a repository may imply more than it delivers.
|
||
7. **A published range that cannot be re-rolled** — livekit advertises
|
||
50000–50100 to clients, so the port allocator's randomisation does not apply.
|
||
Nothing checks such a range against the host before compose-up, and the
|
||
result is an app that comes back part-started. A preflight that compares
|
||
fixed published ranges against `ss` output would catch it; the open question
|
||
is what to *do* about it, since the app cannot simply be moved elsewhere.
|
||
6. **Import under a different name** is refused today: the app's `CFG_<APP>_*`
|
||
namespace and its compose identities (container names, Traefik routers,
|
||
backup labels) would all need rewriting. `instance create` already solves
|
||
"a second copy", so this may never be worth building.
|
||
4. **Where does the repository password go once entered** — straight into the location config it will restore over, or held only in memory until the system config lands and then reconciled?
|