LibrePortal/docs/roadmap/first-run-restore.md

# LibrePortal — First-run: New Install or Restore (Roadmap / Proposal)

**Status:** Phases 1–4 **built**. · **Audience:** us, future-self · **Scope:** make "I'm rebuilding my server" a first-class path at first run, not a CLI expedition · **Origin:** "on the first install/setup we need 2 option blocks (New Install and Restore from Backup)" (2026-08-27)

---

## 0. The one idea

> **The wizard's first question becomes "is this a new server, or a replacement for one?"** Everything else follows from the answer.

Today a rebuild means knowing that `libreportal backup location add`, `libreportal restore first-run discover` and `libreportal restore first-run bulk` exist, in that order. That is a fine CLI story and a terrible first-run story — and rebuilding after a disk dies is exactly when someone is least able to go reading docs.

## 1. How much of this already exists

More than it looks. The restore path is built; what is missing is the front door.

| Piece | Status |
|---|---|
| Connect a backup repo (local / sftp / rest / s3 / b2 / gs / azure / rclone) | ✅ `locationAdd` |
| Read a repo without touching live state | ✅ `restoreFirstRunDiscover`, `migrateDiscoverHosts/Apps/AppDetail` |
| Restore the system config (settings + credentials) into staging | ✅ `backupRestoreSystemConfig`, `restore system` |
| Restore many apps in one go from another host | ✅ `restoreFirstRunBulk`, `restore first-run bulk` |
| Rewrite host-bound `CFG_*` (URL/HOST/DOMAIN) to this machine | ✅ `migrateUrlRewrite` |
| Preflight a migrate before committing | ✅ `migratePreflight` |
| Safety snapshot of the destination before overwriting | ✅ `migratePreBackupDestination` |
| Restore an app whose path differs from this host's | ✅ `storageRestoreAppTo` (§9 of storage-locations) |
| Snapshot records where the app lived | ✅ manifest `storage` block |
| **The wizard branch, and the reconciliation screen** | ❌ this document |

So this is mostly **assembly plus one genuinely new screen**, not new plumbing.

## 2. What the user sees

**Step 0 becomes two blocks**, before Experience:

```
┌───────────────────────────┐   ┌───────────────────────────┐
│  🌱  New install          │   │  ♻️  Restore from backup   │
│  Set this machine up      │   │  Rebuild a server from    │
│  from scratch.            │   │  an existing backup.      │
└───────────────────────────┘   └───────────────────────────┘
```

**New install** → the wizard exactly as it is now.

**Restore** → Identity is skipped (it comes from the backup), and the flow becomes:

1. **Where is your backup?** — the existing backup-location fields, rendered from config metadata like the Locations page already does.
2. **Unlock it** — the repository password. See §4; this is the step that decides whether a restore is possible at all.
3. **What's in there?** — hosts found, then apps per host, with sizes and snapshot dates.
4. **What will change on this machine** — the reconciliation screen. §3. The new part.
5. **Restore** — system config first, then the chosen apps, with the task list the wizard already renders.

## 3. Reconciliation — a preflight report, not a screen

Originally written as a WebUI screen. The restore runs in the **installer**
(§2), so it becomes a report printed before anything is written, plus automatic
resolution where the answer is not a judgement call.

A backup describes a machine that no longer exists. The failure mode worth
designing against is re-deciding those facts silently.

**It is answerable before restoring** because every app's snapshot carries its
own `.libreportal-manifest.json`, and `engineDumpFile` can pull a single file
out of a snapshot without restoring it. So the installer can read all 13
manifests, compare them to this machine, and print a verdict per app.

| What differs | Detected from | Resolution |
|---|---|---|
| App no longer shipped by this version | template missing under `install/containers/` | **skip**, and say so — restoring an app whose template is gone produces an unstartable directory |
| Data larger than the target drive | manifest `size_bytes` vs `df` | **skip that app**, not the whole restore |
| Storage location named in the manifest doesn't exist here | manifest `storage.location` vs the registry | fall back to the default location, and say which app moved where |
| Old absolute path ≠ where it goes here | manifest `storage.path` | already handled — `storageRestoreAppTo` stages and moves |
| Domains point at the old host | — | already handled — `migrateUrlRewrite` |
| Ports / IPs already taken | port allocation tables | already handled by the install pipeline; reallocated on start |

Output is a verdict list, then one confirmation:

```
  Checking 13 apps against this machine…

    ✓ bookstack     2.1G   restores as-is
    ✓ nextcloud     44G    -> bigdisk (its old location "ssd" is not on this machine)
    ✗ obsolete-app  120M   skipped — this version no longer ships it
    ✗ jellyfin      1.2T   skipped — needs 1.2T, /mnt/data has 400G free

  11 will restore, 2 skipped.
  Continue? [Y/n]:
```

### 3.1 — Two silent no-ops, and what they have in common

Both were found by running the installer's restore path end to end against a
real repository, not by reading the code — which is the point of doing that.

1. **The manifest is pretty-printed.** `"size_bytes": 123` carries a space, so
   a `"key":[0-9]*` pattern missed it. Size came back empty, and the fit check
   is guarded by `[[ -n "$size_bytes" ]]` — so it was skipped for every app.
2. **The manifest was never read at all.** `storageSnapshotSourcePath` passed a
   snapshot id into an app-tag filter (see storage-locations §9), so it returned
   1 every time and the preflight fell back to a `?` size.

The shared shape: **a check whose failure mode is to not run.** Neither printed
an error, and the report they produced — thirteen green ticks — is exactly what
a healthy run looks like. The only visible tell was the `?` in a column nobody
had a reason to distrust.

So the regression test (`scripts/dev/lp-preflight-test`) asserts the *negative*
cases: an app too big for the disk must be refused, an app this version no
longer ships must be refused, an app whose location is gone must be marked as
moved. Reintroducing either historical bug fails it.

### 3.2 — And a third: four apps of thirteen, reported as success

With the preflight finally reading manifests, the run got as far as restoring —
and restored 4 of 13, printing *"First-run restore complete — 4 apps restored"*
with exit 0.

The app list crosses two chokepoints, and fixing the first had hidden the
second:

1. the CLI dispatcher calls handlers with **no** arguments, so `"$@"` and
   `shift` inside one operate on an empty list. `LP_CLI_ARGS` was added for this.
2. but `LP_CLI_ARGS` was built from `start.sh`'s `"$@"` — and the root wrapper
   invokes `start.sh` with exactly **nine hardcoded positional slots**. The
   array could never hold more than nine entries, so `${LP_CLI_ARGS[@]:5}` gave
   at most four app names.

Same shape as §3.1 once more: the truncation had no failure path. Four apps
restored perfectly, and the success line counted the list it was handed.

Three changes, because one would not have been enough:

- the wrapper forwards the real argv after the nine slots (which stay, since
  every dispatcher reads them and unset ones must keep arriving as `"empty"`);
  `start.sh` reads it back as `"${@:10}"`. `footprint_version` 7 → 8.
- `restoreFirstRunBulk` with no list is a **whole-host restore**: it discovers
  the host's apps and re-applies the preflight itself, because the installer's
  report ran in a different process and its decision was otherwise lost — a
  skipped app would have been restored anyway. The installer now passes no
  list, so there is nothing to truncate.
- it counts what actually landed and returns non-zero naming the failures.

`scripts/dev/lp-cli-argv-test` builds stubs from the real lines in `init.sh` and
`start.sh` and pushes thirteen app names through them.

### 3.3 — Staging: created by one principal, written by another

The worst of the set, because it sat on the step everything else depends on.

`restore system` printed

```
  ✓ Success System config restored to: /libreportal-system/restore/system-config
```

for a directory that did not exist. Nothing had been written.

Restore stages through `$SYSTEM_DIR`, which the **manager** owns — but the thing
writing into the staging tree is restic, and `runBackupOp` runs it as the
**container user**. Both call sites created the directory as the wrong
principal, in opposite directions:

| Call site | Created by | Written by | Result |
|---|---|---|---|
| `backupRestoreSystemConfig` | `runFileOp` → container user | container user | `mkdir` denied on the 0751 manager-owned `restore_dir`, unchecked |
| `storageRestoreAppTo` | `runInstallOp` → manager | container user | restic could not create anything beneath it |

And restic reports a permission denial as `ignoring error …` and **still exits
0**, so the caller's success check was satisfied either way. Same family as
§3.1: the check had no failure path.

Root has to bridge that, the way `webui-bind` already bridges the mirror case.
`libreportal-ownership` gains `restore-stage <path>` (creates it `cowner:MANAGER`
0750 — owner writes, manager traverses to confirm and review) and
`restore-unstage <path>` (removes it: neither principal can, since the manager
cannot delete the container user's files inside and the container user cannot
unlink the entry from the manager's directory, so staging simply accumulated).
Both confine the path to a single component directly under the restore/migrate
area. `footprint_version` 8 → 9.

`backupRestoreSystemConfig` now also verifies the tree landed **as the user that
wrote it**, since the manager cannot read inside its own staging directory.

### 3.4 — What the run finally proved

With all of the above fixed, on a live machine:

- 13 of 13 apps restored from the repository and came up healthy, databases
  included (`bookstack`, `matrix` ×3, `mattermost` ×2, `nextcloud` ×4,
  `rocketchat` ×2, `stoat` ×10, …)
- `restore system` staged 57 real files
- the **relocation** branch of `storageRestoreAppTo` ran for the first time:
  `speedtest`, moved to a second storage location, restored from a snapshot
  taken at `/libreportal-containers/speedtest` into `/libreportal-alt/speedtest`
  via stage-and-move, with the staging tree cleaned up afterwards

One genuine environment collision remains and is **not** a LibrePortal defect:
stoat's livekit publishes a fixed UDP range (50000–50100) that it advertises to
clients and so cannot be re-rolled, and a desktop's `kdeconnectd` held 50016.
Worth a fixed-range preflight check of its own; see §9.

### 3.5 — Ownership was not actually being reinstated

`grafana` restored and then died with *"attempt to write a readonly database"*,
repeatedly. The snapshot records:

```
-rw-r----- 231543 1002  /libreportal-containers/grafana/grafana_storage/grafana.db
```

and what landed was `1002:1002`. The owner was lost, so grafana — running as
231543 — could not write its own database at mode 0640.

Restore runs as the backup user, which has no `CAP_CHOWN`, so it reinstates
ownership through a user namespace. The prefix was:

```
unshare --map-root-user --map-users=231072:231072:65536 --map-groups=231072:231072:65536
```

and `unshare` accepts **one range per option**. So the backup user's own GID was
never mapped — while LibrePortal writes app data as `<container-uid>:<backup-user>`.
The group half of every such chown referred to an unmapped id, `lchown` returned
EINVAL, and the file kept the restoring user's ownership. Measured directly:

| target | old prefix | now |
|---|---|---|
| `231543:1002` (app data) | ❌ → `1002:1002` | ✅ `231543:1002` |
| `1002:1002` (LibrePortal's own) | ❌ → `1002:1002` (right anyway) | ❌ → `1002:1002` (right anyway) |
| `231072:231072` (container root) | ✅ | ✅ |

Same shape as the rest of §3: restic reports the misses as `ignoring error …`
and **still exits 0**, and the caller forgave them as *"expected, they are
already owned correctly"*. 1626 of one 13-app restore's 2086 failed chowns were
grafana's.

**Fixed** with `scripts/backup/engine/restic-userns-exec`, which uses
`newuidmap`/`newgidmap` — those write multi-range maps, which is exactly what
`unshare` cannot express:

```
uid:  0       <- caller        inner root, or capabilities are dropped at exec
      SUB..   <- SUB..         identity, so restic can name the stored uid
gid:  caller  <- caller        identity — the group half of app-data chowns
      SUB..   <- SUB..         identity
```

The caller's own UID is deliberately not identity-mapped: that slot is spent on
inner root. A file stored as `<caller>:<caller>` therefore fails its chown and
lands owned by the caller anyway, because that is who inner root is on the
outside — so the one case this cannot map is the one case that needs no mapping.

`unshare --map-auto --map-current-user` does **not** work as a shortcut: it maps
the subuid range to low inner ids (container-style, `0 → 231072`) while restic
needs identity. Tested.

Measured on the live install, restoring grafana: failed chowns **1626 → 12**
(the 12 being the caller's own files, which are correct), `grafana.db` back to
`231543:1002`, and grafana up and writing. `scripts/dev/lp-userns-ownership-test`
pins all three rows of the table above.

Anything unexpected — no subuid range, no `newuidmap`, a namespace that will not
start — falls back to running the command plainly, which is what happened before
any of this existed.

## 4. The password problem, stated plainly

**An encrypted repository cannot be opened with anything inside itself.** `CFG_BACKUP_LOC_<idx>_PASSWORD` lives in the system config — which is *inside the backup*. So on a fresh machine the user must supply the repository password by hand. There is no way around this and it is not a bug; it is what encryption means.

That single fact drives two requirements:

- The restore step must ask for it early and say what it is, because a user who never wrote it down has no backup, and finding that out at step 2 is better than at step 5.
- Everywhere we generate that password we must push harder than a comment. The location config already says *"back up offline!"*; the WebUI should show it once, prominently, at creation — treated like a recovery key, because that is what it is.

Once the repo opens, ordering is already correct in the CLI and should be preserved: **system config first** (it carries every *other* location's credentials, so one password unlocks the rest), then apps.

### 4.1 — Where does the typed password actually travel? (blocker for phase 2)

Found while building phase 2, and it needs a decision before the restore branch
can be written, because it is a security trade-off rather than an implementation
detail.

The WebUI cannot run restic. So a password the user types in the browser has to
reach the host somehow, and the two existing channels both leak it:

| Channel | Problem |
|---|---|
| Task command string (what the Backup page already does for this exact field, via `config_update CFG_BACKUP_LOC_<n>_PASSWORD=…`) | lands in the task JSON under `frontend/data/tasks/`, which is **0644** so the manager can read it — i.e. world-readable — and is visible in `ps` while the task runs |
| A file in `frontend/data/tasks/` | the container writes as `dockerinstall`; the manager runs as `libreportal`. At 0640 the manager cannot read it (verified), and 0644 is world-readable again |

Note the first row is **existing behaviour**, not something this feature would
introduce: editing a backup location's password on the Backup page already
sends it that way. So this is a product-wide finding that phase 2 happens to
surface, and the restore case is the sharpest version of it — that password is
the key to every backup the user has.

Three ways out, roughly in order of effort:

1. **A one-shot secret drop.** The ownership helper already knows how to make a
   path readable across exactly this boundary (`_webui_bind_access` chowns
   `MANAGER:cowner` 0640 so the container can read manager-owned files). The
   reverse needs the same treatment: a root-helper-created directory owned
   `cowner:MANAGER` 0730, into which the container drops a 0640 file the manager
   reads once and unlinks.
2. **Never persist it.** Hold the password only in the task processor's memory
   for the life of the restore; write it into the location config only after the
   system-config restore lands (and reconcile with what the backup contained).
3. **Accept the existing channel** for consistency, and fix it product-wide
   later — cheapest now, and no worse than what shipping code already does, but
   it does mean a restore password sits world-readable in a task file until that
   task is pruned.

Recommendation: **(1)**, and apply it to the Backup page's password field at the
same time. It is a small, well-scoped addition to a helper that already exists
for the mirror-image case, and it fixes a live weakness rather than only
avoiding a new one.

## 5. "Set up a backup server if you don't have one"

Same components, other direction. After a **New install**, offer: *"Where should your backups go?"* — the same location fields, then `engineInit` and a first `backup system`. That closes a real gap: today backups exist but nothing prompts you to configure them, so the people most likely to need a restore are the least likely to have one.

Worth doing as its own wizard step even without the restore branch.

## 6. On "upload the backup file" — why the format differs

Worth being precise, because the mental model doesn't match the engines. restic, borg and kopia back up to a **repository** — a directory or a remote — not a single file. There is nothing to upload. The equivalents are:

- **local** — a path on a plugged-in disk. "Navigate to it" is right, and a directory picker is the natural UI.
- **sftp / s3 / b2 / …** — credentials, which the existing fields already collect.

If a genuine single-file import is wanted, that is a **different feature**: a portable per-app export (`tar` of the app dir + manifest, optionally encrypted) that could be handed around and imported. Cheap to build on the manifest that already exists, but it is not what the backup engines produce and shouldn't be conflated with them.

## 7. Portable export — the single file people actually mean

§6 explains why "upload the backup file" does not match a restic repository.
But the underlying want is real and worth serving directly: **one file, one
app, hand it around**.

    libreportal app export <app> [file]     # -> <app>-<date>.lpapp
    libreportal app import <file>

The format is deliberately boring: a gzipped tar of the app directory with its
`.libreportal-manifest.json` at the root. That manifest already records the
compose hash, images, volumes, size, databases and storage location, so import
gets the same reconciliation as §3 for free.

This also gives the installer a third answer to "where is your backup?" — a
`.lpapp` file — which is exactly the "navigate to the backup file" flow that
prompted this document.

Not a replacement for the backup engines: no deduplication, no history, no
encryption unless the user encrypts it themselves. It is a courier format —
moving one app between machines, or keeping a copy of something before a risky
change — and the docs should say so plainly so nobody uses it as their backup.

## 8. Phasing

| Phase | Deliverable |
|---|---|
| **1** ✅ | Backup destination step in the WebUI wizard (§5) — the *new setup* half |
| **2** ✅ | Two installer paths: New setup / Restore from backup, through connect → discover → system config → apps |
| **3** ✅ | Preflight reconciliation report in the installer (§3) |
| **4** ✅ | `app export` / `app import` (§7). The installer's `.lpapp` option is still open — see §9.5 |

## 9. Open questions

1. **Does the restore branch also restore the system config's *identity*** — install name, domains, WebUI credentials? Restoring the WebUI login means the user logs into the new box with the old password, which is probably what they expect, but it is a surprise if not stated.
2. **Partial restore of a host** — pick apps individually (already supported by `restoreFirstRunBulk`'s signature) or all-or-nothing at first run?
3. **What if the backup is newer than this LibrePortal version?** The manifest records the commit; refusing is safer than guessing, but it strands someone whose only copy is newer.
5. **Should the installer's restore path accept a `.lpapp` too?** `app import`
   exists, so the third answer to "where is your backup?" is a small addition —
   but a single app file is a thin thing to rebuild a *server* from, and
   offering it beside a repository may imply more than it delivers.
7. **A published range that cannot be re-rolled** — livekit advertises
   50000–50100 to clients, so the port allocator's randomisation does not apply.
   Nothing checks such a range against the host before compose-up, and the
   result is an app that comes back part-started. A preflight that compares
   fixed published ranges against `ss` output would catch it; the open question
   is what to *do* about it, since the app cannot simply be moved elsewhere.
6. **Import under a different name** is refused today: the app's `CFG_<APP>_*`
   namespace and its compose identities (container names, Traefik routers,
   backup labels) would all need rewriting. `instance create` already solves
   "a second copy", so this may never be worth building.
4. **Where does the repository password go once entered** — straight into the location config it will restore over, or held only in memory until the system config lands and then reconciled?