Resolves open question 2. One big disk holding both the app data and its snapshots is what most people actually have, and two facts make it cheap: reconcile() already chowns CONTAINERS_DIR and BACKUPS_DIR to the same container user, so there is no permission negotiation; and sibling dirs on one filesystem don't nest, so §3 admits them today unchanged. The registries stay separate — that was always about trust and lifecycle, not hardware — so §1's non-goal is reworded rather than dropped. One hard rule survives: same drive yes, nested never. A storage location containing a backup repo is a recursive-inclusion trap, and §3's nesting refusal already covers both directions. What needs work is the error — pointing storage at /mnt/bigdisk when /mnt/bigdisk/backups exists fails the empty-dir rule, and the message must suggest a subdirectory rather than saying "not empty", because that is the likely first attempt. Shared fate gets the §6.1 treatment: durable badge on both locations and a line in the backup summary, stated accurately rather than moralised — a same-drive backup still covers accidental deletion, bad updates and ransomware; what it doesn't survive is the disk dying. Also names the compounding case, since nobody pictures it: a shared drive that is also removable takes the apps and the restore path away at the same moment. Checks: free space becomes per-device (two locations on one filesystem draw from one pool, so a growing repo can starve the apps), plus a new shared-device check that warns and never refuses. Notes that this makes the naming collision worse and suggests the resolution — a Disks view with one row per device showing which roles LibrePortal has on it, registries separate underneath. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
349 lines
42 KiB
Markdown
349 lines
42 KiB
Markdown
# LibrePortal — Storage Locations (per-app data placement)
|
||
|
||
**Status:** Proposal — **not built.** · **Audience:** us, future-self · **Scope:** register more than one filesystem root for live app data, choose one per app, move an app between them, and resolve the right one on restore/migrate · **Origin:** "add different locations to set up LibrePortal on, with control per app" (2026-08-24)
|
||
|
||
---
|
||
|
||
## 0. The one idea
|
||
|
||
Today LibrePortal has **three** relocatable roots (`--system-dir` / `--containers-dir` / `--backups-dir`), each chosen once at install and fixed afterwards. This adds a fourth degree of freedom, on top of the existing containers root:
|
||
|
||
> **The containers root becomes a *list*. Every app declares which entry in that list holds its data. Everything else — install, compose, backup, restore, migrate — resolves the app's directory through one function instead of one variable.**
|
||
|
||
Nextcloud's 4 TB of photos go on the spinning disk. Vaultwarden and the control plane stay on the NVMe. Jellyfin's library lives on the external HDD that isn't always plugged in — and when it isn't, LibrePortal refuses to start Jellyfin rather than silently rebuilding it empty on the bare mountpoint.
|
||
|
||
## 1. Non-goals
|
||
|
||
- ❌ Relocating the **system** root (configs/db/logs) per-app. It stays one place, chosen at install. Same for the WebUI's own container dir.
|
||
- ❌ A general volume manager. We don't format, partition, mount, or write `/etc/fstab`. The drive must already be mounted; we validate and use it.
|
||
- ❌ Striping/tiering/RAID-alikes. One app's data lives on exactly one location. No splitting an app across two.
|
||
- ❌ Per-*volume* placement inside an app (`./data` here, `./db` there). Location granularity is the app directory. Revisit only if a real need shows up.
|
||
- ❌ Merging the backup-location and storage-location **registries**. They differ in trust, lifecycle and ownership, so they stay separate — but they may freely share a *drive*, which is a supported and expected setup (§6.2).
|
||
|
||
## 2. What already works in our favour
|
||
|
||
The current code is closer to this than it looks:
|
||
|
||
- **Compose volumes are relative.** Every shipped template uses `./data:/config`, and `dockerComposeUp` does `cd $containers_dir$app_name && docker compose …`. Change the `cd` target and the app comes up unchanged, with the same compose project name (derived from the directory *basename*, which doesn't change). The only absolute host paths in any template are `/etc/localtime`, `/dev/net/tun`, `/etc/ssl/certs` and the docker socket — none of them ours.
|
||
- **An app dir is self-describing.** `<app>/<app>.config` + `docker-compose.yml` + `.libreportal-manifest.json` is everything needed to identify and rebuild it. Discovery by scanning is viable, so we never have to trust a stale index.
|
||
- **The three-root split already exists** end-to-end: flag parsing, `libreportalValidatePaths`, the nesting/protected-path refusals, the baked `__CONTAINERS_DIR__` placeholders, and `paths.sh` as the single source of truth. We're generalising a design that's already there, not inventing one.
|
||
- **Backup locations are the exact template** for the registry UX: `configs/backup/locations/<idx>/location.config`, `locationAdd`/`locationRemove`, `sourceBackupLocations`, a `libreportal backup location …` CLI, a WebUI *Locations* page, and `backupLocationLocalGuard` — which already implements the FAT/exFAT warning and the `REQUIRE_MOUNT` refusal we need verbatim.
|
||
- **`instance create` proves the model.** "An instance is just another app" — a cloned dir with its own slug and `CFG_<SLUG>_*` namespace. Per-app placement inherits multi-instance support for free.
|
||
|
||
## 3. The central constraint — root must not be told where to `chown`
|
||
|
||
This decides the whole design, so it leads.
|
||
|
||
The manager (`libreportal`) runs with a **scoped sudoers allowlist**: it may run the root-owned helpers in `/usr/local/lib/libreportal/` and a fixed system-binary set, and nothing else. No `sudo chown`, no `sudo tee`, no `sudo bash`. The helpers have the three roots **baked in at install** by `sed` (`__CONTAINERS_DIR__` …) precisely so that:
|
||
|
||
> the manager cannot redirect a root `chown` by editing a config file.
|
||
|
||
`libreportal-ownership` even re-checks its baked roots against a dangerous-path list as defence in depth. A naïve "storage locations live in a manager-writable config, and the helpers read it" **reopens that hole completely** — `chown -R dockerinstall /etc` is a full escalation.
|
||
|
||
**The fix: a root-owned registry with an empty-directory admission rule.**
|
||
|
||
1. The truth lives at `/usr/local/lib/libreportal/storage.roots`, **root:root 0644**. One record per line: `id<TAB>path<TAB>dev<TAB>fs_uuid`. Manager reads it; only root writes it. Every helper resolves an app dir through it instead of through a single baked constant.
|
||
2. Adding a location goes through a new helper, `libreportal-storage add <path>`, which is in the sudoers allowlist — so the manager *can* call it, but it accepts a path only if **all** of these hold:
|
||
- absolute, and `realpath -e` returns the input unchanged (no symlink component, no `..`)
|
||
- not in the protected set (`/ /etc /usr /bin /sbin /lib* /boot /proc /sys /dev /run /var /tmp /root /home`), and not inside any user's home unless the install was made with `--allow-home`
|
||
- does not nest — in either direction — with the system/containers/backups roots or any already-registered location
|
||
- **is an empty directory** (tolerating only `lost+found`) **or already carries a `.libreportal-storage` marker** — the adopt case, §3.1
|
||
|
||
The emptiness rule is what makes this safe: **root only ever chowns a directory that contains nothing**, so acceptance can't hand away anything that already existed. Everything created underneath afterwards is ours by construction.
|
||
3. On acceptance the helper writes a root-owned `.libreportal-storage` marker (location id + install id + created-at), `chown`s the root to the container owner, `chmod 0751`, and appends the record. `remove` refuses while any app dir still lives there.
|
||
|
||
Consequence worth stating plainly: **`storage add` is a genuine privilege boundary crossing, not a config edit.** The WebUI can drive it (through the task system → CLI → helper, same as every other mutating action), but the *helper*, not the WebUI, is the gate.
|
||
|
||
### 3.1 — Why "empty **or** marked" is still safe (and why the marker earns its keep)
|
||
|
||
Requiring strict emptiness would break the most valuable case there is: plugging in a drive that **already holds LibrePortal app data** from another install and adopting it. So the rule relaxes to *empty or carrying our marker* — and that relaxation costs nothing, because:
|
||
|
||
> writing the marker into a directory requires already being able to write that directory.
|
||
|
||
The manager can only plant a marker somewhere it can already write, and chowning a directory it already controls grants it nothing. There is no path where the marker gets the manager access it didn't have. (The fixed-path helpers can't be tricked into writing one elsewhere — none of them takes a caller-supplied destination.)
|
||
|
||
That single file then does **three** jobs, which is the main reason to like it:
|
||
|
||
| Job | How |
|
||
|---|---|
|
||
| Admission | "empty or marked" — §3 |
|
||
| **Mount detection** | the marker lives *on the drive*. Not mounted ⇒ bare mountpoint ⇒ no marker ⇒ location unavailable. No `findmnt`, no `fs_uuid` bookkeeping, and it works identically for USB disks, network mounts, and LUKS volumes that haven't been unlocked (§10.1) |
|
||
| Provenance on migrate | it carries the install id and location id the snapshot's manifest names, so "is this the same `bigdisk` the app came from?" is a file read (§9) |
|
||
|
||
**Residual wrinkle, not solved by elegance:** validate-then-`chown` is a TOCTOU window, and bash is a poor language for race-free path handling. The practical closure is to additionally require that the **parent** directory is not manager-writable — true for `/mnt`, `/srv`, `/media`, false for a path inside the manager's home. That's a real restriction on where locations may live, not a free lunch, and it should be stated in the docs rather than hidden.
|
||
|
||
## 4. The resolution layer — one function, ~200 call sites
|
||
|
||
The mechanical bulk of the work. Three new primitives in `scripts/source/paths.sh` (and mirrored inline in `init.sh`, per the existing keep-in-sync note):
|
||
|
||
```bash
|
||
storageRoots # every enabled root, primary first
|
||
appDir <slug> # the app's directory — memoised
|
||
pathIsContainerData <p> # is this path under ANY container root?
|
||
```
|
||
|
||
- **`appDir`** builds a `slug → dir` map once per process by scanning each enabled root for `*/<slug>.config`, memoises it in an associative array, and falls back to the primary root for a slug that doesn't exist yet (fresh install). Discovery-first, config-second: if the map and `CFG_<APP>_STORAGE` disagree, **the directory on disk wins** and we warn. That's what makes the system self-healing after a hand-move or a half-finished migration.
|
||
|
||
It is also **the single place the availability check belongs**. Every one of the ~200 sites has to call it, so a location whose drive is missing fails *once*, centrally, instead of needing a guard sprinkled at each caller. `appDir` returns non-zero **and** prints an unusable sentinel path, so the many callers that won't check `$?` still fail loudly on a path that cannot exist, rather than writing into a bare mountpoint. See §10.1.
|
||
- **`pathIsContainerData`** replaces the `[[ "$p" == "$containers_dir"* ]]` idiom that decides manager-vs-container-user elevation. It appears in ~10 files (`create_folder.sh`, `create_touch.sh`, `copy_file.sh`, `copy_files.sh`, `copy_folder.sh`, `copy_folders.sh`, `move_file.sh`, `runCfgOp`, `tags_manager_update.sh`, `webui_atomic_write.sh`). **Every one is a silent-corruption bug if missed** — a file under an unrecognised root gets written as the manager, lands with the wrong owner, and the container fails to read it at a moment far removed from the cause.
|
||
|
||
Scale of the sweep (measured, not estimated):
|
||
|
||
| Pattern | Count | Action |
|
||
|---|---|---|
|
||
| `$containers_dir$app`-shaped, app-scoped | ~186 across ~90 files | → `$(appDir "$app")` |
|
||
| `$containers_dir/libreportal/…` (the WebUI's own tree) | 63 | → `webuiDir` helper, **pinned to the primary root** |
|
||
| `$containers_dir/{traefik,prometheus,grafana,adguard,…}` — one app reaching into another by literal name | ~29 | → `appDir <name>`; those apps stay pinned in phase 1 (§10.3) |
|
||
| root-helper `$CONTAINERS_DIR/$app` | `libreportal-ownership`, `libreportal-appcfg` | → registry lookup inside the helper |
|
||
| the `== "$containers_dir"*` elevation test | ~10 | → `pathIsContainerData` |
|
||
|
||
`sourceScanFiles app_configs` must also loop the roots. Keep the existing `-maxdepth 3` + prune rules exactly as they are — and remember `*.config` is a reserved extension anywhere under an app dir.
|
||
|
||
**Finding the sites we missed.** A 90-file mechanical diff is exactly where a silent regression hides, and review alone won't catch a `$containers_dir$app` that survived. Make the runtime find them instead: once the sweep lands, **stop defining `containers_dir` as a usable path** and point it at a sentinel (`/nonexistent-libreportal-unconverted/`). Legitimate primary-root users have moved to `primaryRoot`/`webuiDir` by then, so every survivor now fails immediately, loudly, and *harmlessly* — on a path that cannot exist — with a sentinel string that greps straight out of the logs. Silent wrongness becomes a stack trace. Then a `scripts/dev/` linter keeps the pattern from coming back, the way `lp-task-names` guards task titles.
|
||
|
||
## 5. Per-app configuration
|
||
|
||
One new key in every app template, defaulting to the primary location:
|
||
|
||
```
|
||
# STORAGE = which storage location holds this app's data (see Storage on /admin)
|
||
CFG_BOOKSTACK_STORAGE=default
|
||
```
|
||
|
||
- The value is a **location name**, not a path. Names are what survive a migrate to a host whose disks are laid out differently; paths aren't. `default` always resolves to the install-time containers root.
|
||
- It renders as a dropdown on the app's config page for free — the WebUI config renderer already builds selects from the `[a:A|b:B]` comment convention; the option list is generated from the registry into the config comment on regen.
|
||
- Changing it in the config editor does **not** move data. It records intent; the move is `libreportal app move` (§8). The config page shows the current *resolved* directory next to the field, and flags a mismatch.
|
||
- `configBackfillMissingKeys` carries the new key into already-installed apps on the next update, so nothing needs a reinstall.
|
||
- **New compose tag** `#LIBREPORTAL|APP_DIR_TAG|APP_DIR_DATA`, filled in `dockerConfigSetupFileWithData` alongside the existing `CONTAINERS_DIR_TAG`, for the rare template that genuinely needs its own absolute host path. Most apps need nothing — relative volumes already do the right thing.
|
||
|
||
### 5.1 — Why the value is a name and not `CFG_<APP>_STORAGE_PATH`
|
||
|
||
A path reads better in isolation — `grep STORAGE *.config` would tell you where everything lives without consulting anything. It loses on the two things that actually happen:
|
||
|
||
- **Mount paths move.** External drives arrive at `/media/<user>/<label>` or `/run/media/…`, and those change on relabel, on a different desktop session, on a distro that mounts differently. With a name, re-register the location's path once and every app on it follows. With a path, every app config has to be rewritten — N edits, each able to fail halfway, and a half-finished pass leaves some apps agreeing with `appDir` and some not.
|
||
- **Migrate is a lookup, not a string match.** This feature's motivating case is "the restored config names a location this host doesn't have — ask the user." A name makes that a clean miss against the registry. A path makes it a comparison against a string that never meant anything on this host, and it can't distinguish "different disk layout" from "typo".
|
||
|
||
Debuggability doesn't require the path be *stored*, only *retrievable*: `libreportal storage list`, `libreportal app info <app>`, and the resolved directory shown beside the field on the config page (above) all print it. The split that keeps this honest — **the config stores the name because it is intent; the manifest (§9) stores name *and* path *and* `fs_uuid` because it is a record of fact.**
|
||
|
||
Note this deliberately diverges from the backup-location template, which offers `PATH_MODE=auto|custom` + an explicit `PATH`. There, restic runs as an ordinary user and can write anywhere it has permission, so a custom path is genuinely useful. Here, an app directory needs ownership that only root can establish, and root only touches registered roots (§3) — so a path outside the registry isn't discouraged, it's **unusable**. There is nothing for an override to override.
|
||
|
||
### 5.2 — The resolved path rides in the comment
|
||
|
||
The name is the value; the **path is written into the field's own comment**, so the file answers "where does this actually live?" without any tooling — the 2am-recovery case that makes a path attractive in the first place:
|
||
|
||
```
|
||
CFG_NEXTCLOUD_STORAGE=bigdisk # Storage Location - Currently at /mnt/bigdisk/apps/nextcloud [default:Primary (…)|bigdisk:Big disk]
|
||
```
|
||
|
||
Rules that keep this from becoming a liability:
|
||
|
||
- **Written only when it changes** — on install, on `app move`, and on a regen that finds it stale. Never unconditionally on every regen: the app `.config` is user-editable and lives in the container-owned tree, so a no-op rewrite is both file churn and an avoidable `runFileOp` on every pass. Compare, then write only on a difference.
|
||
- **It is a breadcrumb, not a source of truth.** `appDir` resolves by discovery; nothing ever *reads* this comment to decide anything. A stale one is cosmetic — and the regen self-heals it.
|
||
- **The WebUI panel doesn't rely on it.** `webui_generate_configs.sh` composes the field's description from `appDir` at generation time, so the editor shows the live path even if the on-disk comment hasn't caught up.
|
||
- The dropdown option list (`[default:…|bigdisk:…]`) is regenerated from the registry in the same pass, which is what makes the selector reflect locations added since the app was installed.
|
||
|
||
## 6. Fitness checks — what a location must pass before it can hold data
|
||
|
||
Admission (§3) answers *"is it safe for root to accept this path?"*. That is a **security** question and it lives in the root helper. It does not answer *"will app data actually work here?"* — a separate, larger question that runs in the manager, needs no privilege, and therefore can run **speculatively on a candidate the user hasn't chosen yet**, which is what lets the wizard (§7) show a verdict per disk before anything is committed.
|
||
|
||
Split them deliberately: security checks refuse, fitness checks grade. And **a fitness check only refuses when the location cannot work at all** — never merely because it's inconvenient or needs care. A removable drive is a supported configuration, not a mistake to be prevented (§6.1).
|
||
|
||
| # | Check | On failure | Why it matters |
|
||
|---|---|---|---|
|
||
| 1 | **Filesystem type** — `findmnt -no FSTYPE` | **refuse** on `vfat`/`exfat`/`ntfs`/`ntfs3`/`msdos`/`fuseblk` | no POSIX ownership ⇒ a rootless app dir is broken from the first write. Stricter than backups, where `backupLocationLocalGuard` only warns |
|
||
| 2 | **Mount options** — `findmnt -no OPTIONS` | **refuse** `ro` and `noexec`; note `nosuid`/`nodev` | some apps execute out of their data dir; a read-only remount is the silent killer |
|
||
| 3 | **Real ownership probe** | **refuse** | the decisive test, and the only one that catches NFS `root_squash` — which reports a perfectly respectable `nfs4` at check 1 and then can't `chown`. Create a temp dir, chown it to the container user, `stat` it back, remove it |
|
||
| 4 | **Sub-UID probe** | **refuse** | rootless docker maps container users into the 100000+ range. Chown the probe to a high uid; some FUSE and network mounts simply can't hold it, and the failure otherwise surfaces much later as an app that won't start |
|
||
| 5 | **Write / fsync / read-back** | **refuse** | catches a full disk, a flaky USB bridge, a silently-degraded mount |
|
||
| 6 | **Persistence across reboot** — `findmnt --fstab`, else a systemd `.mount` | **loud, durable warning — never refuse** | hand-mount a disk, register it, install Nextcloud, reboot, and the apps there won't start. That's worth saying clearly and repeatedly; it isn't worth blocking (§6.1). We don't write `fstab` (§1), but we say exactly what to add |
|
||
| 7 | **Removable / hot-plug** — `lsblk -o RM,HOTPLUG` | **loud, durable warning** + marker discipline | an external drive is a supported setup. The user gets told plainly what happens when it's absent — once at registration, and standing on the location afterwards |
|
||
| 8 | **Distinct device** — compare `st_dev` with the primary root | warn | same disk ⇒ the location buys nothing. Usually a misunderstanding, not an error |
|
||
| 9 | **Free space — per *device*, not per location** | refuse under a floor, warn under a fraction | locations can share a filesystem with each other and with a backup repo (§6.2), so they draw on one pool. Sum every LibrePortal consumer on the device |
|
||
| 10 | **Encryption at rest** — is it a LUKS/dm-crypt mapping? | informational | privacy-first product; worth surfacing, never worth blocking |
|
||
| 11 | **Shared device with a backup location** — `st_dev` vs every registered backup location | **warn, never refuse** | shared fate: one disk failure takes the data and its only snapshots (§6.2) |
|
||
|
||
Two more things this wants:
|
||
|
||
- **`libreportal storage check [<id>]`** runs the whole table against a registered location, and against an arbitrary path in *dry-run* mode (`--candidate`) so the wizard, the CLI and the WebUI all share one implementation. It fits the existing `scripts/checks/requirements/check_*.sh` convention.
|
||
- **It must repeat, not just gate at add time.** A location that passed in March can be full, remounted read-only, unplugged, or backed by a dying disk in June. Put `storage check` on the existing crontab alongside the other periodic checks and surface degradation as a dashboard warning — the same way a failing backup surfaces today.
|
||
|
||
### 6.1 — Why a non-persistent or removable drive warns instead of refusing
|
||
|
||
Blocking these was tempting and would have been wrong. "This drive isn't in `fstab`" and "this drive is removable" both describe a **supported setup** — the media library on the USB disk is one of the reasons to want this feature at all — not a broken one. A refusal there fails the honest user on their first attempt to do something reasonable, and teaches them to look for an override flag rather than to read the warning.
|
||
|
||
The reason we can afford it: **the dangerous moment is start-up, not registration, and start-up is already gated.** The marker file (§3.1) means a location whose drive is absent simply isn't available; `appDir` fails and `dockerComposeUp` refuses (§10.1). An app on an unmounted drive doesn't quietly rebuild itself empty — it doesn't start. Given that gate, refusing at registration is belt-and-braces that costs a legitimate use case and buys nothing the runtime doesn't already enforce.
|
||
|
||
So the obligation moves from *prevent* to *inform*, and "loud" has to mean **durable**, not a toast that scrolls away:
|
||
|
||
- an explicit warning on the wizard/CLI card at registration, stating the consequence in plain words — *"apps stored here will not start until this drive is mounted"* — plus the exact `fstab` line (or `systemd.mount` unit) that would make it permanent, ready to copy
|
||
- a **standing badge** on the location in the WebUI Storage page and in `storage list`, for as long as the condition holds. Six months later, when an app won't start, the reason should be visible without archaeology
|
||
- the resulting start-up refusal must name the cause: *"nextcloud is on location `bigdisk`, which is not mounted"* — never a generic failure
|
||
- `storage check` re-reports it on its periodic run, so a drive that *was* in `fstab` and silently stopped being one gets caught
|
||
|
||
A one-time acknowledgement at registration ("I understand apps here won't start unless this drive is mounted") is probably worth it too — it makes the choice deliberate without making it hard. That's a UX call, not an architectural one.
|
||
|
||
### 6.2 — Sharing a drive with a backup location
|
||
|
||
**Supported, deliberately.** One big disk holding both the app data and its snapshots is a completely reasonable home setup, and it's what most people will actually have. The registries stay separate (§1) — that's about trust and lifecycle, not hardware — but nothing stops the two from landing on the same device, and two facts make it cheap:
|
||
|
||
- **Ownership already matches.** `reconcile()` chowns both `CONTAINERS_DIR` and `BACKUPS_DIR` to the same container user, so a shared drive has one owner and no permission negotiation.
|
||
- **The existing nesting rule is already the right rule.** Sibling directories on one filesystem — `/mnt/bigdisk/apps` and `/mnt/bigdisk/backups` — don't nest, so §3 admits them today with no change.
|
||
|
||
**The one hard rule: same drive yes, nested never.** A storage location that *contains* a backup repo (or the reverse) is a recursive-inclusion trap — the backup engine walking a tree that holds its own repository. §3's nesting refusal already covers it in both directions and must not be relaxed for the shared-drive case. What does need work is the error: pointing storage at `/mnt/bigdisk` when `/mnt/bigdisk/backups` already exists fails the empty-directory rule, and the message must say *"use a subdirectory such as `/mnt/bigdisk/apps`"* rather than a bare "not empty" — that's the single most likely first attempt.
|
||
|
||
**The honest warning: shared fate.** One disk failure loses the data *and* the only copy of its backups. That deserves saying plainly and permanently, with the same durable-not-a-toast discipline as §6.1 — a standing badge on both the storage and backup location, and a line in the backup summary along the lines of *"3 apps have no copy off this device."*
|
||
|
||
Say it accurately, though, rather than moralising: a same-drive backup still protects against **accidental deletion, a bad update, a botched migration, and ransomware** (with append-only enabled). What it does not survive is the disk dying. Both halves are true and the user should get both — the failure it doesn't cover is specific and nameable, not a general "this is wrong."
|
||
|
||
Worth noting the compounding case explicitly, because it's the one people don't picture: if the shared drive is also *removable* (§6.1), then unplugging it takes the apps **and** the restore path away at the same moment. Neither warning implies the other; a location that is both should say both.
|
||
|
||
**Two things this changes in the checks:**
|
||
|
||
- **Free space becomes per-device, not per-location** (check 9). Two locations on one filesystem draw from one pool, so a growing restic repo can starve the apps sharing the disk — a real new failure mode where before there were two independent budgets. Account for every LibrePortal consumer on the device, keep a reserve, and warn on the *device*, not the location.
|
||
- **Add a shared-device check** — `st_dev` of a candidate storage location against every registered backup location, and vice versa. Never refuse; emit the shared-fate warning above.
|
||
|
||
**Two things it enables**, both nearly free:
|
||
|
||
- the picker on either side can **offer known drives** as suggestions, with the shared-fate note inline, instead of making the user type a path twice
|
||
- the storage marker (§3.1) is a strictly better mount test than the backup subsystem's `findmnt` probe — it catches a mountpoint shadowed by a stale directory, which `findmnt` reports as mounted. A backup location on a drive that also hosts a storage location can borrow it.
|
||
|
||
**Consequence for §13.1 (naming).** Sharing makes the collision worse, not better: the user now genuinely sees "bigdisk" in two places meaning two things. The natural resolution is to stop treating the two registries as the top-level concept in the UI and introduce a **Disks** view — one row per device, showing which roles LibrePortal has on it (*app data*, *backups*, or both), free space for the device as a whole, and mount state. The registries stay separate underneath; the user just stops having to hold that distinction to understand their own hardware. Recommended, but a bigger UI call than this doc should make alone.
|
||
|
||
## 7. Surfaces — first-run wizard, config panels, CLI
|
||
|
||
**First-run wizard.** The setup wizard (`core/setup/js/setup-wizard.js`) currently runs `Experience → Identity → Domains → Recommended → Metrics`, and already has the pattern for a step that isn't always shown: Metrics is advanced-only, and `_effectiveTotalSteps()` makes the count dynamic. A **Storage** step slots in **before Recommended** — locations must exist before apps get placed on them — and follows the same conditional rule, with a better trigger:
|
||
|
||
> **Only show it if the box actually has somewhere else to put things.** One disk, no candidates ⇒ the step never appears, and the 90 % case is unchanged.
|
||
|
||
Candidate detection is `lsblk` + `findmnt`, filtering the noise aggressively — `loop`/`squashfs`/snap mounts, `/boot`, `/boot/efi`, swap, `tmpfs`, `overlay`, anything under `/proc`, `/sys`, `/run`, and the filesystem already holding the primary root. On this dev box that filter has to drop ~15 snap loop devices to find the real answer, so it isn't optional. Each surviving candidate renders as a card: device, size, free, filesystem, mountpoint, and its §6 verdict — green / warned / refused. The three behave differently on purpose:
|
||
|
||
- **green** — selectable, no friction
|
||
- **warned** — fully selectable, with the consequence stated on the card and carried forward as a standing badge (§6.1). A USB drive lands here. It is *not* second-class; it's the reason the feature exists
|
||
- **refused** — shown **greyed with the reason**, never hidden. "Why isn't my drive listed?" is a support burden we don't need, and the reason ("exFAT can't store file ownership") is usually actionable
|
||
|
||
Refusals are reserved for locations that genuinely cannot work (checks 1–5). Nothing is blocked for merely needing care.
|
||
|
||
Registering from the wizard goes through the task system → CLI → `libreportal-storage` helper, exactly like every other mutating action. No chicken-and-egg: by the time the wizard runs, the manager is de-sudo'd but the helper is already in its scoped allowlist.
|
||
|
||
Nice follow-on, nearly free: the **Recommended** step comes next, so once a big disk is registered the bulky picks (Nextcloud, Jellyfin, Immich) can default onto it in the same payload — the wizard already posts one JSON blob to `setupApplyConfig`.
|
||
|
||
For unattended installs the wizard isn't there, so keep an `init.sh --storage-dir=` flag that pre-registers extra locations at install time (§13.5 resolved: build both — the flag is trivial once the helper exists).
|
||
|
||
**Config panels.** Two distinct editors, both generated:
|
||
|
||
- **Per-location config** — `configs/storage/locations/<id>/location.config` holding `CFG_STORAGE_LOC_<id>_*` (`NAME`, `ENABLED`, `NOTES`, `REQUIRE_MOUNT`…). `webui_generate_configs.sh` already has a bespoke loop for exactly this shape for backup locations (`configs/backup/locations/*/location.config`, which the flat-file scan can't reach at that depth) — the storage loop is a near-copy, and the WebUI *Storage* page renders from config metadata like every other page. Note the **path is not among these fields**: it's registry data, root-owned, and changing it is `storage add`/`remove`, not a config edit.
|
||
- **Per-app field** — the `CFG_<APP>_STORAGE` dropdown described in §5, on the app's existing config page.
|
||
|
||
**CLI.** `libreportal storage {list,add,remove,check,scan}` mirroring `libreportal backup location …` — `scan` being the candidate detection the wizard uses, so a headless user gets the same view.
|
||
|
||
## 8. Moving an installed app
|
||
|
||
`libreportal app move <app> <location>` — a task like any other, with progress rows in the WebUI.
|
||
|
||
1. Resolve source and destination; refuse if they're the same, if the app isn't installed, or if the destination isn't mounted/writable.
|
||
2. **Space check** with headroom: `du -sb` of the source vs `df` of the destination, refuse under 110 %.
|
||
3. Compose **down** (data must be quiescent — a live copy of a running Postgres is a corrupt copy).
|
||
4. **Pre-move snapshot** to the first enabled backup location, reusing `migratePreBackupDestination`. Skippable with `--no-pre-backup`, on by default.
|
||
5. `runOwnership app-move <app> <dest-id>` — the copy runs as root inside the helper because app data contains rootless sub-UID files (postgres at uid 231141 etc.) that the manager can neither read nor recreate.
|
||
- same filesystem → `mv` (atomic, instant)
|
||
- across filesystems → `cp -a --reflink=auto` → verify → **only then** remove the source. Never `mv` semantics that could half-delete.
|
||
6. Update `CFG_<APP>_STORAGE`, re-run `dockerComposeUpdateAndStartApp` (the `cd` target moves with `appDir`), bring it up, and health-check.
|
||
7. On any failure before step 6 the source is untouched; after step 6, roll back by pointing the config at the source and restarting.
|
||
|
||
## 9. Restore and migrate — the interesting case
|
||
|
||
This is the scenario that motivated the feature: *"we migrate another install, it checks the path in the config against our current locations, and if there's no match the user picks an existing one or sets up a new one."*
|
||
|
||
**Add to `.libreportal-manifest.json`** (written into the app dir, so it rides inside the snapshot):
|
||
|
||
```json
|
||
"storage": { "location": "bigdisk", "path": "/mnt/bigdisk/apps/nextcloud", "fs_uuid": "…" }
|
||
```
|
||
|
||
**On restore/migrate**, resolve in this order: exact location-name match → a location whose `fs_uuid` matches → **ask**. The "ask" is a real WebUI step in the migrate wizard — *"This app came from location `bigdisk` (`/mnt/bigdisk/apps`), which this host doesn't have. Restore it to: [ default ▾ ] or [ + add a location ]"* — with a CLI equivalent (`--storage=<name>`) for unattended runs.
|
||
|
||
**The blocker that must be fixed first.** Restore currently does:
|
||
|
||
```bash
|
||
engineRestoreSnapshot "$idx" "$id" "/" "$containers_dir$app"
|
||
```
|
||
|
||
— restore to `/` with an include filter built from the **local** containers root. Restic reproduces the snapshot's absolute paths, so this only works when source and destination paths are byte-identical. That means:
|
||
|
||
- relocating on restore is impossible today, and
|
||
- **this is already broken** for the shipped three-root feature: migrating from a host installed with `--containers-dir=/mnt/ssd/apps` onto a default host matches no include path and restores *nothing*, quietly.
|
||
|
||
The fix is needed for this feature and is worth doing on its own: restore into `$restore_dir` staging with the snapshot's own path prefix, then move the tree into `appDir`. `resticRestoreAppLatest`, `restore_app_start.sh`, `migrate_apply.sh` and `migrate_preflight.sh` all take the source path from the snapshot/manifest rather than from the local `containers_dir`.
|
||
|
||
Also worth noting: an app's restic snapshot paths change when it moves. Snapshot *history* stays intact and restorable (each snapshot knows its own path), but path-based filters in the snapshot browser need to accept either.
|
||
|
||
## 10. Risks, in the order they'll bite
|
||
|
||
**10.1 — An unplugged drive is the top data-integrity risk.** `crontab_boot_app_reconcile.sh` brings **every** installed app up at boot. If a location's drive isn't mounted yet — or at all — docker cheerfully creates the bind-mount directories on the bare mountpoint and the app boots **empty**. Plug the drive back in and there are now two divergent copies, with the good one hidden underneath the mount. Mitigation is mandatory, not optional:
|
||
|
||
The mitigation is **one test, not a family of them**: `.libreportal-storage` lives on the drive, so *marker present* ⇒ mounted, and *marker absent where the registry says a location is* ⇒ not mounted, refuse. That subsumes `findmnt`, `REQUIRE_MOUNT` and `fs_uuid` comparison into a single file read, and it behaves correctly for cases those don't cover — a LUKS volume that hasn't been unlocked, an NFS/SMB mount that dropped, a mountpoint shadowed by a stale directory.
|
||
|
||
Where it's enforced:
|
||
|
||
- **`appDir` fails** on an unavailable location (§4) — the central gate, hit by every caller by construction
|
||
- **`dockerComposeUp` refuses** to start such an app — the gate that actually protects data, since it also covers manual and task-triggered starts, not just boot
|
||
- boot reconcile skips those apps with a loud notice and a dashboard warning rather than failing silently
|
||
- keep `fs_uuid` in the registry anyway, but as *diagnostics* ("this is a different disk than the one registered"), not as the liveness test
|
||
|
||
**This gate is load-bearing.** Because registration deliberately does not block removable or non-persistent drives (§6.1), it is the *only* thing standing between an unmounted disk and an app rebuilding itself empty on the bare mountpoint. Weakening it — a "just start anyway" escape hatch, a caller that resolves a path without going through `appDir` — re-arms the exact failure §6.1 assumes is impossible. Anything that relaxes it has to revisit that decision too.
|
||
|
||
**10.2 — Missing an elevation-test site** (§4) produces wrong-owner files that fail much later. The sentinel-plus-linter approach in §4 is the mitigation: make the survivors fail loudly during the sweep, then make the pattern un-reintroducible.
|
||
|
||
**10.3 — Cross-app literal paths.** Traefik, Prometheus, Grafana, AdGuard, Gluetun, CrowdSec, Headscale are reached into by name from other apps' code. They're also small and infrastructural, so pinning them to the primary location costs a user nothing real.
|
||
|
||
Express it with **no new key at all.** "Pinned" is not a fact about the app's storage, it's a statement about whether the field is editable — and the config format already has a place for that: the comment. `**ADVANCED**` is parsed out of a field's comment today (`webui_generate_configs.sh:326`) to drive the editor's Advanced reveal. Add `**READONLY**` beside it, parsed the same way, and these templates simply ship:
|
||
|
||
```
|
||
CFG_TRAEFIK_STORAGE=default # Storage Location - Fixed: other apps reference Traefik by path **READONLY**
|
||
```
|
||
|
||
The value stays a normal `CFG_<APP>_STORAGE`, so `appDir` and the migrate path need no special case; only the renderer treats it differently. Relaxing an app later is deleting one token from a comment plus converting its call sites. And `**READONLY**` immediately earns its keep elsewhere: derived fields already warn in prose that nothing enforces — `crowdsec.config:72` literally says *"editing this value does not re-register the bouncer"* next to an editable input.
|
||
|
||
The WebUI's own dir stays pinned permanently and *structurally* (`webuiDir`), not by any config marker.
|
||
|
||
**10.4 — Filesystem capability.** Superseded by the fitness checks (§6), which is where this risk got its answer: checks 1–5 turn "hope the filesystem is suitable" into a refusal with a stated reason, and check 3 (a real chown probe) is what catches the cases type-sniffing alone misses.
|
||
|
||
**10.5 — `du`/`df` reporting.** The dashboard's disk gauge reads `/` only. With apps spread over disks it needs one gauge per location, or the number is actively misleading.
|
||
|
||
**10.6 — Uninstall/teardown.** `init.sh uninstall` prints and removes a fixed set of paths. It must enumerate the registry, and default to **leaving** external locations alone (with an explicit `--remove-storage` to wipe them) — an external drive is not ours to erase.
|
||
|
||
**10.7 — Footprint version.** New root helper + changed baked helpers ⇒ **bump `footprint_version`** (currently 5), or existing installs update the manager-owned code and keep root helpers that don't know about the registry.
|
||
|
||
## 11. What doesn't get a clean answer
|
||
|
||
Most of §10 collapses into a small number of good primitives — the marker file, `appDir`, the emptiness rule, a comment marker. Four things don't, and pretending otherwise would set us up to be surprised later.
|
||
|
||
- **The TOCTOU window in `libreportal-storage add`.** Validate-then-`chown` can't be made atomic in bash. The closure is a *restriction* — the parent directory must not be manager-writable — which rules out locations inside the manager's home and has to be documented as a rule users will occasionally hit and dislike.
|
||
- **Two registries that look like one.** Storage locations and backup locations will sit next to each other in the UI, both pointing at drives, with different lifecycles and different trust. No amount of engineering fixes that; it's a naming and information-architecture problem (§13.1) and the honest options are a genuinely different word or a single combined "Disks" page that owns both roles.
|
||
- **The sweep is big no matter how good the target API is.** ~90 files. Elegance makes the *result* better and the pattern un-reintroducible; it does not make the diff safe. Only the sentinel, the linter, and a full install → backup → restore → migrate soak on a real box do that — which is why phase 0 ships alone.
|
||
- **Moving a large app is just slow.** Cross-filesystem `cp -a` of a multi-terabyte Nextcloud is hours, needs root for sub-UID data, and the app is down throughout. There is no clever version. The deliverable is honesty about it: size estimate and duration warning up front, real progress in the task row, resumable on interruption, and never a delete of the source until the copy verifies.
|
||
|
||
## 12. Phasing
|
||
|
||
Each phase is independently shippable and independently verifiable. Phase 0 carries almost all of the regression risk and none of the user-visible value — land and soak it alone.
|
||
|
||
| Phase | Deliverable | Visible? |
|
||
|---|---|---|
|
||
| **0** | `appDir` / `storageRoots` / `pathIsContainerData` / `webuiDir`; sweep the ~200 call sites; helpers read a registry that contains exactly one root. Full install + backup + restore verified. | No |
|
||
| **1** | Root-owned registry + `libreportal-storage` helper + the §6 fitness checks + `libreportal storage {list,add,remove,check,scan}`. Locations can exist; nothing uses them yet. | CLI only |
|
||
| **2** | `CFG_<APP>_STORAGE` + the `**READONLY**` comment marker, resolved path in the comment (§5.2), install-time placement, per-location config panels and disk gauges. | Yes |
|
||
| **3** | Setup-wizard **Storage** step (§7) + `--storage-dir=` for unattended installs. Depends on 1 and 2; deliberately after them, so first-run drives a path that already works headlessly. | Yes |
|
||
| **4** | `libreportal app move`. | Yes |
|
||
| **5** | Manifest `storage` block, staged restore + path rewrite (fixes §9's latent bug), the "unknown location" prompt in migrate. | Yes |
|
||
| **6** | WebUI **Storage** page mirroring *Backup → Locations*: add/remove, mount state, free space, apps per location, drag-to-move. Periodic `storage check` + dashboard degradation warnings. | Yes |
|
||
|
||
## 13. Open questions
|
||
|
||
1. **Naming.** "Storage location" vs "backup location" in the same UI — is that confusing enough to want a different word (*volume*? *disk*? *pool*?) for one of them?
|
||
2. ~~**Can a storage location double as a backup location?**~~ — **resolved (2026-08-24):** yes, sharing a drive is supported. Registries stay separate, sibling directories only (never nested), with a durable shared-fate warning and per-device space accounting. See §6.2.
|
||
3. **Does the `libreportal` app itself ever get to move?** Currently pinned. If the primary root fills up, that's a reinstall — acceptable, or worth solving?
|
||
4. **Per-instance vs per-type placement.** Instances inherit the model for free, but should `instance create` offer a location up front (likely yes — "work Nextcloud on the big disk, family on the SSD" is a good demo)?
|
||
5. ~~**`--storage-dir=` install flags**~~ — **resolved:** build it. The wizard covers the interactive case, but unattended installs have no wizard, and the flag is a few lines once the helper exists (§7).
|
||
6. ~~**How hard should check 6 (reboot persistence) refuse?**~~ — **resolved (2026-08-24):** it doesn't. Warn loudly and durably, never block; removable drives are a supported setup, and start-up is already gated by the marker test. No `--allow-transient` flag — there's nothing left for it to unlock. See §6.1.
|
||
7. **Docs promise.** `docs/guide/install-and-use.md` currently states the roots are *"chosen at install and fixed afterward … part of the security model."* That stays true of the three roots; the guide needs a paragraph explaining that additional **containers** locations are addable at runtime, and why the empty-directory rule keeps that honest.
|