# Adding monitoring for an app

There are two shapes. Pick the one that matches the app:

1. **Native** — the app speaks Prometheus on a `/metrics` endpoint. Just point Prometheus at it.
2. **Sidecar** — the app has no native /metrics. Run an exporter container alongside the app; Prometheus scrapes the exporter.

Both use the same plumbing (`monitoringToggleAppConfig` + `monitoringRefreshAll`) and the same on-disk layout under `containers/<app>/resources/monitoring/`. The only thing that differs is what's inside the marker block in the app's compose.

## On-disk layout

```
containers/<app>/
├── <app>.config                         # CFG_<APP>_MONITORING=false (default off)
├── <app>.sh                             # calls the two monitoring helpers
├── docker-compose.yml                   # marker block: env vars OR sidecar service
└── resources/
    └── monitoring/
        ├── prometheus-scrape.yml        # one or more scrape jobs
        └── grafana-dashboards/
            └── <app>.json               # dashboard(s) to provision
```

When `CFG_<APP>_MONITORING=true`, `monitoringRefreshAll` (in `scripts/network/monitoring/monitoring.sh`) copies the scrape fragment into `prometheus/scrape.d/<app>.yml` and the dashboard(s) into `grafana/provisioning/dashboards/libreportal/`, then SIGHUPs Prometheus and restarts Grafana.

## The marker block

Anywhere in the compose you put a block like this:

```yaml
# >>> libreportal-monitoring >>>
#  …lines that should be uncommented when MONITORING=true…
# <<< libreportal-monitoring <<<
```

`monitoringToggleAppConfig <app> <relpath>` strips one leading `#` from every non-marker line in the range when monitoring is enabled, and re-adds it when disabled. The marker lines themselves are never touched.

## Native pattern

For apps with built-in /metrics (e.g. traefik, gitea, headscale, grafana):

- If the metrics endpoint is gated behind config (e.g. gitea needs `GITEA__metrics__ENABLED=true`), put the env vars / config lines inside a marker block in the compose or app config file.
- Scrape fragment targets the app service directly: `targets: ['<app>-service:<port>']`.
- If the endpoint requires a bearer token, generate it via the `CFG_<APP>_METRICS_TOKEN=RANDOMIZEDPASSWORD<N>` mechanism, mirror it into the scrape fragment in `<app>.sh` with a sed substitution against a `<APP>_METRICS_TOKEN_PLACEHOLDER` (see `gitea.sh` for the pattern).

## Sidecar pattern

For apps without native /metrics (pihole, adguard, wireguard, nextcloud, unbound, gluetun, vaultwarden, ollama):

Add the exporter as a sibling service inside the marker block:

```yaml
services:
  <app>-service:
    …

  # >>> libreportal-monitoring >>>
  #<app>-exporter:
  #  container_name: <app>-exporter
  #  image: <vendor>/<app>-exporter:<tag>
  #  restart: unless-stopped
  #  environment:
  #    - …pull credentials from existing CFG_<APP>_* via #LIBREPORTAL tags…
  #  networks:
  #    DOCKER_NETWORK_DATA:
  # <<< libreportal-monitoring <<<
```

Rules:

- **Service name**: `<app>-exporter`. Prometheus scrapes `<app>-exporter:<port>`. No host port mapping — the metrics endpoint stays on the docker network only.
- **Network**: attach to the same `DOCKER_NETWORK_DATA` as the main service. No explicit IP needed; service-name DNS works.
- **Port**: add a `CFG_<APP>_PORT_<n>="<app>-exporter|metrics|<port>:<port>|disabled|tcp|false|false|false|Metrics Exporter (sidecar, docker-network only)|"` entry next to the app's other port slots. `access=disabled` keeps it off the host. Reference the port from the marker block via `PORT_INTERNAL_DATA_<n>` with the matching `#LIBREPORTAL|PORT_INTERNAL_TAG_<n>|...` annotation — single source of truth, plus the port shows up in the webui port list.
- **Scrape fragment port**: the scrape.yml carries the same `#LIBREPORTAL|<TAG>|<DATA>` annotation as compose lines do — e.g. `- targets: ['<app>-exporter:PORT_INTERNAL_DATA_6'] #LIBREPORTAL|PORT_INTERNAL_TAG_6|PORT_INTERNAL_DATA_6`. During `monitoringRefreshPrometheus`, the copied fragment is run through the framework's `tagsManagerUpdateUniversalTag` once per `PORT_INTERNAL_TAG_<n>` found in the deployed compose, using the compose's resolved value. Idempotent (annotation records the resolved value, second run is a no-op), namespaced per app (each iteration passes that app's own compose), no bespoke sed.
- **Freshness**: `monitoringResolveScrapeTags` reads from the **deployed** compose, which reflects the app's last install/reinstall, not the latest CFG. This is intentional — Prometheus scrapes the running container, which matches the deployed compose. To flush a `CFG_<APP>_PORT_<n>` change through to the scrape config, reinstall that app (which re-resolves its compose and then triggers a refresh at the end of its install). If a refresh ever runs against an unresolved annotation (placeholder text still in the value slot), the helper skips that tag and logs an `isNotice` naming the app + tag, plus an `isError` listing any leftover `PORT_INTERNAL_DATA_<n>` in the generated scrape file — so drift surfaces in the install log instead of as a silent Prometheus scrape failure.
- **Credentials**: pull from the app's existing `CFG_<APP>_*` vars via the `#LIBREPORTAL|<TAG>|<DATA>` substitution mechanism. Do not introduce a new admin/user just for the exporter — reuse the one the app already has.
- **No new CFG vars** unless the exporter genuinely needs something the app config doesn't already expose. If you do add one (e.g. an exporter-specific API token), use the `RANDOMIZEDPASSWORD<N>` convention so it auto-generates.
- **Token sync**: if the scrape config needs a credential value (e.g. a bearer token), put a `<APP>_METRICS_TOKEN_PLACEHOLDER` literal in the scrape fragment and `sed`-replace it from the install script when `monitoringAppEnabled "$app_name"` is true.
- **Shared network namespace** (rare): some exporters need to share the main service's network namespace (e.g. wireguard exporter needs to see the WG interface). Use `network_mode: "service:<app>-service"` inside the marker block. Prometheus still reaches it via `<app>-service:<exporter-port>` since the namespace is shared.

## Install-script hooks

Two calls go into `installXxx()`'s install branch:

```bash
# After dockerComposeSetupFile, before dockerComposeUpdateAndStartApp:
monitoringToggleAppConfig "$app_name" "docker-compose.yml"

# If the scrape fragment has a placeholder that needs the runtime credential:
if monitoringAppEnabled "$app_name"; then
    if [[ -n "$CFG_<APP>_<CRED>" ]]; then
        sudo sed -i "s|<APP>_<TOKEN>_PLACEHOLDER|${CFG_<APP>_<CRED>}|g" \
            "$containers_dir$app_name/resources/monitoring/prometheus-scrape.yml"
    fi
fi

# Near the end of the install, after the app and exporter are running:
monitoringRefreshAll
```

`monitoringRefreshAll` is self-correcting: it gathers from every app with `CFG_<APP>_MONITORING=true`, drops everything else from `scrape.d/` and `dashboards/`, then SIGHUPs Prometheus + restarts Grafana. Safe to call from any app's install regardless of which apps have monitoring on.

## Dashboard

Drop a Grafana dashboard JSON in `resources/monitoring/grafana-dashboards/<app>.json`. The datasource UID is `Prometheus` (provisioned by `monitoringRefreshGrafana`). Keep it minimal — a few stat panels for headline numbers and one timeseries for activity is enough for v1. Users can swap in a community dashboard later via Grafana's import flow.

## Reference implementations

- Native, no auth: `containers/traefik/` (compose-level config block toggle)
- Native, bearer token: `containers/gitea/` (token generated + synced into scrape fragment)
- Native, self-monitoring: `containers/grafana/`, `containers/prometheus/`
- Sidecar, env-only auth: `containers/pihole/`, `containers/adguard/`, `containers/vaultwarden/`, `containers/ollama/`, `containers/gluetun/`
- Sidecar, shared netns: `containers/wireguard/`
- Sidecar, requires app-side config change: `containers/unbound/` (enables remote-control in unbound.conf via a second marker block)