LibrePortal/scripts/network/monitoring/HOWTO_SIDECAR.md

Adding monitoring for an app

There are two shapes. Pick the one that matches the app:

1. Native — the app speaks Prometheus on a /metrics endpoint. Just point Prometheus at it.
2. Sidecar — the app has no native /metrics. Run an exporter container alongside the app; Prometheus scrapes the exporter.

Both use the same plumbing (monitoringToggleAppConfig + monitoringRefreshAll) and the same on-disk layout under containers/<app>/resources/monitoring/. The only thing that differs is what's inside the marker block in the app's compose.

On-disk layout

containers/<app>/
├── <app>.config                         # CFG_<APP>_MONITORING=false (default off)
├── <app>.sh                             # calls the two monitoring helpers
├── docker-compose.yml                   # marker block: env vars OR sidecar service
└── resources/
    └── monitoring/
        ├── prometheus-scrape.yml        # one or more scrape jobs
        └── grafana-dashboards/
            └── <app>.json               # dashboard(s) to provision

When CFG_<APP>_MONITORING=true, monitoringRefreshAll (in scripts/network/monitoring/monitoring.sh) copies the scrape fragment into prometheus/scrape.d/<app>.yml and the dashboard(s) into grafana/provisioning/dashboards/libreportal/, then SIGHUPs Prometheus and restarts Grafana.

The marker block

Anywhere in the compose you put a block like this:

# >>> libreportal-monitoring >>>
#  …lines that should be uncommented when MONITORING=true…
# <<< libreportal-monitoring <<<

monitoringToggleAppConfig <app> <relpath> strips one leading # from every non-marker line in the range when monitoring is enabled, and re-adds it when disabled. The marker lines themselves are never touched.

Native pattern

For apps with built-in /metrics (e.g. traefik, gitea, headscale, grafana):

• If the metrics endpoint is gated behind config (e.g. gitea needs GITEA__metrics__ENABLED=true), put the env vars / config lines inside a marker block in the compose or app config file.
• Scrape fragment targets the app service directly: targets: ['<app>-service:<port>'].
• If the endpoint requires a bearer token, generate it via the CFG_<APP>_METRICS_TOKEN=RANDOMIZEDPASSWORD<N> mechanism, mirror it into the scrape fragment in <app>.sh with a sed substitution against a <APP>_METRICS_TOKEN_PLACEHOLDER (see gitea.sh for the pattern).

Sidecar pattern

For apps without native /metrics (pihole, adguard, wireguard, nextcloud, unbound, gluetun, vaultwarden, ollama):

Add the exporter as a sibling service inside the marker block:

services:
  <app>-service:
    …

  # >>> libreportal-monitoring >>>
  #<app>-exporter:
  #  container_name: <app>-exporter
  #  image: <vendor>/<app>-exporter:<tag>
  #  restart: unless-stopped
  #  environment:
  #    - …pull credentials from existing CFG_<APP>_* via #LIBREPORTAL tags…
  #  networks:
  #    DOCKER_NETWORK_DATA:
  # <<< libreportal-monitoring <<<

Rules:

• Service name: <app>-exporter. Prometheus scrapes <app>-exporter:<port>. No host port mapping — the metrics endpoint stays on the docker network only.
• Network: attach to the same DOCKER_NETWORK_DATA as the main service. No explicit IP needed; service-name DNS works.
• Port: add a CFG_<APP>_PORT_<n>="<app>-exporter|metrics|<port>:<port>|disabled|tcp|false|false|false|Metrics Exporter (sidecar, docker-network only)|" entry next to the app's other port slots. access=disabled keeps it off the host. Reference the port from the marker block via PORT_INTERNAL_DATA_<n> with the matching #LIBREPORTAL|PORT_INTERNAL_TAG_<n>|... annotation — single source of truth, plus the port shows up in the webui port list.
• Scrape fragment port: the scrape.yml carries the same #LIBREPORTAL|<TAG>|<DATA> annotation as compose lines do — e.g. - targets: ['<app>-exporter:PORT_INTERNAL_DATA_6'] #LIBREPORTAL|PORT_INTERNAL_TAG_6|PORT_INTERNAL_DATA_6. During monitoringRefreshPrometheus, the copied fragment is run through the framework's tagsManagerUpdateUniversalTag once per PORT_INTERNAL_TAG_<n> found in the deployed compose, using the compose's resolved value. Idempotent (annotation records the resolved value, second run is a no-op), namespaced per app (each iteration passes that app's own compose), no bespoke sed.
• Freshness: monitoringResolveScrapeTags reads from the deployed compose, which reflects the app's last install/reinstall, not the latest CFG. This is intentional — Prometheus scrapes the running container, which matches the deployed compose. To flush a CFG_<APP>_PORT_<n> change through to the scrape config, reinstall that app (which re-resolves its compose and then triggers a refresh at the end of its install). If a refresh ever runs against an unresolved annotation (placeholder text still in the value slot), the helper skips that tag and logs an isNotice naming the app + tag, plus an isError listing any leftover PORT_INTERNAL_DATA_<n> in the generated scrape file — so drift surfaces in the install log instead of as a silent Prometheus scrape failure.
• Credentials: pull from the app's existing CFG_<APP>_* vars via the #LIBREPORTAL|<TAG>|<DATA> substitution mechanism. Do not introduce a new admin/user just for the exporter — reuse the one the app already has.
• No new CFG vars unless the exporter genuinely needs something the app config doesn't already expose. If you do add one (e.g. an exporter-specific API token), use the RANDOMIZEDPASSWORD<N> convention so it auto-generates.
• Token sync: if the scrape config needs a credential value (e.g. a bearer token), put a <APP>_METRICS_TOKEN_PLACEHOLDER literal in the scrape fragment and sed-replace it from the install script when monitoringAppEnabled "$app_name" is true.
• Shared network namespace (rare): some exporters need to share the main service's network namespace (e.g. wireguard exporter needs to see the WG interface). Use network_mode: "service:<app>-service" inside the marker block. Prometheus still reaches it via <app>-service:<exporter-port> since the namespace is shared.

Install-script hooks

Two calls go into installXxx()'s install branch:

# After dockerComposeSetupFile, before dockerComposeUpdateAndStartApp:
monitoringToggleAppConfig "$app_name" "docker-compose.yml"

# If the scrape fragment has a placeholder that needs the runtime credential:
if monitoringAppEnabled "$app_name"; then
    if [[ -n "$CFG_<APP>_<CRED>" ]]; then
        sudo sed -i "s|<APP>_<TOKEN>_PLACEHOLDER|${CFG_<APP>_<CRED>}|g" \
            "$containers_dir$app_name/resources/monitoring/prometheus-scrape.yml"
    fi
fi

# Near the end of the install, after the app and exporter are running:
monitoringRefreshAll

monitoringRefreshAll is self-correcting: it gathers from every app with CFG_<APP>_MONITORING=true, drops everything else from scrape.d/ and dashboards/, then SIGHUPs Prometheus + restarts Grafana. Safe to call from any app's install regardless of which apps have monitoring on.

Dashboard

Drop a Grafana dashboard JSON in resources/monitoring/grafana-dashboards/<app>.json. The datasource UID is Prometheus (provisioned by monitoringRefreshGrafana). Keep it minimal — a few stat panels for headline numbers and one timeseries for activity is enough for v1. Users can swap in a community dashboard later via Grafana's import flow.

Reference implementations

• Native, no auth: containers/traefik/ (compose-level config block toggle)
• Native, bearer token: containers/gitea/ (token generated + synced into scrape fragment)
• Native, self-monitoring: containers/grafana/, containers/prometheus/
• Sidecar, env-only auth: containers/pihole/, containers/adguard/, containers/vaultwarden/, containers/ollama/, containers/gluetun/
• Sidecar, shared netns: containers/wireguard/
• Sidecar, requires app-side config change: containers/unbound/ (enables remote-control in unbound.conf via a second marker block)