Skip to main content

Observability

observability (VMID 115, 10.0.0.15) is the centralised monitoring stack for the whole fleet. Every other Docker-hosting VM (and Amergin) ships its logs and metrics here.

Configured by ansible/hetzner-main/setup-observability.yml.

What's running​

  • Loki (:3100) — log storage, receiving pushes from Grafana Alloy on every other server.
  • Prometheus (:9090) — metrics storage, --web.enable-remote-write-receiver so other hosts can push metrics rather than being scraped.
  • Grafana (:80) — dashboards, provisioned automatically on every Ansible run from the JSON files in grafana-dashboards/ in the automation repo. Dashboard permissions are set per-dashboard (e.g. the services-user dashboard is viewable by anyone, the API gateway dashboard is admin-only).
  • Uptime Kuma (:3011) — public status page, exposed at status.abair.ie via the webserver reverse proxy.

The fleet-wide shipping pattern​

Every other Docker VM runs a standalone Grafana Alloy container (not part of any app's Compose stack) that:

  • Tails logs — either a specific file (nginx access log on webserver, Docker's own log files on synthesis-test) or via Docker socket discovery (services-user, services-staff, api-gateway) — and pushes them to Loki here.
  • Scrapes host and application metrics and remote-writes them to Prometheus here.

Amergin follows the same pattern, additionally shipping GPU metrics (via a dcgm-exporter container). Puca ships host metrics and full journald logs the same way, but to a separate, publicly-reachable observability endpoint (corais.abair.ie) rather than this VM — see Puca.

Firewall​

Only the specific hosts that need to push data are allowed to reach the Loki/Prometheus ports; Grafana (80) and Uptime Kuma (3011) are reachable only via the webserver VM.