Skip to main content

Webserver

webserver (VMID 114, 10.0.0.14) is the only server that takes web traffic from the internet. The Proxmox host forwards public ports 80 and 443 to it. It terminates TLS for almost every *.abair.ie site and reverse-proxies each one to the right back end.

Playbookansible/hetzner-main/setup-webserver.yml
Site confignetworking/nginx-config/ (one file per hostname)
Make targetssetup-webserver, update-webserver, enroll-wazuh-webserver
SecretsCloudflare DNS Token, Webserver htpasswd admin

What runs on it​

  • nginx, tuned for many long-lived WebSocket connections: worker_processes auto, worker_connections 8192, and a systemd override raising the open-file limit to 65536.
  • Certbot, which issues a single wildcard certificate through Cloudflare's DNS-01 challenge. The certificate covers abair.ie, *.abair.ie, *.geabaire.abair.ie and *.scealai.abair.ie, and lives in /etc/letsencrypt/live/abair.ie/. Every vhost uses it.
  • Real client IPs. On every run, the playbook downloads Cloudflare's current IP ranges and adds them to nginx as trusted proxies, reading the client address from CF-Connecting-IP.
  • JSON access logs in /var/log/nginx/access.log: time, host, method, URI, status, bytes, referrer, user agent, client IP and duration.
  • Grafana Alloy, which ships the access log to Loki and host metrics to Prometheus on Secops. See Monitoring.

How site config is deployed​

networking/nginx-config/ in the repo is the source of truth. On every make setup-webserver or make update-webserver, the playbook:

  1. deletes everything in /etc/nginx/sites-available/ and /etc/nginx/sites-enabled/;
  2. copies every file from networking/nginx-config/ into sites-available/;
  3. enables each one with a symlink;
  4. runs nginx -t, then restarts nginx.

Changes made by hand on the server are wiped on the next run. To add or change a site, edit the file in the repo and run make update-webserver. The Networking and Ingress page lists every hostname and where it goes.

A few vhosts serve static files from /var/www/ (for example screenreader.abair.ie, srv.abair.ie and synthesis.abair.ie/audio). Those files are placed by hand and aren't in the repo.

Firewall​

UFW allows 22, 80 and 443 from anywhere and denies everything else.

Resizing this VM​

The VM has no CPU or memory hotplug, so a resize needs a full stop and start. Do it on the Proxmox host, not with terraform apply:

qm shutdown 114
qm set 114 --cores <N> --memory <MB>
qm start 114

Then update terraform/hetzner-main/webserver.tf to the same values, so Terraform doesn't see a difference.

Changing the VM config regenerates its cloud-init drive. On the next boot cloud-init creates new SSH host keys, so your next SSH or Ansible run fails with REMOTE HOST IDENTIFICATION HAS CHANGED. Clear the old keys:

ssh-keygen -R 10.0.0.14
ssh-keygen -R webserver-vm

This applies to any hetzner-main VM, not just the webserver.

Notes​

  • The playbook also creates /etc/nginx/.htpasswd for a user admin, but no vhost uses it at the moment. The same password (Webserver htpasswd admin) is what the fleet's Alloy agents use to push to the monitoring stack.
  • Certbot issues the certificate only once. After that it relies on the certbot package's own renewal timer. nginx picks up a renewed certificate the next time it restarts, which happens on every update-webserver run.