Amergin (GPU Server)

Amergin is a Hetzner dedicated server with a GPU. It runs speech recognition, the Fotheidil subtitling back end, speech synthesis and predictive text. It isn't a Proxmox VM: it's reached over a private Hetzner vSwitch at 192.168.100.2.

Public address	5.9.9.57 (SSH only, alias amergin, user ansible-control)
vSwitch address	192.168.100.2 (all services listen here)
Hardware	20 vCPU, 62 GB RAM, one NVIDIA RTX 4000 SFF Ada with 20.4 GB VRAM (no MIG)
Playbooks	ansible/amergin/
Further notes in the repo	ansible/amergin/VSWITCH.md, RESOURCES.md, PREDICTIVE-TEXT.md

Amergin has no public web server of its own. Public traffic arrives through the webserver or the API gateway, over the vSwitch.

Services

Each service has its own playbook and Make targets. Every service is built from source on Amergin with docker build; none of them uses the registry. Every container restarts automatically.

Port	Service	Playbook	GPU	Reached by
8080	triton_asr_streaming: streaming speech recognition (NVIDIA Triton)	setup-streaming-recognition.yml	yes	webserver (streaming-asr.abair.ie) and API gateway
4040	fotheidil-api: Fotheidil subtitling API. Handles uploads, extracts audio with ffmpeg, and calls asr-diarisation and Supabase.	setup-fotheidil-api.yml	no	webserver (fotapi.abair.ie), API gateway, and the Fotheidil front end on services-user
8000	asr-diarisation: NeMo speech recognition plus pyannote speaker diarisation, for Fotheidil	setup-fotheidil-recognition.yml	yes	API gateway and fotheidil-api
8002	batch-asr: batch speech recognition for short clips (max 30 s, 5 MB)	setup-batch-recognition.yml	yes	webserver (recognition.abair.ie/v3-5/) and API gateway
8003	predictive-text: Irish autocomplete (FastAPI and KenLM)	setup-predictive-text.yml	no	API gateway only
8004	synthesis: Irish speech synthesis (Piper voices plus an LTS server)	setup-synthesis.yml	no	API gateway only (api.abair.ie/v4/synthesis)

There are also two capitalisation-and-punctuation containers. Neither publishes a port; they're reached only on their stack's Docker network:

• marian-cpr (GPU) sits behind asr-diarisation, on the fotheidil-net network.
• batch-marian-cpr (CPU) sits behind batch-asr, on the batch-recognition-net network.

The batch and Fotheidil stacks use the same models but are kept separate on purpose. The Marian server handles requests strictly in order, so long Fotheidil jobs would otherwise hold up short batch requests.

Every port is bound to 192.168.100.2 only, so none of them is reachable on the public address. UFW rules on each port list the allowed callers (10.0.0.14 for the webserver, 10.0.0.17 for the API gateway) as a second layer.

Where the code and models come from

Everything comes from Gitea, using the read-only bot account's token (Gitea Automation Bot Token). Nothing is copied from an admin's machine.

Service	Source code (Gitea, main)	Models (Gitea)
streaming	ABAIR/triton_asr_streaming	ABAIR/streaming-asr-model
fotheidil-api	ABAIR/fotheidil-api	none
fotheidil-recognition	ABAIR/fotheidil-recognition	ABAIR/fotheidil-asr, ABAIR/fotheidil-cpr-marian. Pyannote's diarisation models still come from Hugging Face, with Huggingface Token Amergin: they're gated third-party models, not ours to redistribute.
batch-recognition	ABAIR/batch-recognition	ABAIR/fotheidil-asr, ABAIR/fotheidil-cpr-marian (the same models as Fotheidil)
predictive-text	ABAIR/predictive-text	ABAIR/predictive-text-model
synthesis	ABAIR/synthesis-production-server, plus ABAIR/ltsserver-rest cloned inside it	ABAIR/piper-irish-voices, cloned with Git LFS inside the source checkout

The model repos store their files with Git LFS. Except for synthesis, each container downloads its model files when it starts, into a cache directory on Amergin (for example /opt/batch_recognition_models/). It skips any file that's already there, so a restart doesn't download them again. The download uses Gitea's /media/ URLs, because /raw/ returns the small LFS pointer file instead of the model.

To deploy a code change, merge it into main on Gitea and run the service's make update-… target. It rebuilds and restarts only the services whose repo has changed.

To deploy a new model, push it to the model repo on Gitea, then:

• synthesis: run make update-synthesis. The new commit in piper-irish-voices triggers a rebuild on its own. The playbook stops before building if any voice arrives as an LFS pointer instead of the real file (its LFS object is missing on Gitea).
• predictive-text: run make update-predictive-text-model.
• the other services: delete the cached file on Amergin, then restart the container, for example ssh amergin 'sudo rm /opt/triton_asr_models/streaming_model.nemo && sudo docker restart triton_asr_streaming'. Running make update-… alone isn't enough: if the source hasn't changed, it doesn't restart anything, and the container keeps using the cached copy.

fotheidil-api keeps its uploads and processed media in /opt/fotheidil-api-data, outside the source checkout, so a fresh clone never touches them.

Tuning notes

• asr-diarisation runs with segmentation and embedding batch sizes of 32 and 8. A test at 64 and 16 caused three GPU faults (Xid 31) within 24 hours on 2026-09-03 and was reverted. Don't raise them without a fresh test in isolation.
• batch-asr processes at most 2 jobs at a time (MAX_CONCURRENT_JOBS=2). asr-diarisation processes 1. Raising it to 2 is a separate experiment that hasn't been run yet, and should be tested on its own. Both apps hold a lock around each GPU model call, because NeMo's decoder breaks if two threads use it at once on a freshly started container.
• make update-predictive-text-model reloads only the predictive-text model. It deletes the cached model files and restarts the container, with no rebuild. Use it when the model changes on Gitea but the code hasn't.

Host setup

make setup-amergin-host runs setup-amergin-host.yml:

• the NVIDIA driver and CUDA from Debian's non-free packages (versions aren't pinned), and the NVIDIA Container Toolkit registered with Docker;
• GPU persistence mode, so the first CUDA call after an idle period doesn't pay a multi-second start-up cost;
• the vSwitch interface enp4s0.4000 (192.168.100.2/24, MTU 1400), with a route to 10.0.0.0/24 through 192.168.100.1;
• UFW: SSH from anywhere, and the per-service rules added by each service's playbook;
• monitoring: Alloy (container logs, host metrics, GPU metrics and asr-diarisation metrics) and DCGM exporter (GPU metrics, port 9400);
• docker-buildx-plugin, so docker build uses BuildKit. Without it, Docker falls back to its old builder, which takes about a minute per Dockerfile step on the large NVIDIA base images;
• a daily Docker clean-up (03:00, root's crontab) that removes stopped containers, untagged images and build cache older than 7 days. It deliberately doesn't use docker image prune -a, because that would also delete the NVIDIA base images (NeMo, Triton, CUDA) and the next build would download them again;
• gpu-watchdog, a small container that checks /health on asr-diarisation, batch-asr and triton_asr_streaming every 60 seconds. After 3 failures in a row it restarts the container, then waits 5 minutes before acting on it again.

make update-amergin-host re-applies persistence mode, UFW, the daily Docker clean-up, Alloy, DCGM and the watchdog. It skips the driver and Docker installation. It restarts the Alloy and DCGM containers (a few seconds' gap in logs and GPU metrics), but none of the services.

caution

make setup-amergin-host runs every tag, including wazuh-agent, but doesn't pass the Wazuh password, so it fails at the last step. Everything before that step has already been applied. Enroll Wazuh separately with make enroll-wazuh-amergin.

No playbook turns UFW on or sets its default policy; the playbooks assume it's already active. The DCGM exporter publishes port 9400 on all interfaces, and Docker-published ports bypass UFW.

GPU testing sandbox

ssh testing@amergin (or the amergin-testing alias) opens a throwaway GPU sandbox. Admins use it to try out a candidate model without disturbing production.

• Who can use it: admins only. make setup-testing-user reads the admin list, and their SSH keys, from the Vaultwarden Secure Note users.json. Re-run it to add or remove someone.
• What you get: a shell in a container from the testing-sandbox image (CUDA 12.4 runtime, Python, git). It's limited to CPU cores 16–19, 8 GB of RAM and about 4 GB of VRAM. The VRAM limit only applies to PyTorch; other CUDA code can use more.
• Shared session: everyone who connects joins the same container.
• What's kept: only /workspace, which is also reachable over SFTP. Everything else is deleted when the last person disconnects, and /workspace itself is wiped when the next session starts.
• Safety: the testing user isn't in the docker group. It can only run one fixed, root-owned script through sudo.

Removed services

qomhra-llm (the Qomhrá LLM) was removed completely on 2026-09-01, to free GPU memory. qomhra.abair.ie no longer exists.