Common Tasks for Admins

This page gives examples of what admins are typically asked to do. For how changes flow through the repo in general, see Making Changes. For the full list of targets, see the Makefile Reference. For the developer side, see Common Tasks for Developers.

Applying a developer's pull request

A developer opens a pull request on the infrastructure-automation repo, for example to add an nginx vhost or update their app's compose file.

Turnaround

Developers are usually waiting on these changes. Review, merge and apply them within a few hours.

Review the pull request on Gitea and merge it into main.

Pull the latest main on your machine:

cd infrastructure-automation
git checkout main
git pull

Ask Claude which make targets need to run to apply the changes, for example:

What make targets do I need to run to apply the changes from the latest merged PR?

Claude will look at the diff and advise.
Unlock the vault (eval "$(make unlock-bw)") and run the targets.

The most common cases are below.

An nginx config file has been added or updated

The PR adds or changes a file in networking/nginx-config/. Run:

make update-webserver

If it's a new hostname, also add the DNS record in Cloudflare.

A developer updates an app's Docker Compose file or `.env` file

Any of these changes needs the app redeployed:

The compose file changes. The PR changes ansible/hetzner-main/services-user-apps/<app>/docker-compose.yml or ansible/hetzner-main/services-staff-apps/<app>/docker-compose.yml.
An existing .env file is updated in Vaultwarden. The developer edits the <app> .env Secure Note directly, so there's no PR. They should tell you when they've done it.
A new .env file is added. The developer creates an <app> .env Secure Note in their project's collection, and opens a PR that adds the app to user_apps_with_env (or staff_apps_with_env) in the playbook and adds env_file: .env to the compose file. See Adding an App → Add secrets.

Run the target for the VM the app lives on:

make deploy-services-user-apps    # app is on services-user-vm
make deploy-services-staff-apps   # app is on services-staff-vm

The directory the compose file is in tells you which one. The deploy fetches every app's .env note from Vaultwarden, so you must be a member of the app's collection. If the note is missing, empty, or you can't access it, the deploy stops.

Increasing a VM's CPU, RAM or disk on hetzner-main

Use this when a VM on hetzner-main needs more CPU cores, RAM or disk space. The VMs and their VMIDs are listed in the Server List.

Terraform could make the change, but do it on the Proxmox host with qm instead, then update the Terraform file to match:

It's quicker and you control the restart. CPU and RAM changes only take effect after a full shutdown and start (the VMs don't support hot-adding them). With qm you choose exactly when that happens.
It's safer. A terraform apply can change other things at the same time, such as the cloud-init drive.
The Terraform file must still be updated. If we ever rebuild from Terraform, or someone runs make provision, Terraform puts the VM back to the values in its .tf file. For disks, that apply may even fail.

Watch the VM's resources in Corais before, during and after the change

Open Corais (Grafana) and look at the VM's dashboard (for example services-user or services-staff):

Before: check CPU, RAM and disk usage over the last few days, so you know the change is needed and how much to add.
During: keep the dashboard open while the VM restarts, and check every app on it comes back up.
For a couple of days after: check again that usage has settled where you expected, and that nothing new is running out.

Also check the host itself still has room (step 2 below). Every VM shares hetzner-main's CPU, RAM and disk, so giving one VM more leaves less for the others.

1. Before you start

Agree a time with the team, and post in Rocket.Chat. CPU and RAM changes restart the VM, so every app on it is down for a minute or two.
Find the VM's VMID in the Server List. The examples below use services-user, VMID 118. Replace it with yours.

2. SSH to the Proxmox host and check the current values

You log in as ansible-control, so every qm command needs sudo.

ssh proxmox-host

Check what the VM has now:

sudo qm list                                          # all VMs, their status, RAM and boot disk size
sudo qm config 118 | grep -E '^(cores|sockets|memory|scsi0):'

Check the host has room for what you're about to add:

nproc              # CPU threads on the host
free -h            # host RAM. Look at "available"
df -h /var/lib/vz  # host disk. VM disks are on the "local" storage, which lives here

Add up the RAM in qm list for the running VMs, plus what you're adding, and make sure it stays well under the host's total. Leave a few GB for Proxmox itself. CPU cores can be shared between VMs, so going over nproc in total is fine within reason. RAM and disk can't.

3a. Adding CPU cores

This needs a restart, so the VM's apps are down while it's stopped.

sudo qm shutdown 118 --timeout 180    # clean shutdown through the guest; waits up to 3 minutes
sudo qm status 118                    # must say "status: stopped" before you continue
sudo qm set 118 --cores 6             # the NEW total, not the number to add
sudo qm start 118

If qm shutdown times out, run it again. Only use sudo qm stop 118 (the same as pulling the plug) if the VM really won't shut down.

Go to step 4.

3b. Adding RAM

This also needs a restart. --memory is in MB and is the new total: 8 GB = 8192, 12 GB = 12288, 16 GB = 16384.

sudo qm shutdown 118 --timeout 180
sudo qm status 118                    # must say "status: stopped"
sudo qm set 118 --memory 12288
sudo qm start 118

You can change cores and RAM in one go: sudo qm set 118 --cores 6 --memory 12288.

Go to step 4.

3c. Adding disk space

This doesn't need a restart. It's done in two parts: make the virtual disk bigger on the host, then grow the partition and filesystem inside the VM.

Disks can only grow

You can't shrink a disk with qm resize. Double-check the number before you press Enter. Unlike --cores and --memory, the + means add this much: +32G adds 32 GB.

On the Proxmox host:

sudo qm resize 118 scsi0 +32G
sudo qm config 118 | grep '^scsi0:'   # size= should show the new total
exit

Then SSH into the VM itself (the SSH aliases are in the SSH config from Getting Started):

ssh services-user-vm
lsblk                                  # sda is now bigger, but sda1 (mounted on /) isn't yet
sudo growpart /dev/sda 1               # grow partition 1 to fill the disk
sudo resize2fs /dev/sda1               # grow the filesystem to fill the partition
df -h /                                # "Size" should now show the extra space

If lsblk shows the root filesystem on a different device or partition number, use those instead of /dev/sda and 1.

4. Check the VM after the change

Check Proxmox shows the VM running with the new values:

ssh proxmox-host
sudo qm status 118
sudo qm config 118 | grep -E '^(cores|memory|scsi0):'
exit

SSH into the VM and check it sees the new resources:
```
ssh services-user-vm
nproc       # number of CPU cores
free -h     # total RAM
df -h /     # disk size and free space
```
If SSH fails with "REMOTE HOST IDENTIFICATION HAS CHANGED", the restart made cloud-init regenerate the VM's SSH host keys. This is expected, not an attack. Remove the old keys, then connect again and accept the new ones:
```
ssh-keygen -R 10.0.0.18 && ssh-keygen -R services-user-vm
```
Check all the containers came back up (only after a restart):
```
docker ps --format 'table {{.Names}}\t{{.Status}}'
```
Every container should say Up. Anything missing or Restarting needs looking at.
Check status.abair.ie is all green and the VM's dashboard in Corais shows the new totals.

5. Update the Terraform file to match

In infrastructure-automation, open the VM's file in terraform/hetzner-main/ (for example services-user.tf) and change the value:

What you changed	Field in the `.tf` file
CPU cores	`cores` inside the `cpu { ... }` block
RAM	`memory` (in MB, the same number you passed to `qm set`)
Disk	`size` inside `disks { scsi { scsi0 { disk { ... } } } }` (the new total in GB, without a unit)

Add a comment above the value saying the date, the old and new values, why it was needed, and that it was done with qm on the host. For example:

  # Bumped 8G -> 12G on 2026-10-07: RAM was regularly above 90% in Corais after
  # adding <app>. Done cold on the Proxmox host (qm set 118 --memory 12288 + restart),
  # this file updated to match so terraform apply shows no drift.
  memory = 12288

Don't run make provision or terraform apply for this. The change is already live. The file is only being brought into line with it.

Also update the VM's row in the Server List in these docs. Commit both changes and push.

scealai

scealai (VMID 106) isn't managed by Terraform. Steps 1–4 work the same way, but there's no .tf file to update in step 5.

Applying a developer's pull request​

An nginx config file has been added or updated​

A developer updates an app's Docker Compose file or .env file​

Increasing a VM's CPU, RAM or disk on hetzner-main​

1. Before you start​

2. SSH to the Proxmox host and check the current values​

3a. Adding CPU cores​

3b. Adding RAM​

3c. Adding disk space​

4. Check the VM after the change​

5. Update the Terraform file to match​