Build a Proxmox VE Cluster, Quorum and QDevice
Create a Proxmox VE cluster, understand quorum, secure a 2-node cluster with a QDevice, migrate VMs and know the network and hardware limits.
On this page
This guide explains how to join several Proxmox VE servers into a cluster, how quorum works, how to make a 2-node cluster more reliable with a QDevice, and which limits to check before you start.
A cluster lets you manage all the nodes from a single interface, share the configuration and migrate VMs between servers. It does not replace a backup, and on its own it is not a high availability solution.
Check That a Cluster Is Suitable
The deciding factor is the network between your servers.
| Requirement | Details |
|---|---|
| Latency | Less than 5 ms between all nodes for stable operation |
| Stability | Corosync is sensitive to latency variations: avoid a link loaded by storage or backups |
| Ports | UDP 5405-5412 (Corosync), TCP 22 (SSH), TCP 8006 (API), TCP 60000-60050 (migration) |
| Clock | Time synchronized on all nodes |
| Versions | Same Proxmox VE version on all nodes |
| Identity | Final hostname and IP address: they can no longer be changed once the cluster is created |
| New node | No VMs or containers on the server joining the cluster |
What a Cluster Does Not Provide Automatically
- No automatic VM failover without configuring high availability (HA).
- HA requires at least 3 nodes, shared storage and hardware redundancy.
- Local disks do not follow on their own: a VM on a local disk is copied during migration, and it is no longer accessible if its node fails.
- Private networks and NAT remain local:
vmbr1must exist on each node, and port forwards stay tied to the public address of the original server. - Live migration is only guaranteed between processors from the same manufacturer.
Prerequisites
- At least 2 servers with Proxmox VE installed, for example with the Install and Configure Proxmox VE guide
rootSSH access and the KVM IP of each server- A known
rootpassword for the existing node - A recent backup of your VMs
The examples use a network dedicated to the cluster:
| Node | Name | Cluster network address |
|---|---|---|
| 1 | pve1 |
10.30.30.11 |
| 2 | pve2 |
10.30.30.12 |
| 3 (optional) | pve3 |
10.30.30.13 |
| QDevice (2-node cluster) | arbitre |
10.30.30.20 |
Understand Quorum
Each node has one vote. To change the cluster state, a majority of votes is required: this is the quorum. It prevents two isolated parts from each changing the configuration on their own side (split-brain).
| Nodes | Votes required | Failures tolerated |
|---|---|---|
| 2 | 2 | 0 |
| 2 + QDevice | 2 of 3 | 1 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Without quorum, the cluster goes read-only: /etc/pve can no longer be modified, and you can no longer start VMs or change their configuration. VMs that are already running keep running as long as HA is not in use. With HA, an isolated node may force itself to stop (fencing) to protect the data.
Step 1: Prepare Each Node
On all nodes, update the system, then check the following:
apt update
apt full-upgrade
pveversion
hostname
hostname --ip-address
chronyc trackingOn the node or nodes that will join the cluster, check that there are no VMs or containers:
qm list
pct listIf a node already contains machines, back them up, delete them, then restore them after joining under a free ID. Joining replaces the /etc/pve configuration, including the storage configuration.
Test the cluster network from each node to the others:
ping -c 20 10.30.30.12The avg column of the summary must stay below 5 ms, with no packet loss.
Adapt the Firewall
If you set local_network to the server's address, as suggested in the installation guide, replace this value with the cluster network in /etc/pve/firewall/cluster.fw, on each node, before creating the cluster:
[ALIASES]
local_network 10.30.30.0/24Then check:
pve-firewall localnetThe Proxmox firewall automatically generates the Corosync rules for this network and adds it to the management hosts. Once the cluster is created, the same command lists the nodes accepted for Corosync traffic: check that each one appears. If you use ALLOW_FROM in /etc/default/pveproxy, add the node addresses there too.
Step 2: Create the Cluster
On pve1 only. The cluster name cannot be changed afterwards.
pvecm create my-cluster --link0 10.30.30.11
pvecm statusIn the web interface, the equivalent is in Datacenter → Cluster → Create Cluster.
Step 3: Add the Other Nodes
Connect over SSH to the node being added, here pve2:
pvecm add 10.30.30.11 --link0 10.30.30.12The command asks for the root password of pve1 and has you confirm its certificate fingerprint. Compare it with the one shown in pve1 → System → Certificates.
In the web interface, use Datacenter → Cluster → Join Information on pve1, then Join Cluster on pve2. After joining, the node's certificate changes: reload the page and log in again.
Add the nodes one at a time, checking the status between each addition.
Step 4: Verify the Cluster
pvecm status
pvecm nodes
systemctl status corosync pve-cluster --no-pagerIn pvecm status, check these lines:
| Line | Expected value |
|---|---|
Quorate |
Yes |
Nodes |
Planned number of nodes |
Expected votes |
Total number of votes |
Flags |
Quorate |
Create a test file on one node and read it on another: /etc/pve is replicated in real time.
echo test > /etc/pve/test-cluster.txtcat /etc/pve/test-cluster.txt
rm /etc/pve/test-cluster.txtStep 5: Add a QDevice to a 2-Node Cluster
A QDevice provides a third vote from an external machine. It does not run VMs and does not need the low latency required by Corosync. Choose a machine independent of the two nodes, running Debian: a small server will do, provided it is not hosted on either of the two nodes.
| Number of nodes | Proxmox recommendation |
|---|---|
| 2 | QDevice recommended |
| Other even number | QDevice possible |
| Odd number | Not recommended: it becomes almost a single point of failure |
Install the Packages
On the external machine:
apt update
apt install corosync-qnetdOn each node of the cluster:
apt install corosync-qdeviceOpen Access
The external machine must accept, from the nodes only:
- TCP 5403 for the
corosync-qnetdservice - SSH for
root, for the duration of the configuration
With UFW on the external machine, repeating for each node:
ufw allow from 10.30.30.11 to any port 5403 proto tcp
ufw allow from 10.30.30.11 to any port 22 proto tcpThe next command copies the cluster's SSH key to the external machine. Set up key-based root access, or temporarily allow root to log in with a password.
Enable the QDevice
Check that all nodes are online, then run this on one node only:
pvecm qdevice setup 10.30.30.20
pvecm statusThe result should look like this:
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate Qdevice
Membership information
Nodeid Votes Qdevice Name
0x00000001 1 A,V,NMW 10.30.30.11 (local)
0x00000002 1 A,V,NMW 10.30.30.12
0x00000000 1 Qdevice| Indicator | Meaning |
|---|---|
A / NA |
Communication with corosync-qnetd active or not |
V / NV |
The QDevice grants its vote to the node or not |
NMW |
Default setting (Not Master Wins) |
Then restore the SSH restriction on the external machine if you had relaxed it.
Step 6: Test a Controlled Failure
Run this test outside production, after making your backups. In a cluster with a QDevice or 3 nodes, stop a node with no important VMs, then observe from the other:
pvecm statusQuorate must stay at Yes. Then restart the node and check that it comes back in pvecm nodes.
Migrate a VM
A stopped or running VM can be moved between nodes:
qm migrate 100 pve2
qm migrate 100 pve2 --online
qm migrate 100 pve2 --online --with-local-disksThe last command also copies local disks, which can take a long time. Before migrating, check that:
- the same bridge (
vmbr1, for example) exists on the target node; - the target storage has enough space;
- the VM's processor type is compatible,
x86-64-v2-AESbeing preferable tohostbetween different machines.
Migration traffic is encrypted by default. Keep this mode if the network is not entirely under your control. If the VM uses a node's NAT, its port forwards do not follow it and its active connections will be interrupted.
High Availability and Replication
HA automatically restarts a VM on another node after a failure. Proxmox requires at least 3 nodes, shared storage and redundant hardware. Detection and restart take about 2 minutes.
Without shared storage, ZFS replication regularly copies disks to another node. It is asynchronous: in the event of a failure, writes made since the last replication are lost.
Updates
Update one node at a time:
- Check that the cluster is healthy with
pvecm status. - Migrate the VMs off the node concerned, or shut them down cleanly.
- Run
apt update, thenapt full-upgrade. - Restart if necessary and wait for the node to come back in
pvecm nodes. - Check the quorum, then move on to the next node.
Do not leave the nodes on different versions longer than necessary. For a major version, follow the Proxmox upgrade guide.
Backups
Replication of /etc/pve is not a backup: a mistake or a deletion is immediately propagated to all nodes. Configure backups in Datacenter → Backup, to external storage.
The HostMyServers Proxmox Backup Server offering is added at the datacenter level and becomes usable by the nodes. See Connect PBS to Proxmox VE.
Remove a Node
-
Migrate all its VMs and containers, then back up its local data.
-
Delete its replication jobs. Remove the QDevice if there is one.
-
Note the node's name with
pvecm nodes, then shut it down. -
From another node:
Terminalpvecm delnode pve3 pvecm nodes
The error could not kill node (error = cs_err_not_exist) can be ignored. The old files remain in /etc/pve/nodes/pve3: retrieve what you need, then delete this folder. To bring the server back into the cluster, reinstall Proxmox VE before adding it again.
Troubleshooting
this host already contains virtual guests
The node being added contains VMs or containers. Back them up, delete them, join the cluster, then restore them under a free ID.
corosync is already running, is this node already in a cluster?!
The server already belongs to a cluster, or a previous attempt left traces behind. Check with pvecm status. The cleanest solution is to reinstall the node.
cluster not ready - no quorum?
The node has no quorum, and /etc/pve is read-only.
pvecm status
systemctl status corosync --no-pager
journalctl -b -u corosync --no-pager | tail -n 50First restore the network or the missing node. As a last resort, and only if you are certain the other side is actually stopped:
pvecm expected 1Corosync Links Go Down Regularly
Messages reporting a down link or Retransmit entries in journalctl -u corosync indicate an unstable or saturated network. Measure the latency and separate Corosync from storage and backup traffic. You can also add a second link with --link1.
Host key verification failed
Update the cluster's certificates and SSH fingerprints:
pvecm updatecertsThe QDevice Shows NA
The node cannot reach corosync-qnetd. On the external machine, check the service and port TCP 5403:
systemctl status corosync-qnetd --no-pager
ss -tln | grep 5403Then test access from a node:
timeout 3 bash -c '</dev/tcp/10.30.30.20/5403' && echo "port reachable"Migration Fails
Read the task log in the web interface. Common causes are a bridge missing on the target node, unavailable storage, an ISO still mounted from local storage or an incompatible host processor type. Remove the ISO with qm set 100 --ide2 none,media=cdrom.