Proxmox HA Cluster — Shared iSCSI Storage & Automated VM Failover
Project Type
Homelab infrastructure project — nested 3-node Proxmox VE cluster running inside VMware ESXi, with TrueNAS providing ZFS-backed iSCSI shared storage, full network traffic separation across four segments, Windows Server 2022 VM deployment on shared storage, and validated Proxmox HA automated failover.
Project Overview
This project demonstrates a full software-defined storage and high availability stack built entirely inside a VMware ESXi homelab. Rather than deploying on bare metal, all Proxmox nodes and the TrueNAS storage server run as nested virtual machines on a single ESXi host — making it possible to simulate a real multi-node enterprise cluster without additional physical hardware.
The end goal is a working Proxmox HA configuration where a hard failure of any single Proxmox node automatically triggers a VM restart on a surviving node — no manual intervention required. Getting there means solving four distinct problems: nested VM networking, shared storage over iSCSI, cluster quorum, and ensuring every VM disk lives on shared storage before HA will work.
What This Project Proves
- A functional 3-node Proxmox VE cluster can be built and operated inside nested VMware ESXi
- TrueNAS ZFS is a reliable, enterprise-grade iSCSI storage backend for Proxmox shared storage
- Network traffic separation (management, cluster, storage, VM) is achievable even in a nested lab environment
- Proxmox HA failover works end-to-end — VM restarts automatically on node failure when storage and quorum are correctly configured
- The entire stack maps directly to real enterprise Proxmox deployments, making it a credible production reference
Architecture
Design Decisions
Before building anything, several key decisions were made that shaped the entire architecture:
The choice of 3 nodes over 2 was deliberate — a 3-node cluster provides natural quorum (2 votes survive a single node failure) without needing an external quorum device. A 2-node cluster loses quorum the moment one node goes down, which defeats the purpose of HA.
TrueNAS over Ceph was chosen for shared storage because Ceph requires at minimum 3 separate storage nodes with dedicated OSDs, which is impractical in a nested lab where everything shares a single physical host. TrueNAS presents iSCSI as a single block device that Proxmox then manages as LVM — simpler, lower overhead, and directly mirrors how many enterprise environments use external SAN storage.
iSCSI with LVM over NFS was chosen because iSCSI presents block storage, which is required for Proxmox to run VMs at full performance with proper locking. NFS-backed VM storage works, but introduces file locking overhead and is not the standard for production VM disk storage.
Mirror vdevs (two pairs of mirrored disks) over RAIDZ was chosen for the TrueNAS pool because mirror vdevs deliver better random I/O throughput — the workload pattern for VM disks is predominantly small random reads and writes, not sequential streaming, so mirror outperforms RAIDZ in practice despite using more raw capacity.
Logical Topology
The full stack, from ESXi host to running VM:
VMware ESXi Host
└── vSwitch0
├── PG-PROXMOX-MGMT → 10.54.30.0/24 (management + OOB)
├── PG-PROXMOX-CLUSTER → 172.16.10.0/24 (Corosync cluster heartbeat)
├── PG-PROXMOX-STORAGE → 172.16.20.0/24 (iSCSI storage traffic)
└── PG-PROXMOX-VM → 10.54.30.0/24 (nested VM traffic)
Proxmox Cluster: YYC-PRX-CL01
├── YYC-PRX-N01 (Mgmt: 10.54.30.90 | Cluster: 172.16.10.11 | Storage: 172.16.20.11)
├── YYC-PRX-N02 (Mgmt: 10.54.30.91 | Cluster: 172.16.10.12 | Storage: 172.16.20.12)
└── YYC-PRX-N03 (Mgmt: 10.54.30.92 | Cluster: 172.16.10.13 | Storage: 172.16.20.13)
│
│ iSCSI (172.16.20.0/24)
▼
TrueNAS: YYC-TNAS-01 (Mgmt: 10.54.30.95 | Storage: 172.16.20.20)
└── pool_prx_01 (2× mirror vdevs, 4× 500 GiB virtual disks)
└── zvol-prx-vm-01 (800 GiB)
└── iSCSI target: tnas-iscsi-vm
└── Proxmox LVM: tnas-lvm-vm (shared across all 3 nodes)
└── Windows Server 2022 VM 100 (10.54.30.98)
└── HA resource → auto-restarts on node failure
Network Architecture
Traffic is separated into four logical networks, each on a dedicated ESXi port group. This separation is critical — Corosync cluster heartbeat traffic must not compete with iSCSI storage traffic on the same interface, and VM traffic must be isolated from storage traffic so a busy VM workload cannot saturate the iSCSI path.
| Port Group | Subnet | Purpose | Gateway |
|---|---|---|---|
PG-PROXMOX-MGMT |
10.54.30.0/24 | Node management, TrueNAS UI, VM access | 10.54.30.1 |
PG-PROXMOX-CLUSTER |
172.16.10.0/24 | Corosync heartbeat — cluster state only | None |
PG-PROXMOX-STORAGE |
172.16.20.0/24 | iSCSI storage traffic to TrueNAS | None |
PG-PROXMOX-VM |
10.54.30.0/24 | Nested VM network traffic | 10.54.30.1 |
Nested VM Security Settings
PG-PROXMOX-VM requires three ESXi security overrides to allow nested VM MAC addresses to pass through the Proxmox VM's virtual NIC. Without these settings, nested VMs can reach the Proxmox host but fail to reach the gateway — the traffic is silently dropped by ESXi at the port group level.
Node Network Mapping
Each Proxmox node has four virtual NICs mapped to the four port groups:
| Node | Management (vmbr0) | Cluster (nic1) | Storage (nic2/ens256) | VM Bridge (vmbr1) |
|---|---|---|---|---|
| YYC-PRX-N01 | 10.54.30.90 | 172.16.10.11 | 172.16.20.11 | vmbr1/ens161 |
| YYC-PRX-N02 | 10.54.30.91 | 172.16.10.12 | 172.16.20.12 | vmbr1/nic3 |
| YYC-PRX-N03 | 10.54.30.92 | 172.16.10.13 | 172.16.20.13 | vmbr1/nic3 |
Only vmbr0 (management) has a default gateway — cluster and storage interfaces are point-to-point only.
Naming Convention
All resources follow a location-prefixed naming standard using YYC for Calgary lab:
| Component | Name |
|---|---|
| Proxmox Cluster | YYC-PRX-CL01 |
| Proxmox Node 1/2/3 | YYC-PRX-N01/N02/N03 |
| TrueNAS | YYC-TNAS-01 |
| TrueNAS Pool | pool_prx_01 |
| iSCSI Storage (Proxmox) | tnas-iscsi-vm |
| LVM Storage (Proxmox) | tnas-lvm-vm |
| LVM Volume Group | vg_tnas_prx_vm_01 |
| Windows Test VM | Windows2022 (VM ID 100) |
Implementation
Phase 1 — ESXi Nested Lab Setup
The foundation is four ESXi port groups on vSwitch0, each carrying a distinct traffic type. The most important and least obvious step is configuring the security settings on PG-PROXMOX-VM. By default, ESXi drops frames where the source MAC address doesn't match the VM's own MAC — which is exactly what happens with nested VMs, since the Windows Server VM inside Proxmox has a MAC that ESXi doesn't know about. Enabling Promiscuous Mode, MAC Address Changes, and Forged Transmits on the VM port group overrides this enforcement and allows nested VM traffic to flow.
Phase 2 — Proxmox VE Installation
Three Proxmox VE VMs were deployed on ESXi, each with four vNICs mapped to the four port groups. Static IPs were configured on all three networks during installation. The management interface (vmbr0) receives a default gateway; the cluster and storage interfaces deliberately do not — traffic on those segments is node-to-node only and should never reach the default gateway.
Phase 3 — Proxmox Cluster Formation
With all three nodes running and reachable, the cluster was created from the first node:
The remaining nodes joined via the management IP of the first node. After joining, quorum status was confirmed:
Why quorum matters for HA
Proxmox HA only operates when the cluster is quorate. A 3-node cluster requires 2 votes to maintain quorum, so it tolerates exactly one node failure. If two nodes fail simultaneously, the cluster loses quorum and HA is suspended — this prevents split-brain scenarios where both surviving fragments try to start the same VM.
Phase 4 — TrueNAS Storage
TrueNAS was deployed as a VM on ESXi with virtual disks passed through as storage. A ZFS pool was created using two mirror vdevs across four 500 GiB virtual disks, giving 1 TiB of usable space with drive-level redundancy within each mirror pair.
An 800 GiB zvol (zvol-prx-vm-01) was carved from the pool to serve as the iSCSI backing store. A TrueNAS iSCSI target was configured pointing to this zvol, with an initiator group allowing connections from all three Proxmox node storage IPs.
Mirror vdevs vs RAIDZ for VM storage
Mirror vdevs deliver better random I/O performance than RAIDZ for VM disk workloads. VM disk I/O is dominated by small random reads and writes — the pattern where mirrors outperform RAIDZ. RAIDZ has better sequential throughput and raw capacity efficiency, but those advantages don't materialize in a VM storage workload.
Phase 5 — Proxmox Shared Storage
iSCSI storage was added to Proxmox from the Datacenter → Storage menu, targeting 172.16.20.20 (TrueNAS storage IP):
ID: tnas-iscsi-vm
Portal: 172.16.20.20
Target: iqn.2005-10.org.freenas.ctl:zvol-prx-vm-01
Shared: Yes
Nodes: All
LVM was then layered on top:
ID: tnas-lvm-vm
Base storage: tnas-iscsi-vm
Volume group: vg_tnas_prx_vm_01
Content: Disk image
Shared: Yes
After adding LVM storage, the volume group required manual activation on each node — it was not activated automatically at this stage:
This manual activation step is important to know — pvesm status may show the storage as inactive even after the Proxmox storage configuration is complete. Running the above commands on each node brings it online without a reboot.
Phase 6 — Windows Server 2022 VM
Windows Server 2022 was deployed as VM 100 (Windows2022) with the following configuration:
| Setting | Value |
|---|---|
| VM ID | 100 |
| OS | Windows Server 2022 |
| Network bridge | vmbr1 (VM port group) |
| IP address | 10.54.30.98/24 |
| Gateway | 10.54.30.1 |
| Disk storage | tnas-lvm-vm (shared) |
Windows installation required attaching the virtio-win.iso to load VirtIO SCSI drivers — without these, Windows cannot see the virtual disk during installation. After installation, all three required disk images (EFI disk, OS disk, TPM state) were confirmed on shared storage:
cat /etc/pve/qemu-server/100.conf
# efidisk0: tnas-lvm-vm:vm-100-disk-0
# scsi0: tnas-lvm-vm:vm-100-disk-1
# tpmstate0: tnas-lvm-vm:vm-100-disk-2
HA Hard Requirement — All Disks on Shared Storage
Proxmox HA will not migrate a VM if any disk is on local storage. This includes efidisk0 and tpmstate0 — not just the OS disk. If any of these three remain on local-lvm, HA silently fails to fence and restart the VM. Checking the .conf file before enabling HA is mandatory.
Phase 7 — Proxmox HA Configuration
VM 100 was added as a Proxmox HA resource via Datacenter → HA → Resources. The HA manager was confirmed running on all nodes:
HA failover was validated by performing a hard power-off on the node running the VM. The cluster detected the node failure, confirmed quorum was maintained (2 of 3 nodes remaining), and automatically restarted the VM on a surviving node within the configured timeout.
Troubleshooting
LVM Storage Shows Inactive After Reboot
Symptom: pvesm status shows tnas-lvm-vm as inactive after a node reboot, even though iSCSI is connected.
Root cause: LVM volume group activation is not automatic after iSCSI login on some node configurations. The volume group is present but not activated.
Resolution:
Long-term fix: Ensure open-iscsi is enabled and starting before LVM at boot. Check systemctl status open-iscsi and LVM activation service order.
Windows VM Cannot Reach Gateway
Symptom: Windows Server 2022 VM inside Proxmox can ping the Proxmox node's management IP but cannot reach 10.54.30.1 (gateway) or any external address.
Root cause: ESXi default port group security settings drop frames where the source MAC doesn't match the VM's registered MAC. The Proxmox VM presents one MAC to ESXi, but the Windows VM inside Proxmox has a different MAC — ESXi drops those frames silently.
Resolution: On the ESXi host, edit PG-PROXMOX-VM port group security settings:
HA Failover Does Not Trigger
Symptom: After powering off a Proxmox node, VM 100 does not restart on another node.
Root cause (most common): One or more VM disks are on local-lvm rather than tnas-lvm-vm. Proxmox HA cannot fence and restart a VM if the disk is inaccessible from the destination node.
Check:
cat /etc/pve/qemu-server/100.conf
# If any line shows local-lvm, move that disk to tnas-lvm-vm first
Root cause (less common): Cluster lost quorum due to the node failure. Check pvecm status — if Quorate: No, HA is suspended. This should not happen with a 3-node cluster losing only 1 node.
iSCSI Discovery Fails
Symptom: iscsiadm cannot discover the TrueNAS target.
Diagnostics:
Check: Verify TrueNAS is reachable on the storage network from the affected node:
If ping fails, check the ESXi port group assignment for the storage vNIC on the affected Proxmox node VM.
Validation
Network Validation
| Test | From | To | Result |
|---|---|---|---|
| Management ping N01 → N02 | YYC-PRX-N01 | 10.54.30.91 | ✅ Passed |
| Management ping N01 → N03 | YYC-PRX-N01 | 10.54.30.92 | ✅ Passed |
| Cluster network ping | YYC-PRX-N01 | 172.16.10.12/13 | ✅ Passed |
| Storage network ping | YYC-PRX-N01 | 172.16.20.20 | ✅ Passed |
| VM network (gateway reach) | Windows2022 | 10.54.30.1 | ✅ Passed (after ESXi fix) |
Storage and Cluster Validation
| Test | Result | Notes |
|---|---|---|
| Proxmox cluster quorum | ✅ Passed | Quorate: Yes, 3 votes |
| iSCSI discovery from all nodes | ✅ Passed | All 3 nodes discover target |
| LVM activation on all nodes | ✅ Passed | Required manual activation |
| tnas-lvm-vm shared across nodes | ✅ Passed | Storage active on all nodes |
VM and HA Validation
| Test | Result | Notes |
|---|---|---|
| Windows Server 2022 install | ✅ Passed | VirtIO drivers required |
| All VM disks on shared storage | ✅ Passed | efidisk0, scsi0, tpmstate0 all on tnas-lvm-vm |
| Manual VM migration N01→N02 | ✅ Passed | VM moved with no interruption |
| Manual VM migration N02→N03 | ✅ Passed | VM moved with no interruption |
| Node reboot recovery | ✅ Passed | Cluster reformed, storage reactivated |
| HA hard shutdown failover | ✅ Passed | VM restarted on surviving node automatically |
HA Failover is a Restart, Not a Live Migration
Proxmox HA failover is not live migration. When a node fails hard (power-off), the VM loses its in-memory runtime state and boots fresh on another node. Live migration preserves memory state but requires the source node to be healthy. Understanding this distinction matters when setting RTO expectations — HA restart takes seconds to minutes depending on Windows boot time, not the sub-second failover of a load balancer.
Constraints and Limitations
This lab is intentionally nested, which introduces constraints that would not exist in a bare-metal deployment:
- No hardware pass-through: TrueNAS uses virtual disks, not physical disks passed through an HBA. This means no ZFS disk-level health reporting or SMART data.
- Performance is shared: All Proxmox nodes, TrueNAS, and the test VM share the same physical ESXi host's CPU, memory, and storage. Performance numbers are not representative of production.
- Single-path iSCSI: Multipath iSCSI (MPIO) is not configured in this build. In production, dual-path iSCSI is standard practice for storage redundancy. This is a planned future enhancement.
- Single vSwitch: All port groups share
vSwitch0. In production these would be on separate physical NICs and switches.
These constraints are expected for a nested lab and do not affect the validity of the HA behaviour, cluster logic, or storage configuration being demonstrated.
Key Learnings
- Understanding why nested VM networking fails silently without ESXi port group security overrides — and what to check when connectivity appears to work at the Proxmox level but breaks at the VM level
- Designing a ZFS storage architecture for VM workloads — mirror vdevs vs RAIDZ, zvol sizing, iSCSI target configuration on TrueNAS
- Building Proxmox LVM on iSCSI — the two-layer storage model (iSCSI storage + LVM on top) and why LVM activation must be managed per node
- The hard requirement for shared storage across all disk types (efidisk0, scsi0, tpmstate0) before Proxmox HA works — the silent failure mode when local disks remain
- The difference between HA failover and live migration — HA is a restart after node failure, live migration is a zero-downtime move while the source is healthy
- 3-node quorum design — why 3 nodes are the minimum for a self-healing HA cluster without an external quorum device
- Cluster network isolation — why Corosync heartbeat traffic must be on a dedicated interface and must never share bandwidth with iSCSI storage traffic
Tools & Technologies
| Category | Technology |
|---|---|
| Host Hypervisor | VMware ESXi |
| Guest Hypervisor | Proxmox VE |
| Clustering | Proxmox VE Cluster Manager (Corosync) |
| High Availability | Proxmox HA Manager |
| Shared Storage | TrueNAS (ZFS) |
| Storage Protocol | iSCSI |
| Storage on Proxmox | LVM over iSCSI |
| Test VM OS | Windows Server 2022 |
| VM Drivers | VirtIO (storage + network) |
| Networking | ESXi vSwitch, port groups, VLAN segmentation |
| Diagnostics | pvecm, pvesm, ha-manager, iscsiadm, pvscan, vgchange |