Skip to content

VM HA Failover Doesn't Restart VMs After a Node Goes Down

Symptom: a compute node in an HA-enabled cluster goes down, but its VMs never come back up on a surviving node. The node's status shows Fence-Failed (or stays stuck there even after it's back up), and clicking around for a way to recover it isn't obvious.

How automatic failover actually works

A background monitor (check_ha_status, runs every 60s) drives four stages per node:

  1. Detect — a node fails a reachability check → its ha_unreachable_since timestamp is stamped (within ~60s of the real failure).
  2. Grace period — the node must stay unreachable continuously for the cluster's HA threshold (default 5 minutes, configurable — see below) before anything else happens. This avoids failing over on a brief network blip.
  3. Fence — the master tries to prove the node is actually down before touching its VMs' shared disks:
    • If the node has a BMC/iDRAC configured, it force-powers it off via Redfish and confirms the power state.
    • If there's no BMC configured (common on nested/lab hosts), it skips the power-fence entirely and instead blocklists the node's Ceph RBD client — that alone is enough to make Ceph-backed VMs safe to restart elsewhere without a confirmed power-off.
  4. Restart — only VMs on shared storage (Ceph RBD, DRBD, NFS) are eligible. Local/ZFS-backed VMs are never auto-failed-over since their disk isn't replicated.

The two possible outcomes

Status Meaning What happened to the VMs
Failed The node was fenced (power-off confirmed, or Ceph-blocklisted for an RBD-backed workload) Eligible VMs were automatically redefined and started on a surviving node
Fence-Failed Neither fence method could be confirmed Nothing was touched — this is a deliberate split-brain guard, not a bug

If you see Fence-Failed, that's the system refusing to guess. Restarting a VM elsewhere while its old host might still be alive and writing to the same disk is exactly the scenario that corrupts shared storage — so it stops and waits for a human to confirm reality.

Recovering a Fence-Failed node

Right-click the node in Compute Nodes (or the ⋯ menu) — the menu now shows the right actions based on status:

  • ⚡ Force Off — sends a BMC power-off command. Only works if the node actually has an iDRAC/BMC configured; for nodes without one (nested/lab hosts), this option won't help — confirm the node is off some other way instead (out-of-band power control, hypervisor console, or physically).
  • Reactivate — clears the stuck Fence-Failed state and cleans up any leftover VM definitions from a partial failover. This does not itself check that the node is powered off — only click it once you've independently confirmed that.

Reactivate before the node is truly down = it'll just fail again

If you Reactivate while the node is still actually unreachable, the monitor re-detects it as down on the next cycle and restarts the whole detect → threshold → fence sequence from scratch.

Adjusting the HA threshold

Each cluster has its own grace period. Open the cluster's Summary panel → the HA failover row shows Enabled · offline <N>m → with a small edit link — click it to set a new value (1–1440 minutes). Useful to shorten temporarily while validating HA behavior on a test cluster.