Fail VMs Over to Another Site (Recovery Plans)¶

A recovery plan keeps copies of chosen VMs at a second site and moves them there when the first site is lost or needs to go down. You can rehearse it at any time with a test failover that doesn't touch the running VMs.

Recovery plans are for Super Admins, under the Recovery tab.

What you need first¶

Two sites. Right-click each datacenter and choose Set Site…: the protected one (for example East) and the standby one (West). A site is the datacenter's physical location; several datacenters can share one. To put a cluster in another datacenter, right-click it and choose Move to Datacenter….
The VMs on ZFS. Recovery plans protect VMs whose disks are on a ZFS pool.
A replica at the standby site. In Datastores → ZFS → Replication, pick the pool the VMs live on and a host at the other site. Hosts at other sites appear under Other sites (relayed through the console): the console carries those copies itself, so the two sites' hosts don't need to reach each other. The target pool must be empty. Run it once (Run now); its VMs can be added to a plan after the first run.

Create a plan¶

Recovery → New plan. Give it a name, an RPO (how old a replica may be, in minutes) and a boot timeout per group.
Protected VMs. Tick the VMs and give each a boot group. Group 1 starts first; the next group starts when every VM in the group before it is up, after the longest Wait after of that group. Put databases before the applications that use them.
Network map. For each network the VMs use, name the bridge or libvirt network they should join at the standby site, and optionally an isolated network for tests (a libvirt network with no forwarding). Leave the test network empty and test copies boot with no network at all.

The plan lists anything that would stop a run: a replica older than the RPO, a network with no target, a VM whose definition hasn't been captured yet.

Whole pools

Failover and failback move whole ZFS pools, so every VM on a replicated pool has to be in the plan before you can fail over. A test failover works with any subset.

Test failover¶

Test failover boots a copy of every VM at the standby site from a clone of its newest replica, on the isolated test network or no network, waits for each guest agent to answer, records how long each took, and deletes the copies and clones. The VMs at the protected site keep running. Replication for the plan's VMs pauses during the test and resumes afterwards.

The run's report shows each VM's boot time and the measured RTO: the time from the start of the run until the last VM answered.

Failover¶

Failover… asks whether the protected site is up, and for the plan's name.

Up (planned): the VMs are shut down at the protected site, their autostart is turned off so they can't come back there, one last sync runs, and they start at the standby site in boot order. Nothing is lost.
Down (unplanned): the newest replica is used as it is. Changes made after it are lost.

The VMs keep their names and MAC addresses and join the mapped networks. The console's records move with them, and replication stays paused until you fail back. If clients reach the VMs by name, update DNS to their new addresses.

Failback¶

Failback… stops the VMs at the standby site, sends what changed there back to the protected site, starts them there again (with autostart back on where it was), and resumes replication from that point.

Keep only the plan's VMs on a protected pool

Failback rolls the whole source pool back to the standby site's copy. Anything else stored on that pool, such as files, backups or other datasets, is lost. Give protected VMs a pool of their own.

After a failback the replication runs straight away, so a daily schedule moves to that time of day. Change it back if the old time mattered.

When the console is lost too¶

If the whole protected site is down, the console may be down with it. Bring up a console at the standby site from your control-plane backup, then run the plan's unplanned failover from there.

Start that console as a standby: set CONSOLE_STANDBY=true in its .env before you restore. A console restored from backup knows every host and schedule of the original. As a standby it runs no schedules, no HA monitoring and no background work against the hosts, and every page shows a banner saying so. You can still sign in, check the plans and run them. Once the original console is confirmed down, promote the standby: set CONSOLE_STANDBY=false and restart app, worker and beat.

Keep the original console's SECRET_KEY and its S3 backup settings somewhere that survives the site. The backup can't be decrypted without them. The backup is taken nightly, so take one by hand after you change a recovery plan.