Skip to content

ZFS Pool Replication & VM Restore (DR)

Async, snapshot-based replication of a local ZFS pool to another host in the same compute cluster, plus a one-click way to rebuild and boot the replicated VMs on the target — MFCloud's answer to "what happens if a ZFS-only node dies."

This is not the same thing as LINSTOR / DRBD or Ceph replication: those are synchronous, per-VM, and support live migration. ZFS replication here is asynchronous, per-pool, snapshot-interval based, and the replica is inert (just files) until you explicitly restore it.

How it works

  1. Schedule: pick a source pool, a target host in the same cluster, and a target pool name. MFCloud verifies node-to-node SSH trust exists (or builds it) before saving.
  2. Replicate: on the interval you set (or on-demand via Run Now), the source node's own Celery worker snapshots the pool and pipes zfs send -R straight into ssh root@target zfs recv -F <target_pool> — data goes node-to-node, never through the master. Incremental after the first run.
  3. Capture: right after a successful send, MFCloud also grabs virsh dumpxml --inactive for every VM whose disk lives on that pool and stores it. This is what makes restore possible — zfs send/recv only ever moves raw files, it never defines a libvirt domain anywhere.
  4. Restore: pick which captured VMs to rebuild on the target. MFCloud adapts the saved domain XML (repoints the disk path at the target pool, drops any device that wasn't actually replicated — e.g. a cloud-init seed ISO — and strips runtime-only details like the old domain id, PCI addresses, and seclabels) and does virsh define + virsh start directly on the target node.
  5. Cut over: restore intentionally does not touch the console's VM record if that VM name is still owned by a different host — that would silently hide a VM that might still be alive elsewhere. Finish the cutover yourself with Hosts → Scan on the target, then Adopt.

Setup

  1. Storage → ZFS, open the source pool's manager.
  2. Under Schedule Replication:
  3. Source ZFS Pool — already scoped to the pool you opened.
  4. Target Host (same cluster) — only shows other Active hosts in the same compute cluster.
  5. Target Pool Name — must be an existing pool on the target, and must match exactly (case and punctuation both matter — DATA_ZFS-02 and DATA-ZFS-02 are different pools; a mismatch doesn't fail validation, it fails at receive time with destination does not exist).
  6. Interval — minutes between automatic runs, 5–10080. Run Now ignores this and fires immediately regardless of interval or enabled/disabled state.
  7. Schedule Replication, then Run Now to trigger the first send instead of waiting for the interval.

Use a dedicated target pool — never a pool with live VMs on it

zfs recv -F doesn't merge — it force-rolls back the entire destination pool to match the incoming stream. If the target pool has its own VMs running, two things go wrong:

  • While anything on the target pool is open (a running VM's disk file), the receive fails outright: cannot unmount '/<pool>': pool or dataset is busy.
  • If you do get it to receive (target VMs stopped first), their disks are silently overwritten to match the source — data loss, not a merge.

Recommendation: create a small dedicated pool (we use backup) on every node that participates in replication, and always target that instead of a pool that also hosts production VMs. It never has anything "busy" on it, so replication runs cleanly on its own schedule without coordinating around what's currently running.

Progress bar

If pv is installed on the source node, the Replication Schedules table shows a live percentage while a send is in flight (auto-refreshes every 2s; falls back to just the status text if pv isn't present — the transfer itself is identical either way). pv is included by default on node ISOs built from the current kickstart; on existing nodes, install it manually:

dnf install -y epel-release && dnf install -y pv

Restoring / failing over

  1. On the replication schedule's row, click 🔁 Restore.
  2. Select which captured VM(s) to rebuild, and whether to start them immediately.
  3. Restore Selected — this runs directly on the target node (no SSH needed, the data's already local) and is idempotent: re-running it destroys/undefines any prior attempt of the same name first, so a failed or repeated restore doesn't collide with itself.
  4. If the source host is genuinely down, finish the cutover: Hosts → Scan on the target host, then Adopt the newly-running VM(s) — this is what re-points the console's record so they show up correctly (until then they're running, just still listed under the old host).

Failing back

Once the original host recovers, decide whether you need to preserve anything written during the outage:

  • No new data to preserve (e.g. it was only down briefly, nothing changed) — just power the original host back on. The original VM disks/definitions are untouched (only powered off, not destroyed); start them directly with virsh start, then re-adopt them back under the original host.
  • Need to preserve what changed on the failover target — schedule replication in the reverse direction (target pool → original pool), Run Now, then use Restore on that new schedule to rebuild on the original host from the up-to-date copy. Same mechanics as the forward direction, just pointed the other way.

Either way, if the pool that's now acting as a receive target was previously used as a source, clear its leftover snapshots first (see below) — otherwise the first receive in the new direction fails.

Troubleshooting

cannot receive new filesystem stream: destination has snapshots (eg. <pool>@mfcloud-repl-...) must destroy them to overwrite it
The destination pool has leftover replication snapshots from a previous run — typically from having been a source pool at some point, or from a schedule that was deleted and recreated. List and clear them on the destination node: 
zfs list -t snapshot -r <pool>
zfs destroy -r <pool>@mfcloud-repl-<timestamp>   # repeat per snapshot
cannot unmount '/<pool>': pool or dataset is busy
Something on the destination pool is actively open — almost always a running VM whose disk lives there. Stop it (virsh shutdown <vm> / virsh destroy <vm>) and retry, or better, switch this schedule's target to a dedicated pool per the recommendation above.
cannot receive new filesystem stream: destination '<pool>' does not exist
Typo in Target Pool Name — it doesn't auto-create the pool, and validation only checks the string shape, not that it matches a real pool. Re-check for underscore vs. hyphen against zpool list on the target.
Restored VM doesn't show up in the VM list
Expected if its name is still owned by another host in the console (e.g. the source is still up, or you haven't adopted yet). It's actually running on the target — check virsh list --all there — you just need Hosts → Scan/Adopt on the target to finish the cutover.
Restore fails with "no domain with matching name" / cdrom errors
A restored VM's cloud-init seed ISO (/var/lib/libvirt/images/<vm>-cidata.iso) isn't part of the ZFS pool, so it never replicates — restore drops that cdrom device automatically rather than pointing at a file that isn't there. This is a no-op for an already-provisioned VM (cloud-init only matters on first boot); it's a real gap for VMs whose first boot hasn't completed on the source.

See also