Skip to content

noVNC Console Fails After a VM Migrates to a New Host

Symptom: a VM's console worked fine, but after migrating it (live or cold) to a different compute node, clicking Open Console hangs on "Connecting…" or fails outright — the VM itself is otherwise healthy (SSH, guest agent, ping all work).

Confirm it

On whichever node the VM is running on now:

virsh dumpxml <vm-name> | grep -A1 graphics

Correct: 

<graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
  <listen type='address' address='0.0.0.0'/>
</graphics>

Broken: 

<graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'>
  <listen type='address' address='127.0.0.1'/>
</graphics>

listen='127.0.0.1' (loopback-only) is the signature of this issue. You can also confirm directly — a plain TCP connect to the reported port should get an RFB handshake back:

python3 -c "import socket; s=socket.create_connection(('<node-ip>', <port>), timeout=5); print(s.recv(12))"

b'RFB 003.008\n' = reachable. A hang or ConnectionRefusedError = confirmed.

Root cause

The console proxy (routers/vms/console.py) always connects directly to the compute node's real IP, never localhost — a VM can be running on any node in the cluster, so it has to. It re-discovers which host the VM is actually on and re-reads its live libvirt XML fresh on every "Open Console" click, so a stale database record pointing at the old host is not the cause here — that part already self-heals.

What actually breaks it is the VM's <graphics type='vnc'> listen address being 127.0.0.1 instead of 0.0.0.0. A VM listening only on loopback is unreachable from the console proxy no matter which host it lands on — it just happens to surface specifically after a migration:

  • VMs deployed the normal way through mfconsole always get listen='0.0.0.0' at create time, so this never shows up for them.
  • A VM that reached inventory some other way (imported, hand-provisioned outside mfconsole, cloned from a template with a different libvirt default) can carry listen='127.0.0.1' from day one — invisibly, because nobody tried the console before it moved.
  • A live migration transfers the domain's currently-running XML configuration to the destination as-is — whatever listen value it already had follows it to the new host unchanged. The persistent (on-disk) config on the source is not what gets migrated, so fixing that alone beforehand wouldn't have helped.

Fix

0.42.5.12-beta and later self-heal this automatically. Both migration paths (execute_migration_task in routers/vms/advanced.py) now force the destination VM's VNC listen address to 0.0.0.0 as part of the move:

  • Live migration builds a corrected copy of the running domain's XML and hands it to the destination via virsh migrate --xml (libvirt's documented mechanism for supplying an alternate destination config) — the normal migrate command otherwise has no way to change what config the new host uses.
  • Cold migration fixes the XML before defining it on the destination.

So on a fixed build, migrating an affected VM to any other Active node repairs it as a side effect — no manual XML editing needed. (This is how the initial batch of affected lab VMs was repaired: migrated once each to swap them onto their cluster's other node, which triggered the self-heal.)

On an older build, or to fix a VM without migrating it:

# on the node the VM currently runs on
virsh dumpxml --inactive <vm-name> > /tmp/fix.xml
sed -i "s/listen='127.0.0.1'/listen='0.0.0.0'/" /tmp/fix.xml
virsh define /tmp/fix.xml

This only updates the persistent config — virsh define doesn't touch a currently-running instance's already-bound socket. A running VM keeps its old (broken) live listen address until it's actually restarted: virsh shutdown + virsh start (or destroy/start if it won't shut down cleanly), or just migrate it once you're on 0.42.5.12-beta+.

Checking a whole fleet at once

If you suspect more than one VM is affected (e.g. a batch imported from the same source), loop virsh dumpxml <name> | grep listen= across every VM on a node rather than waiting for someone to notice the console doesn't work — this issue is silent until someone actually clicks "Open Console."