Skip to content

noVNC Console Fails After a VM Migrates to a New Host

Symptom: a VM's console worked fine, but after migrating it (live or cold) to a different compute node, clicking Open Console hangs on "Connecting…" or fails outright — the VM itself is otherwise healthy (SSH, guest agent, ping all work).

Different symptom: console looks frozen/black instead of failing to connect

If the console was already open in your browser when the live migration happened, you'll see a stale/frozen frame rather than a connect failure — that's a separate, already-fixed issue (0.44.0.16+): live migration kills the source QEMU process on cutover, which drops the proxied connection an open console was riding on, but the video canvas kept showing the last frame instead of surfacing the disconnect. Fixed builds show a "Connection lost — the VM may have migrated to another host" overlay with a Reconnect button on any unexpected disconnect. If you see that overlay, it's expected behavior working correctly, not a bug — click Reconnect. The rest of this page covers a different symptom: a console that was not already open, and fails to connect after the fact.

Confirm it

On whichever node the VM is running on now:

virsh dumpxml <vm-name> | grep -A1 graphics

Correct: 

<graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
  <listen type='address' address='0.0.0.0'/>
</graphics>

Broken: 

<graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'>
  <listen type='address' address='127.0.0.1'/>
</graphics>

listen='127.0.0.1' (loopback-only) is the signature of this issue. You can also confirm directly — a plain TCP connect to the reported port should get an RFB handshake back:

python3 -c "import socket; s=socket.create_connection(('<node-ip>', <port>), timeout=5); print(s.recv(12))"

b'RFB 003.008\n' = reachable. A hang or ConnectionRefusedError = confirmed.

Root cause

The console proxy (routers/vms/console.py) always connects directly to the compute node's real IP, never localhost — a VM can be running on any node in the cluster, so it has to. It re-discovers which host the VM is actually on and re-reads its live libvirt XML fresh on every "Open Console" click, so a stale database record pointing at the old host is not the cause here — that part already self-heals.

What actually breaks it is the VM's <graphics type='vnc'> listen address being 127.0.0.1 instead of 0.0.0.0. A VM listening only on loopback is unreachable from the console proxy no matter which host it lands on — it just happens to surface specifically after a migration:

  • VMs deployed the normal way through mfconsole always get listen='0.0.0.0' at create time, so this never shows up for them.
  • A VM that reached inventory some other way (imported, hand-provisioned outside mfconsole, cloned from a template with a different libvirt default) can carry listen='127.0.0.1' from day one — invisibly, because nobody tried the console before it moved.
  • A live migration transfers the domain's currently-running XML configuration to the destination as-is — whatever listen value it already had follows it to the new host unchanged. The persistent (on-disk) config on the source is not what gets migrated, so fixing that alone beforehand wouldn't have helped.

Fix

0.42.5.12-beta and later self-heal this automatically. Both migration paths (execute_migration_task in routers/vms/advanced.py) now force the destination VM's VNC listen address to 0.0.0.0 as part of the move:

  • Live migration builds a corrected copy of the running domain's XML and hands it to the destination via virsh migrate --xml (libvirt's documented mechanism for supplying an alternate destination config) — the normal migrate command otherwise has no way to change what config the new host uses.
  • Cold migration fixes the XML before defining it on the destination.

So on a fixed build, migrating an affected VM to any other Active node repairs it as a side effect — no manual XML editing needed. (This is how the initial batch of affected lab VMs was repaired: migrated once each to swap them onto their cluster's other node, which triggered the self-heal.)

On an older build, or to fix a VM without migrating it:

# on the node the VM currently runs on
virsh dumpxml --inactive <vm-name> > /tmp/fix.xml
sed -i "s/listen='127.0.0.1'/listen='0.0.0.0'/" /tmp/fix.xml
virsh define /tmp/fix.xml

This only updates the persistent config — virsh define doesn't touch a currently-running instance's already-bound socket. A running VM keeps its old (broken) live listen address until it's actually restarted: virsh shutdown + virsh start (or destroy/start if it won't shut down cleanly), or just migrate it once you're on 0.42.5.12-beta+.

Checking a whole fleet at once

If you suspect more than one VM is affected (e.g. a batch imported from the same source), loop virsh dumpxml <name> | grep listen= across every VM on a node rather than waiting for someone to notice the console doesn't work — this issue is silent until someone actually clicks "Open Console."