Skip to content

VMs Show "Host Offline"

Symptom: every VM on a given compute host shows status Host Offline in the Virtual Machines list — 0% CPU, 0% Memory, — IP address — even though the host itself is up, reachable, and happily running those VMs. Console open, power actions, and telemetry all fail for anything on that host.

Where this status comes from

The VM list doesn't ask the database whether a VM is running — it opens a live libvirt connection to the VM's host (qemu+ssh://root@<host-ip>/system) on every telemetry tick. If that connection throws any exception, every VM on that host gets stamped Host Offline with the exception text attached as status_detail. So this status is really "the master couldn't SSH+libvirt into the host just now," not "the host is down."

Diagnostics

  1. Check the real error. The exact exception is logged server-side and (depending on version) surfaced in status_detail in the API response:

    # On the control-plane node currently serving requests
podman logs --since 10m mfcloud-master_app-1_1 2>&1 | grep -i "<host-ip> unreachable"

    You'll see one of two very different errors — they need different fixes:

    VM telemetry: host <host-ip> unreachable: libvirtError: Cannot recv data: Host key verification failed.: Connection reset by peer
VM telemetry: host <host-ip> unreachable: libvirtError: Cannot recv data: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer

  2. Confirm the host is actually reachable at the network level (rule out a real outage first):

    ping -c3 <host-ip>
ssh -o BatchMode=yes -o ConnectTimeout=5 root@<host-ip> echo OK

    If ping/SSH-port fail too, this is a real network/host problem, not the SSH-layer issue below — check the physical host, not mfcloud.

  3. Reproduce from inside the actual container, not from the host OS. This matters — see the gotcha below.

    podman exec mfcloud-master_app-1_1 ssh -o BatchMode=yes -o ConnectTimeout=5 root@<host-ip> echo SSH_OK

⚠️ Gotcha: the host OS's /root/.ssh is not what the app uses

docker-compose.yml bind-mounts ./ssh_keys (a path inside the compose project directory, e.g. /root/mfcloud-master/ssh_keys) to /root/.ssh inside the app-N container. That is a completely different directory from the control-plane host's own /root/.ssh. Testing SSH/virsh directly on the host OS as root proves nothing about what the containerized app can actually do — always reproduce with podman exec ... ssh ... (step 3 above), or check <compose-project-dir>/ssh_keys/known_hosts directly instead of /root/.ssh/known_hosts.

Root cause 1 — Permission denied: master's key isn't authorized

The control-plane node's SSH public key was never added to (or was removed from) /root/.ssh/authorized_keys on the compute host.

Fix: Hosts page → the affected host → Repair Agent. This re-pushes the master's public key (bootstrapping via the host's stored root password if key auth isn't there yet), among other repair steps. It's non-destructive — it does not require removing and re-adding the host.

Root cause 2 — Host key verification failed: missing/stale known_hosts entry

This is the subtler one, and Repair Agent does not fix it.

The VM list's libvirt probe shells out to the system ssh binary, which enforces host-key checking against the plain OpenSSH known_hosts file (<ssh_keys-dir>/known_hosts, mounted as /root/.ssh/known_hosts in the container). Repair Agent's key-push step uses a separate paramiko-based connection (its own TOFU/pinning logic in core/ssh.py) — it authorizes the master's public key on the target, but it never re-runs ssh-keyscan to (re-)seed that plain known_hosts file. Only the Add Host onboarding flow does that.

So if a compute host was ever reimaged, had its SSH host key regenerated, or simply never got scanned (e.g. it was added before the master's known_hosts was seeded, or known_hosts was reset/rotated), Repair Agent will report success and the host will still show Host Offline — because it fixed the key auth but not the host-key pin the libvirt probe actually checks.

Fix — manually seed the missing entry:

# On the control-plane host, into the SAME directory the containers bind-mount:
ssh-keyscan -H -T5 <host-ip> >> <compose-project-dir>/ssh_keys/known_hosts

# Verify from inside the container that actually serves requests:
podman exec mfcloud-master_app-1_1 ssh -o BatchMode=yes -o ConnectTimeout=5 root@<host-ip> echo SSH_OK

In an HA deployment, remember each control-plane node has its own ssh_keys/known_hosts (not synced), and Caddy round-robins requests across all of them — check/fix this on every control-plane node, not just the one you happen to be SSHed into. Use ssh-keygen -F <host-ip> -f known_hosts to check for an existing (possibly hashed) entry rather than a plain grep, which silently misses hashed hostnames.

Once seeded, the "Host Offline" status clears on the next telemetry tick (a few seconds), no restart needed.

Known product gap (not yet fixed in code)

routers/hosts/repair.py's _repair_agent_task() should call the same ssh-keyscan seeding step that services/host_tasks.py's Add Host flow already does, so Repair Agent alone is sufficient to recover a host in either failure mode. Today it only covers root cause 1. Tracked as a follow-up — until it lands, root cause 2 needs the manual fix above.