Skip to content

HA Manager (Out-of-Band Console Management)

The HA Manager is a small web UI for operating a 3-node HA console cluster: live cluster status, log viewing, service restarts, controlled VIP failover, and rolling code updates — all from one page.

Its defining property is that it runs outside the console stack: a plain systemd service (mfcloud-haman) directly on each control-plane host, not a container. When the console itself is broken — database down, containers crashed, VIP missing — the HA Manager still answers. That is exactly the moment you need it.

Access

  • URL: https://<any-control-plane-host>:9444 — every host runs its own copy and each copy manages the whole cluster, so use whichever host answers.
  • Login: admin / mfpro (default). The TLS certificate is self-signed; accept the browser warning.

Change the default password

Manager Settings → Change Password updates the credential on the host you are browsing — repeat it on each of the three hosts. Credentials are stored as bcrypt hashes in /etc/mfcloud-haman.htpasswd, independent of console accounts.

The dashboard

One card per control-plane host:

  • VIP badge — which host currently holds the floating IP (shown live in the header too).
  • Patroni role — primary / replica per host, straight from each host's Patroni REST API.
  • keepalived state, disk usage, load average.
  • Every container on the host with an up/down indicator and a one-click restart.

The page refreshes itself every 10 seconds.

Logs

Pick a host, pick a target — any container, the keepalived journal, or the HA Manager's own journal — choose a tail length, and optionally tick follow for a 10-second refresh loop. Log retrieval crosses hosts transparently, so you can read Host 3's database log while browsing Host 1.

Controlled VIP failover

The current VIP holder's card shows a Fail over VIP button. It stops keepalived on that host; the next-priority host takes the VIP within ~3 seconds.

Failover is sticky until you say otherwise

keepalived stays stopped on the old holder so the VIP doesn't bounce back mid-maintenance. When you're done, restart keepalived on that host (button on its card) — it will reclaim the VIP by priority.

Rolling updates

Push Update rolls new console code across all three hosts with the console staying up throughout:

  1. Choose a source — either a host whose /root/mfcloud-master tree already has the new code, or upload a .tar.gz bundle of the repo tree from your browser.
  2. Ship — the source tree is copied to the other hosts. Each host's own .env (its cluster identity) is never overwritten, whether shipping or uploading.
  3. Build — container images build on all three hosts in parallel while the old containers keep serving traffic. A build failure on any host aborts the update before anything restarts.
  4. Rolling restart — containers are recreated from the new image one host at a time, lowest keepalived priority first, VIP holder last. Each host must pass its health check before the roll continues; a failure stops the roll and leaves the remaining hosts untouched.

The live log streams into the page; leave it open or come back later — the update runs server-side.

How it works / security notes

Aspect Detail
Process mfcloud-haman.service (systemd, uvicorn), port 9444/tcp, per host
Peer control Root SSH key mesh between the three control-plane hosts (BatchMode, set up at install)
Auth HTTP Basic against a per-host bcrypt htpasswd; closed-fail if the file is missing
CSRF State-changing requests require the X-Requested-With: mfcloud-haman header plus a same-origin check
TLS Self-signed cert at /etc/pki/tls/certs/mfcloud-haman.crt, generated at install
Config /etc/mfcloud-haman.conf — cluster hosts (priority order), VIP, repo path

Because it can restart anything and push code as root, treat port 9444 as an admin-network-only port: restrict it at your firewall to operator subnets.

Install / reinstall

The HA Manager ships in the console repo under haman/. On each control-plane host:

bash /root/mfcloud-master/haman/install.sh

The installer is idempotent — it installs Python dependencies, the systemd unit, TLS cert and default credential only if missing, and (re)copies the application files. Re-run it to deploy a newer HA Manager version.

Troubleshooting

Symptom Check
:9444 not answering systemctl status mfcloud-haman and journalctl -u mfcloud-haman -n 50 on that host; try another host meanwhile.
A peer card shows UNREACHABLE The SSH mesh to that host is broken or the host is down: ssh -o BatchMode=yes root@<peer> true from the host you're browsing.
Update stuck at build phase Builds legitimately take several minutes per host. Check journalctl -u mfcloud-haman and disk space (df -h) on the slow host.
Login refused after password change Credentials are per host — you may have changed it on a different host than the one you're browsing now.