Skip to content

Deploy the Console in High Availability (3-Node)

A standard MFCloud install runs the console on a single host. The Enable HA wizard converts that single-node console into a 3-host active cluster in one guided operation — you supply two fresh hosts and one spare IP, and the console does the rest itself.

After the wizard finishes you get:

Layer Before After
Entry point Host 1's IP A floating VIP (keepalived/VRRP) that moves to a surviving host in ~3 s
Web/API 1 app instance 3 load-balanced app instances (any host can serve any request)
Database 1 PostgreSQL 3-node Patroni cluster with streaming replication and automatic primary failover
Cluster brain — 3-node etcd quorum (leader election / split-brain protection)

One-time, one-way

Enable HA is a single-node → 3-node bootstrap and can only run once per console. Growing an existing HA cluster to 4+ hosts is not supported yet, and there is no "disable HA" path. Plan the three hosts and the VIP address before you start.

Architecture

flowchart TB
    U[Operators / API clients] --> VIP{{"Floating VIP<br/>(keepalived, VRRP unicast)"}}
    VIP --> C1[Caddy · Host 1]
    C1 --> A1[app-1] & A2[app-2] & A3[app-3]
    A1 & A2 & A3 --> HAP["HAProxy<br/>(routes writes to the<br/>current Patroni primary)"]
    HAP --> P1[(db-1<br/>primary)]
    P1 -. streaming replication .-> P2[(db-2<br/>replica)] & P3[(db-3<br/>replica)]
    E[etcd-1/2/3 quorum] -. leader election .- P1

Every host runs the full stack (Caddy, app, worker, Redis, HAProxy, Patroni PostgreSQL, etcd, keepalived). Whichever host holds the VIP is the entry point; if it dies, the next host takes the VIP and its HAProxy/Patroni promote a new database primary automatically.

Prerequisites

  • A running single-node console (this becomes Host 1, the highest-priority cluster member) with root SSH enabled on itself — the wizard provisions Host 1 the same way it provisions Hosts 2/3: by SSHing out to its own LAN IP (CONTROL_PLANE_IP), since the console runs inside a container and can't manage its own host's packages/services from inside itself.
  • Two fresh hosts — Rocky/RHEL/Alma family, root SSH enabled, no MFCloud software installed (the wizard installs podman, podman-compose, and keepalived itself).
  • One free static IP on the same subnet as all three hosts, to become the VIP. All three hosts must share an L2 segment — VRRP does not cross routers.
  • Sizing: each host should meet the normal console minimums; the database is fully replicated to all three.

Run the wizard

  1. Log in as a global admin and open the datacenter view.
  2. Open the actions menu (where Add Host… lives) → 🧬 Enable HA (3-Node Cluster)… The entry only appears while the console is still single-node.
  3. Fill in the six fields: Host 1's own root password, Host 2 IP + root password, Host 3 IP + root password, and the VIP address.
  4. Submit and confirm the prompt — the modal switches to a live progress log and the bootstrap runs unattended for several minutes. Partway through, the console itself restarts as Host 1 switches from its single-node database/broker to the HA stack; the progress log keeps streaming once it comes back (a few seconds of gap in the log during that restart is expected, not a failure).

Behind the scenes, in order: SSH reachability is verified on all three hosts before anything is touched, the current database is backed up, the console code ships to all three (this is also what puts the HA Manager in place on each host), packages install, per-host configuration is generated (each host keeps its own .env identity), the single database is adopted as db-1 and the replicas sync automatically, all services start, keepalived brings up the VIP on Host 1, any already-enrolled compute nodes are re-pointed at the VIP, and — last, since it recreates the console's own container — Host 1's own app tier switches over.

After it finishes

  • Browse to the VIP — that is the console's address from now on. Update your DNS record to point at it.
  • The 🧬 HA Cluster widget appears bottom-left in the console, showing each host and its database role.
  • The Enable HA menu entry disappears permanently.
  • Install the HA Manager tooling for day-2 operations (out-of-band status, logs, rolling updates, controlled failover) — the wizard already placed it on all three hosts, so this is just running its installer.

If a step fails

There is no automatic rollback — a half-built Patroni/etcd cluster is riskier to auto-undo than to finish by hand. Instead:

  • The progress log states exactly which step succeeded and which failed.
  • The original single-node database volume is never deleted; worst case, the pre-HA console can be brought back by starting its original containers.
  • Fix the cause (usually SSH credentials — including Host 1's own, firewall, or a taken VIP address) and re-run, or finish the remaining steps manually.

Troubleshooting

Symptom Check
VIP never appears journalctl -u keepalived on Host 1 — look for VRRP state transitions; confirm no other device answers the VIP address (arping <vip>).
A replica won't sync curl http://<host>:8008/patroni on each host — roles should be one primary + two replica.
Console intermittently errors after failover Give Patroni ~10 s to promote; the app replaces broken pooled DB connections automatically on the next request.
Which host owns the VIP right now? The HA Manager header shows the live holder, or ip -4 addr show | grep <vip> per host.