Skip to content

Deploy the Console in High Availability (3-Node)

A standard MFCloud install runs the console on a single host. The Enable HA wizard converts that single-node console into a 3-host active cluster in one guided operation — you supply two fresh hosts and one spare IP, and the console does the rest itself.

After the wizard finishes you get:

Layer Before After
Entry point Host 1's IP A floating VIP (keepalived/VRRP) that moves to a surviving host in ~3 s
Web/API 1 app instance 3 load-balanced app instances (any host can serve any request)
Database 1 PostgreSQL 3-node Patroni cluster with streaming replication and automatic primary failover
Cluster brain — 3-node etcd quorum (leader election / split-brain protection)

One-time, one-way

Enable HA is a single-node → 3-node bootstrap and can only run once per console. Growing an existing HA cluster to 4+ hosts is not supported yet, and there is no "disable HA" path. Plan the three hosts and the VIP address before you start.

Architecture

flowchart TB
    U[Operators / API clients] --> VIP{{"Floating VIP<br/>(keepalived, VRRP unicast)"}}
    VIP --> C1[Caddy · Host 1]
    C1 --> A1[app-1] & A2[app-2] & A3[app-3]
    A1 & A2 & A3 --> HAP["HAProxy<br/>(routes writes to the<br/>current Patroni primary)"]
    HAP --> P1[(db-1<br/>primary)]
    P1 -. streaming replication .-> P2[(db-2<br/>replica)] & P3[(db-3<br/>replica)]
    E[etcd-1/2/3 quorum] -. leader election .- P1

Every host runs the full stack (Caddy, app, worker, Redis, HAProxy, Patroni PostgreSQL, etcd, keepalived). Whichever host holds the VIP is the entry point; if it dies, the next host takes the VIP and its HAProxy/Patroni promote a new database primary automatically.

Prerequisites

  • A running single-node console (this becomes Host 1, the highest-priority cluster member).
  • Two fresh hosts — Rocky/RHEL/Alma family, root SSH enabled, no MFCloud software installed (the wizard installs podman, podman-compose, and keepalived itself).
  • One free static IP on the same subnet as all three hosts, to become the VIP. All three hosts must share an L2 segment — VRRP does not cross routers.
  • Sizing: each host should meet the normal console minimums; the database is fully replicated to all three.

Run the wizard

  1. Log in as a global admin and open the datacenter view.
  2. Open the actions menu (where Add Host… lives) → 🧬 Enable HA (3-Node Cluster)… The entry only appears while the console is still single-node.
  3. Fill in the five fields: Host 2 IP + root password, Host 3 IP + root password, and the VIP address.
  4. Submit. The modal switches to a live progress log and the bootstrap runs unattended for several minutes.

Behind the scenes, in order: SSH reachability is verified on both new hosts before anything is touched, the current database is backed up, the console code ships to both hosts, packages install, per-host configuration is generated (each host keeps its own .env identity), the single database is adopted as db-1 and the replicas sync automatically, all services start, keepalived brings up the VIP on Host 1, and any already-enrolled compute nodes are re-pointed at the VIP.

After it finishes

  • Browse to the VIP — that is the console's address from now on. Update your DNS record to point at it.
  • The 🧬 HA Cluster widget appears bottom-left in the console, showing each host and its database role.
  • The Enable HA menu entry disappears permanently.
  • Install the HA Manager tooling for day-2 operations (out-of-band status, logs, rolling updates, controlled failover).

If a step fails

There is no automatic rollback — a half-built Patroni/etcd cluster is riskier to auto-undo than to finish by hand. Instead:

  • The progress log states exactly which step succeeded and which failed.
  • The original single-node database volume is never deleted; worst case, the pre-HA console can be brought back by starting its original containers.
  • Fix the cause (usually SSH credentials, firewall, or a taken VIP address) and re-run, or finish the remaining steps manually.

Troubleshooting

Symptom Check
VIP never appears journalctl -u keepalived on Host 1 — look for VRRP state transitions; confirm no other device answers the VIP address (arping <vip>).
A replica won't sync curl http://<host>:8008/patroni on each host — roles should be one primary + two replica.
Console intermittently errors after failover Give Patroni ~10 s to promote; the app replaces broken pooled DB connections automatically on the next request.
Which host owns the VIP right now? The HA Manager header shows the live holder, or ip -4 addr show | grep <vip> per host.