Skip to content

Architecture Overview

MFCloud is a control plane plus an agent. The control plane (the master) runs the web console, the REST API, the task queue, and the database. Agents run on every compute / storage / LXD node and execute work the master schedules for them.

High-level data flow

            ┌────────────────────────────────────────────────────────┐
            │                     OPERATOR BROWSER                   │
            └────────────────────────────┬───────────────────────────┘
                                         │  HTTPS (Caddy TLS)
                                         ▼
┌────────────────────────────────────────────────────────────────────┐
│                            MFCLOUD MASTER                          │
│                                                                    │
│   ┌─────────┐    ┌──────────────┐    ┌────────────┐    ┌────────┐  │
│   │ Caddy   │───▶│  FastAPI app │───▶│  Celery    │───▶│ Redis  │  │
│   │ (TLS)   │    │  (Uvicorn)   │    │  Beat      │    │ broker │  │
│   └─────────┘    └──────┬───────┘    └─────┬──────┘    └────┬───┘  │
│                         │ async              │                │    │
│                         ▼                    │                │    │
│                ┌─────────────────┐           │                │    │
│                │  TimescaleDB    │           │                │    │
│                │  (Patroni HA)   │           │                │    │
│                └─────────────────┘           │                │    │
└──────────────────────────────────────────────┼────────────────┼────┘
                                               │ per-host queue │
                                               ▼                ▼
            ┌─────────────────────────────────────────────────────┐
            │                COMPUTE / STORAGE NODES               │
            │                                                     │
            │   ┌────────────┐  ┌──────────────┐  ┌───────────┐   │
            │   │ mfcloud-   │  │ libvirt /    │  │ LXD /     │   │
            │   │ agent (TLS)│  │ KVM          │  │ Incus     │   │
            │   └─────┬──────┘  └──────────────┘  └───────────┘   │
            │         │                                           │
            │         ▼                                           │
            │   ┌──────────────────────────────────────────┐      │
            │   │ Ceph OSDs / LINSTOR + DRBD / ZFS / NFS   │      │
            │   └──────────────────────────────────────────┘      │
            └─────────────────────────────────────────────────────┘

Master components

Component What it does
Caddy TLS termination, HTTP/2, reverse-proxy to the FastAPI app, also serves the KB at /kb/.
FastAPI Web UI + JSON REST API. Sync routes use psycopg2; async routes asyncpg.
Celery worker Master-side queue (celery) for tasks that run inside the master container.
Celery beat Cron-style scheduler for periodic tasks (HA monitor, telemetry rollup, etc.).
Redis Celery broker.
TimescaleDB Persistent state. Patroni provides HA failover; HAProxy routes writes to the primary.

Node components

Component What it does
mfcloud-agent Per-host Celery worker subscribed to compute-<ip-with-dashes>. Runs the per-host tasks (VM create, snapshot, backup, etc.).
Emergency Host UI TLS web UI on 8443 — runs even when the master is down so an operator can reach the node directly.
libvirt / QEMU KVM virtualization.
LXD / Incus Container backend (optional — only enrolled nodes).

Auth flow

Operator login (POST /api/login)
  ├─ Try local users.password (passlib bcrypt)        ← ALWAYS first
  ├─ If miss + LDAP enabled → bind to AD as service account
  │     → bind as user → auto-mirror with auth_provider='ad'
  └─ Issues JWT (24 h, signed with SECRET_KEY)

Operator login (SSO — /api/auth/oidc/start)
  ├─ Generate state, nonce, PKCE S256
  ├─ Sign into 10-min HttpOnly cookie
  ├─ 302 → IdP authorize endpoint
  ├─ /callback: validate cookie + state + PKCE
  ├─ Exchange code → ID token, verify against IdP JWKS
  └─ Auto-mirror with auth_provider='oidc' → issue JWT

All routes (REST + WebSocket) accept the JWT in Authorization: Bearer <token> or, for WebSockets, in Sec-WebSocket-Protocol.

Storage routing

A VM disk is just a libvirt-managed file or block device. The backend decision is made at VM-create time:

  • LINSTOR / DRBD → block device backed by /dev/drbd<N>, replicated synchronously across 2–3 nodes.
  • Ceph RBD → librbd-backed block device, accessible from any node in the cluster.
  • ZFS → qcow2 file at /<pool>/<vm>.qcow2 on the node's local pool; not live-migratable, but can be asynchronously replicated to another host in the same cluster and restored there if the source goes down.
  • NFS → qcow2 file on the mounted share; migratable but limited by NFS performance.

The Repair Agent flow pushes core/lxd.py, core/ssh.py, and the entire tasks/ directory to a node so any update to those (e.g. a fingerprint-pinning patch) takes effect without re-deploying the OS image.

See also