Skip to content

Console Host Sizing

How much machine the console needs — the VM or physical server that runs MFConsole itself, not the hypervisor nodes it manages. Four tiers, and the .env settings that go with each.

The console is an all-in-one node

One box runs the whole control plane: Caddy, the app, the Celery worker, Celery beat, Redis, HAProxy, etcd and PostgreSQL (TimescaleDB) — plus the HA Manager as a host-level systemd service.

That co-tenancy is why the Postgres numbers below look small for the RAM total. A dedicated database host conventionally gets ~25% of RAM in shared_buffers; here Postgres shares the box with everything else, so it gets ~12–15%. The rest of the machine still has work to do.

Tiers

Tiny Small Medium Large
Target lab / PoC ≤ 150 VMs ≤ 750 VMs 750+ VMs
vCPU 2 4 8 16
RAM 8 GB 16 GB 32 GB 64 GB
/var 60 GB 100 GB 200 GB 400 GB
shared_buffers 512 MB 2 GB 4 GB 8 GB
effective_cache_size 3 GB 8 GB 20 GB 44 GB
work_mem 4 MB 8 MB 12 MB 16 MB
max_connections 100 200 400 600
DB_POOL_MAX / replica 10 25 50 80
asyncpg max_size 5 10 20 30
Celery concurrency 2 4 8 16

Pick by managed-VM count, not by host count. The console's load is driven by telemetry rows and task volume, both of which scale with VMs and containers, not with how many hypervisors those VMs are spread across.

In HA, these are per host

A 3-node HA console runs the full stack on every host — all three are active. Size each of the three hosts at the tier you picked; do not divide the tier by three. See Connection budget for the one number that HA does change.

Applying a tier

Everything except Celery concurrency and the asyncpg pool is set in .env and picked up on the next docker compose up -d. None of these keys are in .env.example — add the block to the end of the file.

# 2 vCPU / 8 GB — lab, PoC, demo
DB_SHARED_BUFFERS=512MB
DB_EFFECTIVE_CACHE_SIZE=3GB
DB_WORK_MEM=4MB
DB_MAINT_WORK_MEM=128MB
DB_MAX_CONNECTIONS=100
DB_SHM_SIZE=1gb
DB_POOL_MIN=2
DB_POOL_MAX=10

# 4 vCPU / 16 GB — up to ~150 VMs
DB_SHARED_BUFFERS=2GB
DB_EFFECTIVE_CACHE_SIZE=8GB
DB_WORK_MEM=8MB
DB_MAINT_WORK_MEM=256MB
DB_MAX_CONNECTIONS=200
DB_SHM_SIZE=2gb
DB_POOL_MIN=3
DB_POOL_MAX=25

# 8 vCPU / 32 GB — up to ~750 VMs
DB_SHARED_BUFFERS=4GB
DB_EFFECTIVE_CACHE_SIZE=20GB
DB_WORK_MEM=12MB
DB_MAINT_WORK_MEM=512MB
DB_MAX_CONNECTIONS=400
DB_SHM_SIZE=5gb
DB_POOL_MIN=5
DB_POOL_MAX=50

# 16 vCPU / 64 GB — 750+ VMs
DB_SHARED_BUFFERS=8GB
DB_EFFECTIVE_CACHE_SIZE=44GB
DB_WORK_MEM=16MB
DB_MAINT_WORK_MEM=1GB
DB_MAX_CONNECTIONS=600
DB_SHM_SIZE=9gb
DB_POOL_MIN=5
DB_POOL_MAX=80

DB_SHM_SIZE must exceed shared_buffers

PostgreSQL allocates shared_buffers out of /dev/shm, and a container's /dev/shm defaults to 64 MB. The shipped compose file raises it to 2gb, which covers the default shared_buffers=2GB and nothing beyond it.

Raise DB_SHARED_BUFFERS above 2 GB without raising DB_SHM_SIZE and Postgres will not start — it dies with could not create shared memory segment, and every worker then reports Connection refused, which looks like a broker problem rather than a memory-sizing one. The Medium and Large blocks above already account for this.

The two knobs that are not in .env

Setting Where it lives How to change it
Celery concurrency docker-compose.yml, the worker service's command: — hardcoded --concurrency=4 Edit the line to --concurrency=<tier>, then docker compose up -d worker.
asyncpg max_size core/database.py, init_async_db() — hardcoded min_size=2, max_size=5 Requires a code change; ships as 2/5 for every tier today.

The asyncpg pool serves only the console's async def routes and is small by design, so leaving it at the shipped 2/5 is safe well past the Small tier. Treat the asyncpg row in the tier table as a target for a future release rather than something you can set today.

Worker concurrency is not a CPU count

Do not raise Celery concurrency above the tier value just because the box has cores. This workload is I/O-bound orchestration — SSH and virsh calls out to hypervisor nodes, waiting on the network — not computation. Celery's default is one prefork child per CPU, and each child opens its own database pool, so an unbounded concurrency multiplies the connection footprint without adding throughput.

Connection budget

Each process that touches the database keeps its own psycopg2 pool, and each Celery prefork child counts as a process. The ceiling for a single-node console is:

peak connections = DB_POOL_MAX x (1 app + 1 beat + N worker children)
                 + asyncpg max_size

DB_POOL_MIN connections per process are opened eagerly at startup; the pool grows toward DB_POOL_MAX only under real demand.

Tier Eager floor Arithmetic ceiling max_connections
Tiny ~22 45 100
Small ~32 160 200
Medium ~52 520 400
Large ~92 1,470 600

At Medium and Large the ceiling exceeds max_connections on paper. In practice it is not reached: DB_POOL_MAX is a per-process ceiling, not a reservation, and a prefork child executes one task at a time, so it holds a handful of connections rather than fifty. The tiers are sized against realistic steady state, not against the arithmetic worst case.

What to do if you do hit FATAL: sorry, too many clients already:

  1. Lower DB_POOL_MAX first — halve it. It costs nothing; the pool was never using that headroom.
  2. Only then consider raising DB_MAX_CONNECTIONS, and remember that each connection also carries up to work_mem per sort node. At the Large tier, 600 × 16 MB is ~9.6 GB of worst-case sort memory sitting outside shared_buffers.

HA multiplies the app-side pools, not the database

All three HA hosts run their own app, worker and beat — but they all connect to the one Patroni primary. The ceiling above is therefore roughly 3× on an HA cluster while max_connections stays where you set it. This is exactly the failure that drove the 2026-07-10 pool right-sizing. If you run HA at Medium or Large, start DB_POOL_MAX at half the tier value and raise it only if you measure contention.

Sizing /var

/var carries the container images (~4 GB), the PostgreSQL data volume, and local backups. The database is the part that grows, and telemetry dominates it:

telemetry rows/day = (VMs + containers) x 3 samples/min x 1,440 min

The telemetry sampler runs on a 20-second beat, so 750 VMs produce roughly 3.2 million rows/day. Retention is what bounds the total:

Table Retention Why
vm_telemetry, container_telemetry, host_telemetry, datastore_telemetry, zfs_telemetry 30 days Chart history for the Monitor tabs.
vm_power_samples 400 days Backs usage reports and billing periods — needs to span a full year of invoicing.

The tier's /var figure allows for that steady state plus headroom for image pulls during an update and a local backup. Use NVMe or SSD — the shipped Postgres config sets random_page_cost=1.1 and effective_io_concurrency=200, both of which assume flash. On spinning disk those values make the planner choose badly.

VM or physical?

Either works. The console has no hardware requirements of its own — no GPU, no passthrough, no special NICs.

  • As a VM — the common choice. Do not host it on a hypervisor that the console itself manages and expect it to recover that hypervisor: if the node dies, so does the tool you would use to fix it. Put it on separate infrastructure, or run the 3-node HA cluster with the hosts spread across failure domains.
  • On physical hardware — a single small 1U server comfortably covers the Large tier. Worth it when the console must survive the loss of any managed hypervisor, which is the whole point of an out-of-band control plane.
  • Thin-provisioned storage is fine for /var, but the database will eventually write into everything the retention window allows. Make sure the backing store can actually deliver the full tier figure.

See also