Skip to content

Changelog

The same release notes shown in the console's What's New prompt (bell icon, top nav), in full and in order.

MFCloud Console — Changelog

All notable changes to the MFCloud console, curated by release. Dates are the completion date of each version. This is a distilled highlight list — the full per-item engineering history lives in patch_note/RAOD_MAP_SHIPPED.txt.

Format loosely follows Keep a Changelog. Versions are MFCloud product versions (the APP_VERSION baked into each build).

Versioning scheme (from 0.40.0-beta on): semver, pre-1.0 — 0.<minor>.<patch>-<stage>. The 0.x major signals the product hasn't hit a stable 1.0 yet; <minor> increments per release the way the old V<n> counter did. Older headers below (V39, V38, …) predate this and are left as originally written.

0.45.5.3 — 2026-09-08

Found while investigating why every compute node on the fleet dashboard showed a "needs repair" badge.

Fixed

  • The automatic fleet worker-sync never recorded the version it just pushed, so SMB-7's version-compatibility badge could never clear on a node that was only ever auto-synced. core/background.py:: _sync_and_restart_worker (runs against every registered compute node each time the console restarts and becomes leader) pushes fresh code, patches the broker URL, and reconciles the TLS pin — but never called routers/hosts/repair.py::_patch_worker_version_on_node, the step that writes WORKER_CODE_VERSION into /etc/mfcloud-worker.env so tasks/app.py's heartbeat can report it. The manual "Repair Agent" button has always called it; the automatic path simply never did. Result: a node repaired only by the automatic sync ran fully current code but reported an empty version forever, permanently flagged "needs repair" regardless of how up to date it actually was — confirmed live on KVM-CEPH-02, whose worker restarted with fresh code today but had no WORKER_CODE_VERSION line in its env file at all. Now wired in the same place the manual path already has it.

0.45.5.2 — 2026-09-08

Found by a post-deploy fleet health check on the live 3-node control plane, right after 0.45.5.1 went out.

Fixed

  • The Ceph health monitor misreported a healthy Quincy cluster as an unreachable/failed poll on every 60-second cycle. core/background.py::_poll_ceph_clusters_once read cluster health only from summary['health_data']['health']['status'], but Ceph 17.2.8 (Quincy)'s /api/summary endpoint exposes a flat health_status string instead of that nested shape — the same version difference routers/ceph/core.py::get_ceph_telemetry already accounts for. The cluster itself was HEALTH_OK the entire time; the poller just couldn't read it, which would have silently swallowed a real HEALTH_WARN/ HEALTH_ERR transition too, since a failed poll is deliberately never treated as a state change. Now checks the flat key first, falling back to both nested shapes, matching the precedence already proven correct elsewhere in the codebase.

0.45.5.1 — 2026-09-08

Caught in code review of 0.45.5.0's new AI Assistant features before that release saw any wider use.

Fixed

  • Chat compaction could split a tool_use/tool_result pair and permanently break a session. core/chat_compaction.py::compact_if_needed cut the message history at a fixed 8-entry boundary with no awareness of tool_use/tool_result pairing. A session with recent tool-call turns could have that boundary land between a tool_use-bearing assistant message and its matching tool_result — summarizing away the former while keeping the latter, which the Anthropic SDK then rejects on the very next turn, and every turn after it, since the message list had already been mutated in place. The cut point now walks backward past any message that starts with a tool_result block so it can never split a pair.
  • A message sent while compaction was summarizing could vanish silently. The same function snapshotted the messages to keep (kept = messages[-8:]) before awaiting the summarization call, then unconditionally overwrote the whole list with [summary] + kept afterward. routers/chat.py's Approve/Cancel handlers append to messages without turn_lock by design, so an engineer clicking Approve/Cancel while compaction's summarization call was in flight had that message wiped out with no error on either side. Compaction now re-slices the kept tail from the live list after the await instead of trusting the pre-await snapshot.
  • "Explain this task" bypassed the new AI usage ceiling and usage log. routers/chat.py::explain_task falls back to the Anthropic backend when Ollama/llamacpp aren't configured, but never called ai_usage.check_ceiling/record_usage the way the main chat turn does — a tenant who'd hit their monthly spend ceiling in the chat panel could still generate unlimited, unlogged, unbilled-against-cap Anthropic calls from task-row "Explain" buttons. Now routed through the same check/record pair.
  • The 0.45.5.0 GPU-reservation release fix could clear a different VM's live assignment. tasks/core.py::_release_gpu_reservation (added this same version to stop a failed create_vm from permanently stranding a reserved GPU) cleared gpu_devices.assigned_vm/assigned_kind unconditionally on failure, without checking the reservation still belonged to the VM that failed. If create_vm failed late and, in that window, the same PCI device was legitimately reassigned to a different VM or container, this cleanup wiped the new owner's assignment instead of only its own stale one. Now a compare-and-clear scoped to the failing VM's own name.

0.45.5.0 — 2026-09-08

Added

  • Per-tenant AI Assistant usage tracking, and an optional monthly Anthropic spend ceiling. routers/chat.py::_run_turn already computed the token totals for every turn's WS done envelope; they were never kept anywhere. New ai_usage_log table persists one row per completed turn (tenant, backend, tokens, cost). A new ai_usage_ceiling_cents setting (Settings -> AI Assistant, default 0 = disabled) refuses a turn before it starts once a tenant's monthly Anthropic spend — computed at Anthropic's real published rate for claude-opus-5, $5/M input tokens / $25/M output tokens, checked live rather than estimated — reaches the configured limit, reusing the same refusal-envelope shape already used for a disabled backend. Ollama/llamacpp usage is logged too (volume visibility), never gated — neither has a real per-token dollar cost. New GET /api/chat/usage lets a tenant see their own running total against the ceiling.
  • Conversation-history compaction for the AI Assistant chat panel. messages in ws_chat was only ever appended to for the life of a session — every turn resent the full history, tool results included, with no context-window guard anywhere. New core/chat_compaction.py summarizes everything but the last few exchanges in a one-shot, no-tools model call (the same shape explain_task already uses for a different reason) once the previous turn's reported input-token count crosses a configurable threshold (ai_context_compact_threshold_tokens, default 8000 — a conservative placeholder pending the real --ctx-size configured on the llamacpp backends, not a measured number). The summarization call's own tokens are logged through the usage tracking above, so a long session that triggers several compactions doesn't under-count.
  • Dual-stack (IPv6) primitives in the OVN wrapper. core/ovn.py's lr_connect_ls/lsp_add (and their _ensure variants) now accept an optional IPv6 CIDR/address alongside the existing IPv4 one — OVN's own CLI already supports multiple addresses per port, this wrapper just never used that. Validated with Python's ipaddress stdlib rather than extending the existing hand-rolled IPv4-only regex. New lrp_set_ipv6_ra() turns on Router Advertisement/SLAAC on a dual-stack port. Every existing IPv4-only caller is unaffected — the new parameters default empty. Foundation only: not yet wired into any real tenant-network provisioning path, and lrp_set_ipv6_ra specifically hasn't been live-tested against this deployment's actual OVN version.

Fixed

  • A failed VM create could permanently strand a reserved GPU. tasks/core.py::create_vm's three internal failure paths (template missing, VmBuildError, any other exception) now release a find_available_gpu()/admin-picked GPU reservation before returning error. Previously only routers/iac.py's synchronous exception handler did this — a failure inside the async Celery task itself (the common case) left gpu_devices.assigned_vm stuck pointing at a VM that was never created, recoverable only by retrying under the exact same VM name (the expected_owner reclaim path). Added a target_host passthrough kwarg, threaded through all three create_vm.apply_async call sites, so the task can resolve which host's reservation to release. New admin escape hatch for anything this doesn't catch (a worker killed mid-task, not just a normal failure): DELETE /api/hosts/{ip}/gpus/{pci_address}/release.

0.45.4.1 — 2026-09-06

Caught immediately after deploying 0.45.4.0 — verified live against the real fleet before this saw any wider use.

Fixed

  • SMB-7's version-compatibility gate never actually detected anything on a real deployment. It compared each node's reported version against a celery-named row that never exists in practice — CONTROL_PLANE_IP is set on the control plane's own .env too (there, it's typically the VIP), so the control plane's own heartbeat registers itself as just another compute host (e.g. compute-<vip>) for queue-routing purposes; queue_name never actually resolves to 'celery' on a real 3-node deployment. Fixed: compare directly against this process's own APP_VERSION instead of trying to look up a baseline row that isn't there, and use NODE_IP's presence (only ever set on a real compute node, never a control-plane host) rather than queue name to tell the two apart in tasks/app.py's heartbeat.

0.45.4.0 — 2026-09-05

Added

  • SMB-7: console↔node-worker version-compatibility gate, and "this node needs Repair Worker" surfacing. There was no version tracking on compute nodes at all until now — an operator had to infer from release notes whether a node needed a Repair Worker pass. Every worker (control plane and compute alike) now reports its own code version on its existing 30-second heartbeat; a compute node's version is written to /etc/mfcloud-worker.env at the moment code is actually pushed (full provision or Repair Worker), not merely requested. The Hosts table now shows a needs repair badge — click it to run Repair Worker directly — on any node whose reported version doesn't match the console's; GET /queues carries the same flag for anything polling it directly. Every already-enrolled node will show this badge once, harmlessly, until its first Repair Worker run under this release — it has never reported a version before now, and an unknown version is treated as a mismatch rather than silently assumed current.

0.45.3.7 — 2026-09-05

Caught in review before shipping 0.45.3.6's haman restore panel any further.

Fixed

  • haman's new control-plane restore route hardcoded -f compose.dev.yml, which would abort the app/worker/beat stop/start on any host that doesn't have that file — every registry-pull customer (pack.sh never ships it) and any not-yet-migrated cluster whose docker-compose.yml still carries build: inline. This is the exact regression _run_update's own comment already documents fixing once — reintroduced here in the new code path instead of reusing that resolution. Fixed: the stop/start loops now call a shared _compose_files_for() helper (dev-override / prod-override / neither, same 3-way check _run_update already does) instead of a hardcoded flag. The internal 3-host control plane happened to have compose.dev.yml present, so this was invisible there — caught only by re-reading the new code against _run_update's own documented incident before pushing to Docker Hub.

0.45.3.6 — 2026-09-05

Added

  • ENT-5 restore is now reachable from haman (the out-of-band HA Manager, :9444 on all 3 hosts) — a new "Control-Plane Backup / Restore" panel lists backups and can trigger a full restore, out-of-band like the rest of that UI (works even with the console app down). scripts/ cp_restore.py gained list --json, get <id>, and a restore <id> subcommand (the DB-side restore only — TimescaleDB pre_restore -> pg_restore -> post_restore against DB_HOST); haman.py owns stopping/ starting app/worker/beat on all 3 hosts around it, since that needs the hosts' own root SSH mesh, not something a container can do.

Fixed

  • scripts/cp_restore.py download's TOC-validation failure was a warning, not a hard failure — a caller acting on exit code (like the new haman integration above) could have proceeded past a corrupted download. Now exits 1.

0.45.3.5 — 2026-09-05

Found live running ENT-5's first real backup against a working S3 target (0.45.3.4 fixed the connection; this is what the actual restore then uncovered).

Fixed

  • Every ENT-5 restore-verification failed — and a real DR restore following the runbook as written would have failed too. The console DB is TimescaleDB, not plain Postgres. pg_restore into a freshly created database has no TimescaleDB extension loaded yet, so every hypertable chunk's COPY (vm_telemetry/host_telemetry/ continuous_agg, core/schema.py) failed with could not find hypertable with id N — the backup and upload halves worked perfectly, only the restore half was broken. Fix: wrap the restore in TimescaleDB's own documented timescaledb_pre_restore()/timescaledb_post_restore() calls, on a database that already has CREATE EXTENSION timescaledb applied. Fixed in tasks/cp_backup.py's nightly verification, deployment/DR_RUNBOOK.md's documented restore procedure, and scripts/cp_restore.py's printed restore commands — all three now agree. Also: the verification step now captures and surfaces pg_restore's actual stderr on failure instead of a bare "exit status 1", which is what made this diagnosable at all.

0.45.3.4 — 2026-09-05

Found live while testing ENT-5 against a real Backblaze B2 bucket.

Fixed

  • S3 cloud backups (VM backups and ENT-5's control-plane backup) hung and failed against Backblaze B2 — every upload threw ConnectionClosedError. botocore's default Expect: 100-continue header, plus its newer "aws-chunked" trailer-checksum upload encoding, both desync the connection mid-response against B2's S3-compatible API. Confirmed with a wire-level trace: dropping Expect and reverting to a plain Content-Length body (request_checksum_calculation/ response_checksum_validation = when_required) fixes it immediately — AWS and MinIO are unaffected either way. Fixed in tasks/cloud_backup.py's shared _s3_client(); routers/system/ settings.py's /settings/s3/test route built its own separate client and now reuses the same fix instead of drifting further from it.

0.45.3.3 — 2026-09-05

mfconsole-ship packaging gap closed: the public Docker Hub image was missing two pieces of ENT-5.

Fixed

  • The public image shipped with no pg_dump/pg_restore — nightly control-plane backups silently failed on every registry-pull deployment. 0.45.3.0 added postgresql to mfcloud-master/Dockerfile for tasks/cp_backup.py, but the separate, Cython-compiling mfconsole-ship/Dockerfile.ship that builds the public image was never updated to match — every customer on 0.45.3.0–0.45.3.2 had a worker container with no pg_dump binary at all. Fixed: Dockerfile.ship's runtime stage now installs postgresql too.

  • The manual DR restore helper, scripts/cp_restore.py, was never packaged into the public image at all. pack.sh's allowlist shipped deployment/DR_RUNBOOK.md (which tells operators to run this script) and tasks/, but never scripts/. Fixed: pack.sh now ships that one file specifically — the rest of scripts/ (CI/dev tooling) stays out.

Verified locally: a real pack.sh + docker build -f Dockerfile.ship + running the built image confirmed both fixes. Image not yet rebuilt/pushed to Docker Hub. The 3-node control plane builds from mfcloud-master's own, unaffected Dockerfile and needed no redeploy for this fix.

0.45.3.2 — 2026-09-05

Critical fleet-wide regression from 0.45.3.0. Every enrolled compute node's mfcloud-worker service crash-looped from the moment it picked up 0.45.3.0 (2026-09-04 ~07:46 UTC) until this fix — no VM/container create, power, migration, or any other node-dispatched task could run anywhere on the fleet, silently: task history showed new work stuck at "Queued" forever with no error surfaced. /api/system/queues showed every compute-* queue as Offline.

Fixed

  • tasks/cp_backup.py imported core.cluster_log at module level. core/cluster_log.py is not in WORKER_CORE_MODULES (core/config.py) — it's never pushed to compute nodes, by design. tasks/__init__.py unconditionally imports backup_control_plane from cp_backup.py at package-import time, and every compute node's Celery worker must import the tasks package just to start (celery -A tasks.app:celery_app) — regardless of which queue it consumes. The result: ModuleNotFoundError: No module named 'core.cluster_log' killed the worker process on every node, immediately, in a systemd auto-restart crash loop. The control plane's own worker containers were unaffected (they run the full app image, where core/cluster_log.py is present) — only real compute nodes, which run the stripped node-side bundle, were hit. The two audit_admin_action/ log_cluster_event calls this module needs are now imported inside backup_control_plane() itself, which only ever runs on the control plane's own celery queue — never on a node — so the import is never reached there. Caught live: a fleet-wide /api/system/queues check found every node offline since the exact minute 0.45.3.0 shipped.

0.45.3.1 — 2026-09-04

Found live during a post-deploy cluster health check: the LXC Marketplace catalog was failing to refresh on all 3 CP hosts.

Fixed

  • routers/containers/images.py / tasks/lxc/marketplace.py: TurnKey mirror requests now send a real User-Agent. GET /marketplace/catalog was throwing TurnKey mirror unreachable: HTTP Error 403: Forbidden on every refresh — Cloudflare (fronting mirror.turnkeylinux.org) silently 403s the bare Python-urllib/x.y User-Agent urllib sends by default, while a real browser or curl gets 200 from the same host at the same time. Confirmed by reproducing the exact failing request live and isolating it to the UA string, not an IP block or an actual mirror outage. All three urlopen() call sites against the mirror (the catalog scrape, the live-version discovery probe, and the actual appliance download) now send a browser-like UA via a shared _TURNKEY_HEADERS constant — the TurnKey appliance install path was silently broken by the same root cause, not just the catalog refresh the user actually saw.

0.45.3.0 — 2026-09-04

Opens the 1.0 GA cut line (patch_note/roadmap/ROADMAP_1.0_GA.md, scope locked 2026-08-24): full audit-trail coverage for identity/tenancy mutations, and the first real control-plane disaster-recovery mechanism.

Security

  • ENT-2b: every identity/tenancy mutation is now audited. core/cluster_log.py's log_cluster_event() was only ever called from the login path and task completions — user creation, role grants, and tenant changes were invisible to the audit trail; "who granted this person Super Admin, and when" was unanswerable. New audit_admin_action() helper, called after commit at all 13 mutating routes across routers/users.py (add/delete user, generate/revoke API key) and routers/iam.py (tenant create/update/delete/provision, password reset, role create/update/delete, role assignment). Enforced going forward by scripts/check_audit_coverage.py, an AST-based CI gate (.gitlab-ci.yml's security:project-rules job) that fails the pipeline if a future mutating route in those two files ships with no audit call.

Added

  • ENT-5: automated, verified control-plane backups. tasks/cp_backup.py runs nightly (leader-elected — the same cluster-wide-singleton guarantee core/leader.py already gives the VM/container backup crons, not Celery beat, whose per-host singleton has a known duplicate-firing mode): pg_dumps the console database and tars the non-DB state (.env, ssh_keys/, tls.d/, tls/), then encrypts and uploads both through the exact same MFB1/S3 machinery tasks/cloud_backup.py already uses for VM backups — reused, not reinvented. Every run also pg_restores the dump into a throwaway database on the live cluster and queries it — real restore verification, not a pg_dump-exit-code check. New cp_backups table, retention pruning, settings + history + on-demand trigger (routers/system/backup.py), and a manual restore helper (scripts/cp_restore.py) that downloads, decrypts, and validates a backup pair without ever auto-applying the restore itself. docker-compose.yml's worker service now also mounts .env/tls.d/ tls read-only (previously Caddy-only) so the task can see live config instead of a stale, dockerignored build-time copy; Dockerfile now installs postgresql for pg_dump/pg_restore. See deployment/DR_RUNBOOK.md for the restore procedure and stated RPO (~24h) — RTO is not yet measured against a live rehearsal, which remains open before this item is fully done.

0.45.2.3 — 2026-09-02

Security/code-review hardening pass on 0.45.2.2's WireGuard relay feature, found before any relay was ever provisioned against it in production.

Security

  • tasks/relays.py: WireGuard private keys no longer transit the Celery worker. provision_relay and enable_relay_chassis previously read cat /etc/wireguard/privatekey back over SSH into a Python variable, then wrote it back out over SFTP — crossing the wire twice instead of the documented zero times. Both wg0.conf writes now expand PrivateKey = $(cat /etc/wireguard/privatekey) in a remote shell command instead, so the key never leaves the box it was generated on, matching what core/schema.py's table comment already claimed.
  • routers/networks_relay.py: removing a chassis now revokes its WireGuard peer on the relay. Unpairing previously deleted the DB row and best-effort disabled the tunnel on the chassis side only — the relay's live wg0 config kept trusting that chassis's key indefinitely. A new tasks.relays.revoke_chassis_peer() runs wg set wg0 peer <key> remove + wg-quick save on unpair; a failed revocation now surfaces as an explicit warning in the API response and a toast in the UI instead of being swallowed.
  • routers/networks_relay.py: relay/chassis inventory now requires can_manage_networks. list_relays/get_relay previously accepted any valid JWT, exposing every relay's public IP, home-network tunnel IP, and WireGuard public key to any authenticated user on any tenant.
  • routers/system/config.py: /api/system/db-health is now loopback-only. The container healthcheck probe added in 0.45.2.2 had no auth and no rate limit, and echoed the raw DB exception (including host/ port) to the caller. It now rejects anything not from 127.0.0.1/::1 with a 403 and logs the real error server-side instead of returning it.
  • tasks/relays.py: relay SSH host-key pinning now fails closed on a DB error, mirroring core/ssh.py's _PinningPolicy (previously a DB blip during a relay connection surfaced as a raw, unlogged exception instead of a clear refusal) — and now imports core/ssh.py's fingerprint helper instead of duplicating it.
  • tasks/relays.py: firewall port rules now use shlex.quote(), closing a latent (not yet exploitable) gap against this repo's no-f-string-into-shell rule.

Fixed

  • tasks/relays.py: chassis tunnel-IP allocation no longer collides after a chassis is removed and re-added. _next_chassis_tunnel_ip allocated off a row count, so removing a middle chassis and pairing a new one could hand out an IP still held by a surviving chassis — the relay would then silently reassign that AllowedIPs route away from the original owner. It now allocates one past the highest already-assigned IP, backed by a new UNIQUE(relay_id, wg_tunnel_ip) constraint on relay_chassis.
  • tasks/relays.py: check_tunnel_health failures now update stored status. An SSH-level failure (box powered off, credentials rotated, port 22 blocked) was silently swallowed, leaving the Networks tab showing a stale "Active" status indefinitely instead of flipping to "Unreachable."
  • docker-compose.yml: app-1/2/3 healthcheck timeout widened 6s → 12s. The 0.45.2.2 healthcheck added a second sequential urlopen(timeout=4) probe without widening the CMD's own timeout, so elevated (not full- outage) latency could get the second probe killed mid-flight and flip a healthy container unhealthy.
  • routers/networks_relay.py: wg_tunnel_ip is now validated at relay creation. A malformed value previously passed the 200 response and only surfaced as a stuck status='Error' deep inside the Celery task.

0.45.2.2 — 2026-08-31

Turnkey WireGuard provisioning for the Home VPS Public IP Relay design (patch_note/design_artifacts/HOME_VPS_PUBLIC_IP_RELAY_DESIGN.html) — MFConsole now SSHes into the relay VPS and the home chassis itself and pairs their WireGuard keys automatically, instead of an admin running shell scripts and copy-pasting public keys by hand.

Added

  • New relays/relay_chassis tables (core/schema.py) — a relay VPS's SSH creds (encrypted via tasks.crypto.encrypt_secret, same as proxmox_connections) and WireGuard identity, plus one row per home chassis paired to it. The WireGuard private key never leaves the box it was generated on; only the public key is read back and stored.
  • tasks/relays.py: provision_relay, enable_relay_chassis, check_tunnel_health — three Celery tasks that port deployment/wireguard-relay-server-setup.sh and deployment/wireguard-home-client-setup.sh to run over Paramiko instead of a human's shell session, and collapse the manual key-exchange steps from deployment/HOME_VPS_WIREGUARD_RELAY.md into one task (enable_relay_chassis generates the chassis's keypair, brings up its tunnel, then hot-adds it as a peer on the relay via wg set + wg-quick save). check_tunnel_health is beat-scheduled at the same 60s tier as ha-monitor-heartbeat and only writes a task_history row on a status transition, not every tick.
  • routers/networks_relay.py — /api/networks/relays CRUD plus POST .../{id}/chassis to pair an existing compute node, gated by the same can_manage_networks permission every other OVN/IPAM router already uses.
  • Networks tab: "Public IP Relays" panel (static/js/networks/relays.js) — Add Relay form, per-relay chassis pairing, and an expandable row showing each chassis's tunnel status and last-handshake age.
  • deployment/wireguard-relay-server-setup.sh, deployment/wireguard-home-client-setup.sh, deployment/HOME_VPS_WIREGUARD_RELAY.md — the manual, run-by-hand path for standing up a relay/chassis outside MFConsole. The Celery tasks above port this same logic to run automatically.

Scope

Covers build-order steps 1-2 of the design doc only (stand up the relay, tunnel the chassis in) — turnkey'd. Deliberately does not touch OVN bridge-mapping, the Failover-IP pool, or the "Assign Public IP" action: bridging a WireGuard (L3-only) interface straight into an OVS bridge doesn't carry VM traffic (ovs-vsctl add-port <bridge> wg0 would silently pass nothing), so that piece needs its own design pass — likely VXLAN-over- WireGuard, the pattern Proxmox's own SDN stack uses for the same problem — before tasks/ovn.py::bootstrap_ovn_chassis grows a relay-chassis variant.

Not yet live-tested end to end — no relay VPS has been provisioned yet to exercise provision_relay against.

0.45.2.1 — 2026-08-31

Fresh installs (empty database, no prior lxd_containers table) crashed on every startup with psycopg2.errors.UndefinedTable: relation "lxd_containers" does not exist — including the current mfconsole/kvm-manager:latest image on Docker Hub.

Fixed

  • core/schema.py: container-CRS's lxd_containers.crs_last_migrated_at column-add ran ~194 lines before CREATE TABLE IF NOT EXISTS lxd_containers itself, a migration-ordering bug introduced with the container CRS (LXD-W5) work. Existing deployments were unaffected (their lxd_containers table already existed), but any brand-new database hit this on first boot and never came up. Moved the column-add to after the table creation, alongside lxd_containers's other add_column_if_not_exists calls.
  • core/schema.py was also missing CREATE TABLE IF NOT EXISTS for the Home VPS Relay feature's relays/relay_chassis tables — the same class of fresh-install gap: the feature's routers/tasks worked wherever the tables already existed by hand, but a brand-new database never got them. Added alongside the other CREATE TABLE statements.

0.45.2.0 — 2026-08-30

Every Windows VM deploy was failing outright on this fleet's Rocky Linux hypervisor nodes.

Fixed

  • tasks/vm/storage.py: Windows boot disk moved off the LSI SAS1068 (MPT Fusion) SCSI controller onto SATA (AHCI). The boot disk had unconditionally used an emulated lsisas1068 controller for Windows guests -- the same chip VMware defaults new Windows VMs to, chosen so Setup sees the disk with no "Load driver" step. But this fleet's Rocky/RHEL qemu-kvm build doesn't compile that legacy HBA in, so dom.create() failed immediately with unsupported configuration: This QEMU doesn't support the LSI SAS1068 (MPT Fusion) controller, and the VM row got auto-rolled-back -- on every single Windows deploy, not an edge case. SATA needs no explicit <controller> (q35, already forced for Windows guests, gets one from libvirt by default) and Windows has shipped an inbox AHCI driver since Vista too, so the original "no driver-load step" goal is unaffected.
  • tasks/settings/controllers.py / routers/vms/settings.py / models/schemas.py / modals_hardware.html: dropped lsisas1068 from the manual "Add SCSI controller" hardware option, for the same reason -- it was offering a controller model this fleet's QEMU can't actually attach.

0.45.1.13 — 2026-08-30

Intel GPU support for the Host Monitor tab's GPU telemetry (0.45.1.12 was NVIDIA-only) -- confirmed live against real fleet hardware.

Added

  • tasks/telemetry.py: driver-agnostic Intel GPU probe (_INTEL_GPU_PROBE) as a fallback when _host_gpu_stats() finds no NVIDIA GPU. Reads /proc/<pid>/fdinfo/<fd>'s drm-* counters -- the same cross-vendor DRM interface intel_gpu_top/nvtop read internally -- rather than shelling out to intel_gpu_top itself, which turned out not to be packaged for this fleet's Rocky Linux 10 hosts (checked BaseOS/AppStream/CRB/EPEL/ oneAPI live on MF-AI: no match). Two /proc scans ~0.5s apart turn each DRM engine's cumulative busy/total cycle counters into a %; the busiest engine stands in for "GPU utilization" the way nvidia-smi's single utilization.gpu figure does. VRAM total comes from the same Resizable-BAR heuristic tasks/gpu.py's inventory scan already trusts (largest 64-bit prefetchable BAR) -- reliable here because this probe only ever matches a host-driven card (a passthrough-bound GPU has no drm-driver: xe/i915 fdinfo entries and correctly falls through as "no GPU", same as before).
  • Why this needed a live diagnosis, not just code: after 0.45.1.12 shipped, both of MF-AI's Arc Pro B50 and ROCKY-VDI's Tesla P4 showed "No GPU detected" despite genuinely having GPUs. Confirmed via direct SSH + the app's own GPU inventory table that these are two different, both-expected cases: MF-AI's Arc B50 is driven directly by the host (for the llama.cpp/SYCL inference backend from 0.45.1.11) and just needed Intel support, added here; ROCKY-VDI's Tesla P4 is mode='passthrough', assigned_vm='ROCKY-VDI' in gpu_devices -- it's bound to vfio-pci and invisible to the host under either vendor probe, which is correct behavior, not a gap this change closes (see _host_gpu_stats's docstring).
  • Live-verified on MF-AI: 0.0% util / 39.7% VRAM (a resident ~6.5GB model against the card's real 16GB), matching what the card is actually doing.

0.45.1.12 — 2026-08-30

GPU utilization/VRAM added to the Host Monitor tab's telemetry.

Added

  • Host-level GPU % and VRAM % charts in the Monitor tab (static/js/hosts.js, static/js/telemetry/chart-theme.js), sampled the same way CPU/RAM/IO Wait already are: a new _host_gpu_stats() probe in tasks/telemetry.py runs nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total over the existing SSH connection factory (subprocess for the console's own localhost row), in its own parallel thread pool alongside the libvirt snapshot so it adds no sequential latency to the 20s beat cycle. Persisted to two new nullable columns on host_telemetry (core/schema.py: gpu_pct, gpu_mem_pct -- NULL means "no GPU here", kept distinct from a real 0% idle reading) and surfaced through /telemetry/history and /telemetry/rollup (routers/hosts/status.py). A host with no GPU (or no nvidia-smi) shows a clean "No GPU detected on this host" state instead of a flat zero line.
  • Deliberately host-level only, not per-VM: a GPU passthrough-assigned to a VM is bound to vfio-pci and invisible to the host's own nvidia-smi once attached -- monitoring that would need a guest-side agent, out of scope here. NVIDIA-only for now -- no vendor-neutral one-liner as cheap as nvidia-smi --format=csv exists for AMD/Intel. 20s cadence, not live -- same as IO Wait/Net/Disk (no sub-20s live tick, unlike CPU/RAM).

0.45.1.11 — 2026-08-30

New AI Assistant backend: llama.cpp (SYCL), alongside Claude and Ollama.

Added

  • core/llm_client.py: a third chat backend, llamacpp, speaking llama.cpp's OpenAI-compatible /v1/chat/completions API (new _call_llamacpp + _anthropic_messages_to_openai). Settings -> AI Assistant gained matching ai_assistant_llamacpp_enabled/host/model fields (routers/system/settings.py, static/js/admin.js, templates/components/dashboard/view_settings.html); the chat panel's backend picker (static/js/chat-panel.js) needed no changes -- it already renders whatever backends the server reports as available.
  • Why: MF-AI's ollama.service (the AI Assistant's long-standing backend, running the custom ornith15-mfcloud:9b model) was discovered live to be running CPU-only the entire time despite accessing /dev/dri/card0 directly -- measured 3.9 tok/s, consistent with a plain CPU decode, not the Intel Arc Pro B50 physically in that host. A from-source llama.cpp build (-DGGML_SYCL=ON, Intel's oneAPI DPC++ compiler) with the same model file (Ollama's own GGUF blob, /var/ollama/models/blobs/sha256-70c112..., no re-download or conversion needed) hit 32.3 tok/s on the same card -- confirmed with a real tool-call request (get_vm_power_state), matching this exact model's tool-calling behavior under Ollama with correct JSON-string arguments and a proper tool_calls finish reason.
  • Real SR-IOV VRAM gotcha found getting this onto the host itself (not a VM): this GPU has 16GB total VRAM; with sriov_numvfs=2 already set (for VM passthrough testing), the two VFs statically claim 8GB each -- all 16GB, leaving the PF (what a host-native service like Ollama/llama.cpp uses) with zero free VRAM. A host-side llama-server against the PF's own device (level_zero:gpu:0) crashed with a SYCL ggml_backend_sycl_buffer_clear exception, not a clean OOM message. Fix: an SR-IOV VF not currently attached to a running VM stays bound to the host's own xe driver and exposes its own real render node (/dev/dri/card1/renderD129 here) -- pointing ONEAPI_DEVICE_SELECTOR at that VF (level_zero:1) gets a real, unshared 8GB slice to run on, with zero impact on any VM using a different VF. Deployed as llamacpp-console.service on MF-AI (systemd, Restart=on-failure), port 8081, firewalld rich rules scoped to the 3 CP-host IPs + VIP (mirroring the existing port-11434 rules exactly).
  • Ollama is left running, unchanged, as an instant rollback -- both backends can be enabled simultaneously in Settings; the engineer picks per chat session.

0.45.1.10 — 2026-08-30

A diagnostic gap found while validating the AI/LLM hosting blueprint (Intel SR-IOV, mfcloud-hosting's ai_blueprints.py): a clean create_vm failure's real reason was unrecoverable after the fact.

Fixed

  • tasks/core.py::create_vm's clean failure paths (template missing, VmBuildError) now print locally, not just report to the master. report_status's update is supposed to land on the task_history row Celery's own "Queued"/staging row created, but a cross-host template-staging race can leave that row stuck on an earlier phase — the final ❌ Failed status silently has nowhere to attach, so GET /api/logs/vm/{name} never shows why a create failed, only that an earlier stage happened. Confirmed live 2026-08-30: a create_vm failure was invisible via the API entirely, recoverable only by reading the compute node's own systemd journal. Real fix (task-identity correlation through a staging step) is a bigger change, deferred — this at least makes the reason recoverable from the node itself in the meantime.

Also found live, not yet fixed (both compute-node tasks.core.create_vm gaps, same testing pass): - A failed create can leave a libvirt domain defined but never undefined — conn.defineXML() succeeds before the network interface is actually resolved at dom.create() time, so a bad network_name (or any other start-time-only failure) leaves a stale, unstartable domain definition behind under the same name. A same-named retry then fails with "domain already exists," compounding the original error. No automatic dom.undefine() on this failure path today. - GPU release-on-failure only covers synchronous failures. routers/iac.py::iac_create_vm releases a gpu_pci_address reservation in its own except Exception block, which only ever sees a failure from create_new_vm's synchronous call (validation, DB insert, queuing) — an async failure inside the compute-node's own tasks.core.create_vm (the common case) is invisible to that handler, so the GPU stays permanently marked assigned to a VM that was never actually created. With no API path to unassign a GPU from a VM that doesn't exist (api_unassign_gpu/api_assign_gpu both resolve the VM's host via _get_vm_host first, 404ing before ever touching gpu_devices), this currently requires either a same-named retry (assert_gpu_available's expected_owner reclaim) or direct DB access to clear. Real product risk given how few GPU/VF slots typically exist per host.

The rest of the UI improvement plan — Phase 1's remaining item, both of Phase 2, and the last two of Phase 3.

Added

  • AI Assistant panel can dock to the right edge instead of only overlaying. A toggle in the panel header (chat-panel.js, layout.css) pins it as a fixed full-height sidebar; the content column picks up a matching margin-right via a class toggled on .app-container (the widget and the content column aren't nested inside each other, so a plain CSS descendant selector couldn't bridge them). Width is independently adjustable via a left-edge drag handle and capped to 320px below 1440px so it can't crowd out the content column on a laptop screen. Falls back to the existing mobile overlay shape below 900px regardless of dock state. Drag-to-reposition and the undocked overlay's own corner-resize are unchanged.
  • The inventory tree is keyboard-navigable for the first time. It was plain divs with only click handlers — no keyboard user could reach it at all. Added roving tabindex over every real row, Up/Down to move, Enter/Space to activate, Left/Right to expand/collapse. Activation dispatches a real .click() through the existing delegated click handler rather than reimplementing its branching logic, so this needed no changes to any of the row-type-specific selection code. role="treeitem"/aria-expanded are mirrored from state the render code already produces.
  • The ~67 legacy .overlay/.modal-box modals now get Escape-to-close and a Tab focus trap. api.js's mf* modal system already had this; the older pattern used by every *_modals.html file didn't. One MutationObserver-driven mechanism in main.js covers all of them generically (watches each .overlay's style attribute rather than needing a hook at every one of the dozens of scattered places that toggle it) and defers entirely to the mf* system when one of its dialogs is layered on top. Escape prefers clicking the modal's own close button so any custom cleanup — closeCtConsole()/ closeCtTelemetry() tearing down a live WebSocket — still runs.
  • Sidebar auto-collapse now reacts live to window resize, not just page load — shrinking below 1440px collapses it, growing back past 1440px re-expands it, but only for a collapse the code applied itself; a saved manual width or a manual collapse-button click is never overridden in either direction.
  • CPU/RAM CUR/AVG/MAX + Pause/Resume on the VM and Host Monitor tabs (carried over from the 0.45.1.3 pass, now the last Phase 2 item is also in) — Network RX/TX already showed this; CPU/RAM only showed a single live value. Same TelemetryBuffer.stats() call, wired into all three render paths each tab has.
  • Empty-state rollout finished — VM Catalog, container backups, and container snapshots (the last three views named in the original finding) migrated onto the shared .table-empty class alongside Datastores/IAM/Reports from earlier passes.

Fixed

  • Light-theme --success badge/text color only hit ~3:1 contrast against its own backgrounds (dark theme's equivalent clears 8.7-9.7:1). Computed actual WCAG contrast ratios for every light-theme status/text token rather than eyeballing them; --success was the one confirmed light-mode-only regression worth a token change — reused the already-established --success-solid value instead of picking a third green, now 4.46-5.02:1. Two related findings were checked and deliberately not changed: --accent has the same shape of gap but is explicitly commented "deliberately unchanged" between themes in base.css (a brand-consistency call, not this pass's to override), and --text-dim's low contrast turned out to be identical in both themes, not a light-theme regression.

0.45.1.7 — 2026-08-29

A second gap found by the same VPS-blueprint testing pass, minutes after 0.45.1.6 — this one meant POST /api/iac/vms never worked at all.

Fixed

  • IacVMCreateModel was missing gpu_pci_address entirely. routers/iac.py::iac_create_vm has referenced data.gpu_pci_address since the GPU-resolver work landed (checking it, validating it via assert_gpu_available, attaching it to create_new_vm), but the field was never actually added to the Pydantic model those reads come from. Every single call to POST /api/iac/vms — GPU or not — raised AttributeError: 'IacVMCreateModel' object has no attribute 'gpu_pci_address' before ever reaching create_new_vm, surfacing as a bare 500. Nothing caught this earlier because nothing had called this endpoint yet — mfcloud-hosting's new VPS blueprint was its first real caller. Fixed: added gpu_pci_address: str = "" to IacVMCreateModel with the same PCI-BDF validator K8sNodeSpec.gpu_pci_address already uses.

0.45.1.6 — 2026-08-29

A gap found while building the VPS hosting blueprint (mfcloud-hosting): a VM created through the IaC API on a customer's behalf had no way to be attached to that customer's tenant.

Fixed

  • POST /api/iac/vms now accepts an optional tenant_id. create_new_vm always derived a new VM's tenant_id from the caller's own token — fine for a tenant's own Terraform key creating their own VMs, but create_new_vm has no equivalent to K8sClusterCreate's admin-settable tenant_id field for a caller acting on behalf of a different tenant. Every VM mfcloud-hosting would have created (using its own admin API key) landed with tenant_id=NULL — invisible to the customer's own tenant-scoped key querying GET /api/vms, since that route filters WHERE vms.tenant_id = %s for non-admin callers. Fixed the same way K8sClusterCreate already solves it: IacVMCreateModel.tenant_id is honored only for a global_admin caller (iac_create_vm sets it via an UPDATE right after create); a non-admin's own token's tenant_id is used regardless, exactly as before this field existed — so a non-admin passing another tenant's UUID here has no effect, closing the obvious privilege-escalation question before it could be asked.

0.45.1.8 — 2026-08-30

Sync of the Emergency UI wizard (Hypervisor_mfcloud_min / _min_el9 standalone-node agent, unrelated to the main console codebase) to close a gap where the KVM/Ceph test cluster's live agent predated both the VM web console and the Firewall API.

Fixed

  • Emergency UI wizard brought current on the KVM/Ceph test cluster (KVM-CEPH-01/02/03). The three nodes were running an agent.py from before the console-proxy and Firewall API existed, and a ~300-line-old frontend on top of that — the Firewall page's badge/summary could go silently stale on a failed fetch, and a freshly generated break-glass password could be wiped from screen by the next auto-poll's 401 reload before it was recorded. Backend was hand-merged rather than replaced outright: kept the nodes' existing bearer-token auth, /api/host/network, /api/logs/install, and /api/admin/users/tokens endpoints, and added the missing /api/vms/{name}/console-token + /console/ws/{token} noVNC proxy and the full /api/firewall* surface. Verified with check_agent_routes.py before rollout that every endpoint the updated script.js calls is actually served, and confirmed novnc was already installed on all three nodes so the console works immediately. Also brought Hypervisor_mfcloud_min's wizard templates in line with _min_el9's.

0.45.1.5 — 2026-08-29

Phase 3 of the UI improvement plan: a scripted audit of every icon-only button in the console for a missing accessible name.

Added

  • 49 icon-only buttons now have aria-label. A script scanned every <button> in static/js and templates for ones with no visible text and no aria-label — a screen reader had nothing to announce for these beyond "button". Three groups: 23 identical .modal-close (&times;) buttons across container_modals.html, marketplace_modals.html, and kubernetes_modals.html; 2 banner-dismiss buttons in main.js (update notice, license expiry notice); and 24 one-off icon buttons — close/ delete/edit/zoom/reconnect/fullscreen/screenshot/refresh — spread across marketplace.js, tasks-drawer.js, alarms.js, atlas.js, automation.js, networks/security_groups.js, containers/clusters.js, dashboard.html, view_summary.html, view_topology.html, and modals_monitoring.html. Phase 1's .host-menu-btn ("⋯" row-action) pattern was spot-checked separately — all 8 instances already had aria-label, so that earlier fix held up.

0.45.1.4 — 2026-08-29

A gap found while finishing the Managed Kubernetes hosting pipeline's live testing: a partial failure in tenant self-provisioning left an orphaned tenant behind instead of rolling back.

Fixed

  • POST /api/iam/tenants/provision now cleans up after itself on failure. The endpoint chains create_tenant (commits immediately) with a role lookup, a Tenant Admin user insert, and an API key mint — a failure in any of the later steps (missing RBAC role, username collision, DB hiccup) used to leave the already-committed tenant, its OVN logical switch, IP pool, and VLAN mapping permanently orphaned, since nothing ever rolled it back. Surfaced twice during mfcloud-hosting's live K8s testing, worked around at the time by retrying with a fresh tenant name instead of fixing the root cause. Now: any failure past tenant creation deletes that tenant via the same path DELETE /tenants/{id} uses, so a retry with the same name/username works cleanly instead of accumulating orphaned test tenants.

0.45.1.3 — 2026-08-29

Phase 2 of the UI improvement plan's first item: CPU/RAM telemetry gets the same CUR/AVG/MAX breakdown Network already had, plus a way to freeze the live view without losing data.

Added

  • CPU/RAM CUR/AVG/MAX on the VM and Host Monitor tabs. Network RX/TX already showed a current/average/max row computed by TelemetryBuffer.stats(); CPU and RAM only ever showed the latest single value. Both now get the same stat row, wired into all three places each tab redraws from (the initial/20s-window chart build, the 2s live update, and the historical rollup view). Applies to both static/js/vms/vms-operations.js's VM Monitor tab and static/js/hosts.js's Host Monitor tab — two independent implementations of the same pattern, fixed the same way in both.
  • ⏸ Pause / ▶ Resume on both Monitor tabs. Previously the only way to stop a live chart from redrawing was to switch time ranges, which restarted polling from scratch. The new button freezes the visible chart/stat redraw only — the underlying 2s/20s collection timers keep polling and pushing into TelemetryBuffer in the background, so Resume reflects the current state instantly instead of replaying a gap.

0.45.1.2 — 2026-08-29

Two of the gaps from the UI improvement plan's Phase 1 quick wins, closed without touching anything visible when the data's already there.

Changed

  • A shared shimmer skeleton replaces one-off inline copies. The mfc-skeleton shimmer animation shipped earlier with the App Catalog redesign but only existed as a duplicated inline style. It's now a .skel-bar class (components.css), reused as-is by containers/images.js and newly applied to IAM's Tenants and Users tables, which previously sat blank while /api/iam/tenants and /api/iam/users were in flight.
  • The Datastores view's empty/error states now use the shared .table-empty class. disks.js, lvm.js, library.js, policies.js, files.js, and drbd_core.js each had their own style="text-align:center; padding:...; color:var(--text-muted)" (or --danger) copied per message. All six now use .table-empty / .table-empty.is-error — same text, same colspans, padding standardized onto the density-aware --widget-pad token instead of a hardcoded pixel value that drifted 14-20px across call sites.

0.45.1.1 — 2026-08-28

A drift alarm on the OVN fabric turned out to be telling the truth, and investigating it surfaced a way OVN writes could silently go nowhere.

Fixed

  • A failed settings lookup no longer retargets the whole OVN control path at the app container. core/ovn.py::_ovn_central_ip swallowed every exception and returned 'localhost', so a transient DB error sent every ovn-nbctl and ovn-sbctl call to the container itself — where neither binary exists. It surfaced as [Errno 2] No such file or directory: 'ovn-nbctl', and because most OVN writes are fire-and-forget BackgroundTasks that only print their failures, the write was simply lost while the console kept showing it as applied. That is the same silent-divergence shape that left Security Group enforcement dead for the feature's entire lifetime.

The DB is still consulted on every call and stays the source of truth, but the last address that read successfully is now cached and serves as the fallback when a later read fails — so a blip can't move OVN's control path. With no cached value the error is raised instead of guessed at. The connection is returned to the pool via try/finally; this runs on every nbctl call, so a leaked slot per failure would have drained the pool.

Housekeeping

  • Removed an orphaned smoke-test logical switch from the OVN northbound DB — an empty leftover with no ports and no console row, which the topology reconciler had been correctly flagging as drift on every 15-minute sweep since it shipped. OVN and ovn_logical_switches now agree exactly.

Known issue (not fixed here)

  • The Celery worker's psycopg2 pool is created in the parent process before Celery forks, and init_psycopg2_pool() early-returns on an already-set mf_pool — so all four prefork children share one set of connections. This corrupts DB reads unpredictably under concurrency (observed as PGRES_TUPLES_OK and no message from the libpq) and is what triggered the bug above. Worker containers only; the app container is single-process with threads, which is what ThreadedConnectionPool is built for.

0.45.1.0 — 2026-08-28

Windows VMs required a separate virtio-win.iso attached and a manual "Load driver" click during Setup just to see the boot disk. Fixed by defaulting Windows guests to hardware Windows already ships a driver for.

Added

  • Windows guests no longer need virtio-win.iso to install. The boot disk now attaches to an emulated LSI SAS1068 SCSI controller (tasks/vm/storage.py) instead of virtio-blk — the same real, non-paravirtual chip most other hypervisors default new Windows VMs to, and one Windows has shipped an inbox driver for since Vista. Setup now sees the disk immediately with no driver-load step. virtio-win.iso is still staged and attached as a second CD-ROM as before, so it remains available for an optional post-install upgrade to virtio-scsi for higher I/O throughput. Linux guests are unaffected — still vda/virtio, byte-for-byte unchanged.

0.45.0.1 — 2026-08-28

Every EL9 compute node crash-looped its worker after pulling this build, because 0.45.0.0's own HA-Ready fix (below) started importing a core/ module the shipped image's node-push bundle never actually contained.

Fixed

  • core/fencing.py missing from the EL9 node-push bundle, crash-looping mfcloud-worker on every enrolled compute node. 0.45.0.0 added a core.fencing import to tasks/hardware.py, but fencing was never added to mfconsole-ship/build_cython_node.py's SOURCES list or Dockerfile.ship's builder-node-el9 COPY line — the two build-side mirrors of core/config.py's WORKER_CORE_MODULES. The stem compiled/built fine and looked correct; the .so just never reached node_bundles/el9/, so every EL9 node's worker hit ModuleNotFoundError: No module named 'core.fencing' on startup and restart-looped indefinitely — the console could no longer talk to any node running this image. This is the same failure class as the zfs_node_setup incident on 2026-08-02, just for a different module. Both build files now include core/fencing.py.
  • pack.sh now fails the build if WORKER_CORE_MODULES ever drifts from either build-side file again, instead of shipping the gap silently. This is the second time this exact class of bug has reached a real deployment; the check makes a third time a build failure instead of an incident.

0.45.0.0 — 2026-08-27

VM HA's "HA-Ready" badge could tell an operator a VM was safe to fail over when a real incident would have refused to move it. Fixed, along with a live audit of the failover state it reports on.

Fixed

  • HA-Ready badge now accounts for BMC presence on non-RBD storage. tasks/ha_monitor.py's cosmetic tag previously judged a VM "HA-Ready" purely from storage type, whether the cluster had HA on, and survivor count — it never checked whether the host actually had an iDRAC/BMC configured. The real failover engine does: the Ceph storage-fence fallback (used when a node has no BMC) only covers RBD, so a DRBD or NFS VM on a BMC-less host was explicitly skipped during an actual failover ("needs STONITH/iDRAC to fail over safely") — while its badge still showed green. The safety logic was always correct; only the badge was lying, and an operator would only discover that mid-incident. Now checks has_bmc() per host (batched once per host, not per VM) and downgrades to ⚠️ HA-Degraded for any non-RBD VM on a host without a BMC.
  • Live-audited the production database for the separate DB-consistency risk this surfaced (an optimistic vms.host_node write with no rollback if a dispatched restart genuinely fails) — no nodes currently Failed/ Fence-Failed, no VM referencing a non-Active host, and the last 20 ha-failover audit entries all completed successfully. Confirmed clean today; the underlying gap (nothing re-checks that a VM's recorded host actually has a matching running domain) is not yet closed and is tracked as a known risk, not fixed by this release.

0.44.12.0 — 2026-08-27

GPU-as-a-Service closes its SR-IOV gap and reaches into Kubernetes. SR-IOV virtual functions are now first-class rows in gpu_devices instead of a disconnected cluster-level toggle, LXD containers can claim a GPU for the first time, and a Kubernetes node can request one at provision time — with a device plugin and a real GPU-backed Marketplace deploy following automatically.

Not yet live-tested. Everything below is implementation — verified to compile and to reuse this codebase's existing helpers correctly — but not run against real hardware this pass. NVIDIA passthrough into a K8s cluster (whole-card passthrough itself is already proven end-to-end) is the one path realistically testable next; Intel SR-IOV into a Linux guest remains blocked upstream regardless of anything here (see the GPUaaS design doc's own risk list), and AMD SR-IOV stays out of scope entirely.

Added

  • SR-IOV lives in gpu_devices now. A Virtual Function enabled via the existing "Enable vGPU SR-IOV" cluster action gets its own row (mode='sriov-vf'), assignable through the same GPU-picker widget as whole-card passthrough; its Physical Function is tagged mode='sriov-pf' and force-cleared of any assignment, since attaching a PF to a VM destroys the VFs carved out of it. Previously a VF only ever showed up as a generic, unlabeled PCI device. AMD generalization deliberately not attempted this pass — no AMD hardware to validate against.

  • LXD containers can hold a GPU. Previously VM-only. Uses LXD's own native gpu device type — a new "🎮 GPU" panel on the container hardware menu, backed by PUT/DELETE /api/containers/{host}/{name}/gpu.

  • Kubernetes nodes can request a GPU at provision time, reusing the exact whole-card passthrough mechanism a regular VM's GPU assignment already uses — no new attach code. Once a GPU-carrying worker joins, the matching NVIDIA or Intel device-plugin DaemonSet is applied to the cluster automatically, and nvidia-container-toolkit is configured as containerd's default runtime on that node.

  • vLLM deploys onto a real GPU automatically. Deploying the vLLM Marketplace stack onto a cluster with a detected GPU node swaps in a GPU-backed image and adds the real resources.limits request (nvidia.com/gpu / gpu.intel.com/i915) — no toggle to find, it just works when the hardware is there. ComfyUI and Ollama + Open WebUI still deploy CPU-only; wiring them the same way needs a Helm-values path this pass didn't build.

Known gaps

  • Bare-metal (source='host') K8s nodes and source='existing' VMs can't request a GPU yet — only source='new' with an explicit host.
  • No per-tenant GPU count cap yet (core/quota.py has the exact pattern to add one later).
  • The GPU-side-channel tenant-isolation review the design doc flags for shared SR-IOV VFs is still open — this release does not resolve it.

0.44.11.2 — 2026-08-27

Two fixes to ZFS cross-host replication (routers/zfs.py, tasks/zfs.py, core/schema.py) found during a design review of the feature.

Fixed

  • Replication snapshots were never pruned. Every successful run left its mfcloud-repl-* snapshot behind on both the source pool and the target pool it was received onto — nothing ever destroyed the previous one. Left running, this slowly fills the pool the feature exists to protect. Each run now keeps the newest ZFS_REPL_KEEP_SNAPSHOTS (default 3, env-configurable) on both ends and destroys the rest; snapshots outside the mfcloud-repl-* naming pattern (e.g. manual/unrelated snapshots on the same pool) are never touched.
  • A replication schedule could be pointed at a non-empty target pool. zfs recv -F force-rolls-back whatever is already at the destination to match the incoming stream — so a target pool already holding unrelated datasets or snapshots (for instance, live VMs) would have them destroyed on the first run, with no warning. Creating a schedule now probes the target over SSH and rejects it if the pool exists and is non-empty, or if another schedule already owns that (host, pool) pair. A matching UNIQUE database constraint is the actual backstop underneath the friendlier API error.

0.44.11.1 — 2026-08-27

Licence expiry now warns you before it bites. Fixed-term and trial keys carry their own valid_until, and until now the console said nothing until the day the caps actually dropped — on the Settings page only, which nobody visits daily.

Added

  • Licence expiry countdown. A top banner appears from 14 days out, with a Manage License shortcut. Dismissal is per warning tier (14 / 7 / 3 / 1 day, and expired), so silencing the two-week notice doesn't silence the final-day one. Super-Admin only — the roles that can't act on a key don't see it.
  • The Settings → Edition & License card shows the same countdown inline, stating plainly that expiry reverts to Free-tier caps while existing VMs and nodes keep running.

Perpetual keys (no valid_until) are unaffected and never warn. Enforcement is unchanged: core/license.py remains the sole, fully-offline authority on what a licence permits — this release only surfaces what it already knew.

0.44.11.0 — 2026-08-27

VMs on OVN tenant networks get an L4 load balancer. K8s clusters have had MetalLB since V40; VMs had no VIP primitive at all. Closes the last purely buildable capability gap on patch_note/roadmap/networking-sdn-roadmap.html.

Added

  • OVN-native load balancer. POST /api/networks/ovn/load-balancers defines a VIP and its backend pool; /backends grows and shrinks the pool; /attach applies the LB to a logical switch or router. OVN's load balancer is distributed — the translation happens in each chassis's own flow tables, so there is no appliance to place, scale or fail over, and traffic never detours through a central node.

A bare-IP VIP balances every port; an IP:port VIP is protocol-specific (tcp/udp/sctp). VIP and backends must agree on whether a port is present, checked in both the model and the primitive: OVN accepts the mismatch and then matches no traffic at all, which looks identical to a backend outage.

Attaching to a switch covers traffic originating there; attaching to a router also covers traffic arriving through it, which is what makes a VIP reachable from outside the tenant network via a gateway. An LB with no attachment is configured but inert, so the list endpoint reports active per LB rather than letting it read as healthy.

Removing the last backend deletes the VIP from OVN rather than leaving an empty pool — an empty pool blackholes every connection to the VIP, which is worse than the VIP not answering at all.

Changed

  • Rebuild and drift sweep cover load balancers. rebuild_ovn_topology re-pushes each LB and re-applies its attachments; reconcile_ovn_topology reports an LB missing from OVN, a VIP missing from an existing LB, and a backend pool that differs from the configured one. Pool comparison is set-based: OVN preserves neither the order sent nor the spacing, so only membership is meaningful.

Fixed

Both surfaced during live validation of the above, and both were silent — the LB looked correctly configured while doing the wrong thing.

  • Editing a backend pool silently detached the load balancer. The update path was delete-then-add, but removing the last VIP removes the Load_Balancer row itself, and with it every ls-lb/lr-lb reference pointing at it. Adding one backend therefore dropped the LB from every switch and router it served, leaving a VIP nothing routed to. Updates are now a set on the existing row, so the UUID — and every attachment — survives. Two concurrent delete-then-add runs also collided outright ("a load balancer with this vip already exists"); LB syncs now serialize on a per-LB advisory lock, the same mechanism the security-group reapply path uses.

  • Attaching straight after creating lost the attachment. Create and attach each sync to OVN from their own fire-and-forget BackgroundTask, so the attach could reach OVN first and fail with "load balancer name not found" — the natural client sequence, and what any UI would do. The attach now pushes the LB from the database first when it isn't there yet, under the same lock.

0.44.10.0 — 2026-08-27

OVN learns north-south. Logical routers could only ever route east-west between tenant switches; the only way off an OVN network was to bridge a switch straight onto a physical VLAN and let an external device do the NAT. Closes the second sequencing batch in patch_note/roadmap/networking-sdn-roadmap.html.

Added

  • External gateway on a logical router. POST /api/networks/ovn/routers/{name}/gateway builds the standard OVN north-south shape — an external logical switch with a localnet port onto a physnet, and a router port holding the address the outside world sees. One gateway per router (enforced by a UNIQUE on router_id): re-posting updates it in place rather than stacking a second external port, which OVN would accept and then split traffic across unpredictably. physnet must match an ovn-bridge-mappings entry on whichever chassis carries the traffic, or OVN has nowhere physical to put the packets and north-south dies silently at the gateway.

  • SNAT and floating IPs. POST /api/networks/ovn/routers/{name}/nat adds either — snat masquerades a whole tenant subnet behind one external address, dnat_and_snat pins one external address to one VM in both directions. A floating IP is refused a CIDR for its logical address, since translating a whole subnet to one address both ways is never what a floating IP means. The optional distributed form passes the VM's own logical port and an external MAC so OVN translates on the VM's chassis instead of hairpinning through the gateway. Adding NAT to a router with no gateway is refused with a 409 rather than accepted into a configuration that forwards nothing.

  • Gateway chassis HA. POST /api/networks/ovn/routers/{name}/gateway/chassis nominates a hypervisor to forward a router's north-south traffic. Add a second and third at lower priorities and OVN elects the highest-priority live one, failing over on its own — that is the whole of gateway HA. The GET endpoint reports ha: false while only one chassis is scheduled, so a single point of failure reads as one instead of looking healthy.

Changed

  • The topology rebuild and the drift sweep both learned north-south. rebuild_ovn_topology now re-pushes gateways, gateway-chassis scheduling and NAT rules in dependency order (gateway before the chassis that serves it, both before the translations) — without that, recovering a lost central would have silently dropped every floating-IP allocation. reconcile_ovn_topology reports a missing external switch or router port, a NAT rule OVN doesn't have, and a scheduled chassis that isn't set on the gateway port.

It also learned an exemption it needs to stay useful: the external switch behind a gateway (ls-ext-<router>) is created by the app but deliberately has no ovn_logical_switches row, being an implementation detail of the gateway rather than a tenant network. Without that carve-out the sweep would have reported every gateway in the fleet as an unknown switch, every fifteen minutes, forever.

Fixed

Both of these were found by testing the above against live OVN rather than by review, and both failed silently — the reason the gateway path is worth calling out as live-validated rather than merely shipped.

  • Gateway chassis were scheduled by hostname, which OVN never matches. The app keys chassis on hostname (KVM-DCTCK93); OVN's Southbound Chassis.name is the OVS system-id, a UUID. ovn-nbctl accepts a hostname without complaint and northd still compiles the chassisredirect port — but no chassis ever claims it, so the gateway reads as fully configured in the northbound DB and forwards nothing. Hostnames are now resolved through Southbound before being written, and translated back on read so the drift sweep keeps comparing hostnames to hostnames.

  • Removing a gateway chassis always failed. lrp-del-gateway-chassis is one of the few ovn-nbctl subcommands that rejects --if-exists outright, which the delete path was passing on every call. The flag is gone and an already-absent entry is treated as success, while a real failure still raises.

0.44.9.1 — 2026-08-27

Fixed

  • The container HA monitor no longer stops watching when the database blips. A transient connection loss — the kind a Patroni failover or a rolling container update produces — raised straight out of the scheduled task, so the whole HA sweep was abandoned for that cycle. The moments most likely to cause a blip are exactly the moments HA is supposed to be watching, which makes this worse than it sounds.

Each member's health update is now isolated and retried once, and the task reports a summary instead of dying. Critically, a member whose health record can't be read is skipped, never evacuated: without a trustworthy last-seen timestamp there is no way to tell a node that has been offline for six seconds from one offline for six hours, and the wrong guess would relocate a healthy node's containers. Found while verifying the 0.44.9.0 deployment.

0.44.9.0 — 2026-08-26

The LXD container platform closes its remaining capability gaps. Containers now count against tenant quotas, back up off-site, move between unclustered hosts, carry an edition ceiling, and get continuous load balancing — the five areas where they still trailed the VM platform.

Deployed for validation, not yet proven in the field. Every LXD REST interaction added here (backup export/import, cross-server migration, the resources probe behind container CRS) is covered by unit tests but has not run against a live LXD daemon or S3 target. Treat this release as the thing you test with, not the thing you rely on.

Added

  • Containers count against tenant and cluster quotas. core/quota.py had no awareness of containers at all, so a tenant sitting at their vCPU/RAM ceiling could keep allocating compute indefinitely by creating containers instead of VMs. Ceilings are now a combined footprint over VMs and containers — a ceiling means "this tenant may commit N vCPU", and splitting it per hypervisor type just recreates the bypass in a new shape. The node footprint (max_nodes) spans both too, so a node held only by a container counts as occupied. Enforced on container create and on CPU/RAM edits (gated on the delta, so a resize isn't double-counted against itself). The IAM quota gauges no longer under-report.

  • Off-site container backup to S3. Snapshots are same-host and die with the node; there was no disaster-recovery path for containers at all. Backups reuse the VM pipeline's encryption, compression and multipart upload unchanged, with per-container schedules run by the same cron loop as VM backups, retention, and restore — including restore onto a different host for the case where the original is gone.

Two things are stated plainly in the UI because they surprise people: every container backup is a full copy (LXD has no incremental export, so each run uploads the whole rootfs — schedule accordingly), and a running container's backup is crash-consistent unless you tick the option to stop it for the duration. LXD cannot export a snapshot on its own, so those are genuinely the only two choices.

  • Migration between unclustered hosts. Moving a container used to require a formed LXD cluster; anything else was refused outright. Containers can now move between two independently enrolled hosts. Clustered pairs still get the fast metadata move — the new path is used only when they aren't clustered, and it says so before you confirm, because it copies the filesystem rather than just the metadata.

The move is non-destructive by construction: the target is pre-flighted (storage pool names, profiles, name collisions), the transfer runs, and the original is deleted only after the target confirms it arrived. A failure at any earlier point leaves the container running exactly where it was.

  • Container CRS — continuous load balancing. Containers had a placement scheduler at create time but nothing that corrected drift afterwards. CRS now watches CPU, RAM and container-count spread across cluster members and rebalances, with the same anti-flap cooldown, anti-affinity and minimum-improvement guards the VM balancer uses.

Opt-in per cluster and set to recommend, not act, by default — unlike the VM balancer, which defaults to acting. It also refuses to run without healthy shared cluster storage, since without it a rebalance would copy each container's whole filesystem instead of moving its metadata, and this runs unattended every five minutes.

  • Edition ceiling for containers. core/license.py gained a container_limit per edition, enforced at create time alongside the quota check. Every edition currently ships unlimited — the plumbing is in so a ceiling can be switched on later, but no number has been chosen. Existing license keys are unaffected: the limit resolves from the edition, so nothing needs re-issuing.

Changed

  • CRS tuning constants moved to core/crs_config.py so the VM and container balancers read the same env-overridable values. Previously setting something like CRS_COOLDOWN_MIN would have applied to one and been silently ignored by the other. The VM balancer's own logic is unchanged.

0.44.8.0 — 2026-08-26

OVN gets the two things it was missing to be trustworthy on its own: guests that actually receive the address IPAM allocated them, and something watching the fabric for silent drift. Closes the first batch from patch_note/roadmap/networking-sdn-roadmap.html.

Added

  • The IPAM lease now reaches the guest. Allocating an address, binding the OVN port and telling OVN the port's MAC/IP all already worked — and the VM still booted with no address configured. Nothing carried the lease into the guest, and OVN's own DHCP responder was never set up (DHCP_Options had zero uses anywhere in the codebase), so a VM only got an address when an external DHCP server on a bridged physical VLAN happened to answer. A logical switch with no physical uplink — the whole point of a tenant network — could not produce a reachable VM.

Two independent deliveries now cover it. Every port bind ensures a per-subnet DHCP_Options row (server_id, router, lease time, DNS from the pool) and points the port at it, so any guest that DHCPs is answered by ovn-controller on its own chassis — no DHCP traffic leaves the hypervisor. And VM create writes the same lease into cloud-init's static network config, covering images that come up before the offer. An operator-supplied ci_ip/gateway/DNS always wins; this fills blanks only. The prefix comes from the pool's own subnet rather than cloud-init's /24 assumption, which was silently wrong on any other pool. (core/ovn.py, routers/networks_ipam.py, routers/vms/core/lifecycle.py)

  • OVN topology drift detection. Every OVN write path pushes through fire-and-forget BackgroundTasks whose failures are caught and printed, so a switch, router, router↔switch connection, provider uplink or VM port could fail to materialise — or be deleted out of band — while the console kept showing it as healthy. That is the exact pattern that left Security Group enforcement silently dead for the feature's entire lifetime.

New tasks.ovn.reconcile_ovn_topology beat sweep (15 min) diffs the app's tables against live ovn-nbctl show and reports both directions: objects the database expects that OVN doesn't have, and objects OVN has that the database doesn't know about. Detect-and-flag only — it never repairs, because the two sides diverge for opposite reasons (a push that failed, versus an operator fixing something live with ovn-nbctl) and auto-pushing would silently revert the second. Surfaced on the Networks → Topology tab and at GET /api/networks/ovn/drift, with a task_history row written only when drift is found. Complements the existing ACL sweep and central_health's chassis drift; together those three now cover every OVN object type. (tasks/ovn.py, tasks/app.py, routers/networks_ovn.py, static/js/networks/overview.js)

  • Rebuild OVN from the database. POST /api/networks/ovn/central/rebuild re-pushes every logical switch, router, connection, localnet port, VM port, DHCP option and security-group ACL the app's tables describe. OVN's contents are derived state, so a rebuilt or replaced central no longer means hand-running ovn-nbctl — and it is also the remedy for whatever the drift report flags as missing.

Additive and idempotent by construction (new --may-exist ensure variants in core/ovn.py): it never deletes an OVN object the database doesn't know about, since that object may be hand-built plumbing and a sweep that removed it would turn a recovery action into an outage. ACL re-application takes the same per-switch advisory lock the live path uses, so a rebuild can't race an operator's security-group edit. Refuses up front with a 503 when central is unreachable, rather than producing one timeout error per object.

Fixed

  • Deleting an IPAM pool left its DHCP options behind. The DHCP_Options row is keyed by CIDR and shared by every pool on that subnet (non-overlapping ranges on one subnet are legal), so it is now torn down only when the last pool covering that subnet is deleted.

0.44.7.2 — 2026-08-26

Content Library Publish learns to ask where. Follows 0.44.7.1's ISO work.

Added

  • Publish/Copy now lets you choose the source and target nodes. The action previously took no input at all: it sent every missing node as the target and silently used present_on[0] as the source, so "copy this ISO to one node" was not expressible and "copy from the node that isn't across the slow link" was not either. Both endpoints already accepted an explicit list — only the frontend was hardcoding it.

Clicking Publish/Copy now opens a picker: a Copy from select (shown when more than one node holds the item), a checklist of candidate targets each annotated with its cluster, Select all / Clear, and a confirm button that carries the live count and stays disabled at zero.

Defaults follow the media type, matching 0.44.7.1's framing: a template opens with every target ticked, because being everywhere the placement engine might land a VM is the point of a golden image. An ISO opens with none ticked, because it is mounted by one VM on one node — fanning an installer image at the whole fleet by reflex is what the dialog exists to stop.

New shared helper mfMultiPickModal in static/js/api.js — the multi-select sibling of mfPickModal, with the same focus trap, Escape handling and focus-restore contract. Any other fan-out action can now use it rather than growing its own checkbox overlay. (static/js/storage/library.js, static/css/modals.css)

0.44.7.1 — 2026-08-26

Content Library fixes, all of them about ISOs. The library was built around golden templates and ISOs were fitted into the same row afterwards, so every ISO inherited framing, wording and a delete warning that only make sense for a template. Nothing here changes what is stored or where — only what the Datastores tab claims about it.

Fixed

  • The node installer ISOs were listed as deployable VM media. mfcloud-min*, the legacy mfcloud-v<ver>-enterprise builds, mfconsole-node-* and MFOS are bootable metal installers for turning a box into a compute node. The library listed them alongside virtio-win.iso and then reported that seven nodes were "missing" one — a permanent false gap for an image no node was ever supposed to hold. They are filtered out of the Content Library now, and left in the Deploy dialog on purpose: building a nested node from the installer ISO is a real workflow. (routers/storage/images.py)

  • ISO rows nagged about fleet coverage they don't need. A template wants to be on every node — the placement engine can only use a node that holds the golden image, so "3 missing (fleet-wide)" is a real gap. An ISO is boot media: it is only needed on the node whose VM mounts it, so the same line was a standing false alarm. ISO rows now state coverage flatly ("on 5 of 8 nodes"), list only the nodes that actually hold the media instead of a wall of absent ones, and offer a secondary "Copy to nodes…" action rather than a primary "Publish to 7" call to action. Template rows are unchanged.

  • The ISO delete warning was wrong, not just mislabelled. Both confirm dialogs were hardcoded to "Template", and the delete dialog promised "Existing VMs are NOT affected (they boot from their own disks)". That is true of a template, which is only ever copied from at deploy time, and false of an ISO, which is mounted as a live CDROM — a VM still holding one will fail to start once the file is gone. The dialogs now name the right media type and the ISO path states the real blast radius. (static/js/storage/library.js)

  • Library badges rendered as loose floating text. The type and per-node chips used a bare .badge with an inline colour override, which meant a transparent border, no background, and an invisible .badge::before status dot eating 10px of lead before every label. They get their own dot-free classes now (.cl-type, .cl-node, .cl-coverage-note). The type badge is also no longer colour-coded: warning-orange on a healthy ISO implied a problem that wasn't there, and "ISO"/"Template" already carries the distinction — same reasoning already applied to the segment-type badges in networks.css. (static/css/tables.css)

0.44.7.0 — 2026-08-24

A Networks overview dashboard, a packaging fix that makes the published image installable on Podman at all, plus code-review fixes on the 0.44.6.6/0.44.6.7 sizing and Redis-hardening work.

The compose layering change is the one item here that alters how a deployment is brought up (compose.dev.yml is now required for source builds — see below). The rest change nothing an existing deployment runs: the untiered profile reproduces the shipped pool values exactly, and the one behavioural change (the rate limiter) restores what 0.44.6.6 already had.

Added

  • Networks now has a landing pane instead of only leaves. Clicking "Network Objects" at the root of the Networks inventory tree opens a dashboard over the whole fabric: a KPI row (L2 segments, logical routers, chassis online, host bridges, security groups, address pools), a logical topology diagram drawn from the real router→switch attachments, fabric health per chassis, and address-pool utilisation.

Four tabs: Overview, Topology, Address Space (every IPAM pool with allocation meters and free counts), and Segments — the first place in the console that lists portgroups, cluster VLANs and OVN overlays side by side, since those three are configured in three different screens and nothing previously showed them together.

Two things it surfaces that previously required clicking every leaf: a logical switch with no router attached (east–west only, no gateway) is called out by name, and a chassis that is not Active is marked on the topology with both a status colour and a dashed border, so it survives greyscale and colour-vision deficiency.

It reads the same window._netTree payload the tree render already fetched, so it costs zero additional API calls and cannot disagree with the tree beside it. New files: static/js/networks/overview.js, static/css/networks.css.

Fixed

  • The registry-pull install path never worked on Podman — on any host without Docker. docker-compose.yml carried build: and compose.prod.yml removed it per-service with Docker Compose v2.24's !reset tag. podman-compose does not implement !reset: it parses the tag into a ResetTag object and then dies in normalize_service with TypeError: argument of type 'ResetTag' is not iterable. Confirmed on podman-compose 1.5.0 (Rocky 10) and 1.6.0 (upstream). Since install.sh is the documented way to stand up a pulled image, and Podman is the default engine on the RHEL-family hosts this product targets, that was the whole install path.

The layering is now inverted rather than patched: docker-compose.yml is image-based by default, and a new compose.dev.yml turns those image references back into local builds for source-tree work. !reset is gone from the tree entirely and both paths are engine-agnostic. The internal build flow gains a fail-loud property it did not have — the image: names in the base file are local-only tags that exist in no registry, so forgetting -f compose.dev.yml now errors with "image not found" instead of quietly starting a different build than the source just pushed.

  • Watchtower was referenced by a bare image name. Podman enforces short-name resolution and hard-errors ("cannot prompt without a TTY") on a bare name in any non-interactive context, so the container the console's "Update Now" button depends on failed to start on an unattended boot. Now fully qualified.

  • A sizing tier was documentation, not configuration. MFCLOUD_DEPLOYMENT_SIZE had exactly one consumer — the startup log — so selecting a tier changed no pool, no Postgres setting and no worker concurrency, while muting the one genuine warning about the untiered 765-connection peak. core/database.py now takes both its psycopg2 and asyncpg pool bounds from core/sizing.py, the worker's concurrency is ${CELERY_CONCURRENCY}, and the validator reasons about the effective profile (tier + .env) rather than the paper table. Selecting a tier without regenerating .env now warns by variable name instead of silently half-applying.

  • A typo in MFCLOUD_DEPLOYMENT_SIZE was indistinguishable from leaving it unset. smal silently ran the untiered legacy profile while the operator believed their tier was in force. It is now a startup error naming the valid tiers.

  • The worker's pool size ignored .env. docker-compose.yml restated DB_POOL_MIN/MAX under the worker's environment:, which outranks env_file: .env — so an operator who set DB_POOL_MAX=10 per the KB got 50 back on that container, i.e. 4 children x 50 = 200 connections per node from the worker alone. Resolution moved in-process (WORKER_DB_POOL_* → DB_POOL_* → profile), where .env is an override rather than a loser by precedence. Same restatement removed from app-1/2/3, which would otherwise have pinned every replica to the legacy defaults even with a tier selected.

  • A false shared-memory warning on every boot. The untiered profile claimed podman's 64m /dev/shm default while docker-compose.yml sets it explicitly. That comparison is now only made when both numbers are actually knowable — the value is a compose-file constant this module cannot read, and a hardcoded guess has already gone stale twice.

  • The console update check could park forever on a dead socket. The one Redis client the 0.44.6.6 hardening pass missed (tasks/update_check.py) had no timeouts, so a VIP move stalled that beat task with no exception raised and nothing logged. core/state.py moved to the request-path options as well — its callers are synchronous route handlers, so its worst case was being paid by a user waiting on a page (~6s, now ~2s).

  • haman's rolling update hard-required a file that is not always present. Phase 2 passed -f compose.dev.yml unconditionally, which aborts on a registry-pull install (pack.sh does not ship that file) and on any cluster still on the pre-layering inline-build: compose — in the first case after phase 1 had already overwritten every other host's tree. It now detects which of the three layouts is on disk and builds, pulls, or builds-inline accordingly, and refuses before shipping anything if it cannot tell.

Changed

  • The rate limiter keeps its in-memory fallback. It had been removed from the working tree as a supposed no-op; the check behind that decision swapped limiter._storage for a failing double, but Limiter.__init__ builds self._limiter = STRATEGIES[strategy](self._storage) and every hit goes through that — so the live path kept using the original working storage and the fallback was never exercised. Measured correctly against slowapi 0.1.10 with a genuinely closed port and a 5/minute route: without the flag, [200 x 8]; with it, [200 x 5, 429, 429, 429]. Deployed nodes already had it enabled, so this prevents a regression rather than introducing a change. _FailOpenLimiter stays as the belt-and-braces layer beneath it.

Counters are per replica while the fallback is active, so the effective fleet-wide allowance during a Redis outage is up to 3x the configured limit. Three times the limit is a rate limit; no limit is not.

Added

  • tests/test_sizing.py (20 tests): the untiered profile still produces the exact values that shipped before profiles existed, every shipped tier fits its own connection budget, and every misconfiguration is loud.
  • tests/test_ratelimit_fail_open.py grew cover for the fallback actually metering while the shared store is down, and no longer leaks RATELIMIT_STORAGE_URI into the rest of the pytest session.

0.44.6.7 — 2026-08-23

Diagnostics for an unresolved HA defect, and a correction to 0.44.6.6 below.

Added

  • py-spy, so the next control-plane hang can be explained instead of guessed at. Twice now a control-plane host going away has frozen the surviving app replicas, and both times the post-mortem had to be assembled from socket tables and log gaps because there was no way to see what Python was actually executing. py-spy dump --pid <uvicorn pid> prints the live stack of every thread without stopping or restarting the process — which matters because a restart is the only known remedy and it destroys the evidence. Run it from the host against the container's PID (podman inspect --format '{{.State.Pid}}' mfcloud-master_app-N_1).

Fixed

  • Nothing yet. The defect described in 0.44.6.6 is still open. See below.

0.44.6.6 — 2026-08-23

Redis connection hardening. Does NOT fix the outage the original version of this entry claimed it fixed — that entry was written before the fix was tested and was wrong; this is the corrected text.

Fixed

  • Every Redis client in the app can now hang forever on a peer that vanishes; none of them could bound it before. All clients are built from a bare URI with no socket_timeout, aimed at the keepalived VIP. A control-plane host that is powered off or unplugged sends no FIN and no RST, so its sockets stay ESTABLISHED from the kernel's point of view and the default TCP keepalive will not question them for over two hours.

All Redis clients now share one hardened set of connection options (core/redis_opts.py): socket and connect timeouts, TCP keepalive tuned to detect a silent peer in ~25s, and health_check_interval so a connection orphaned by a VIP move is discarded and replaced rather than handed to the next request. Measured against a blackholed peer: the request path is bounded at 2.0s and background/WebSocket clients at 6.0s, where the old clients never returned at all. Postgres was never exposed to this because the psycopg2 pool has set equivalent keepalives since 2026-07.

This is real and worth having. It is hardening, not the fix — see below.

  • The rate limiter could take the console down. Two layers now, because swallow_errors alone was not enough — with Redis unreachable every limited route still returned 500, from an AttributeError in the decorator's header-injection call rather than from the storage error itself:

  • _FailOpenLimiter (core/ratelimit.py) guarantees the request is allowed through and the failure logged, never turned into a 5xx. A component whose job is shedding load must never be the reason the console is unreachable.

  • in_memory_fallback_enabled=True keeps the limits enforced off per-process counters while the shared store is down, instead of leaving /api/login unmetered for the duration of a failover.

Counters are per replica while the fallback is active, so on the 3-host HA stack the effective allowance during an outage is up to 3x the configured limit. That is a deliberate and stated tradeoff: three times the limit is a rate limit; no limit is not. Regression cover in tests/test_ratelimit_fail_open.py.

  • The /api/ws/tasks WebSocket pub/sub client had no timeouts at all. Now uses the shared options.

Known issue — losing a control-plane host still freezes the surviving replicas

Two deliberate failover tests, before and after the above:

2026-08-23 21:09 (before) 2026-08-23 21:41 (after)
Infrastructure VIP moved in seconds; Patroni promoted and bumped the timeline in ~35s; HAProxy followed at +36s; Celery workers reconnected; the returning node rejoined via pg_rewind with no reinit same, all correct
app replicas froze ~7 min, recovered only when the downed host returned froze 173s, recovered only on manual restart

The infrastructure layer is not the problem and has passed both times.

The surviving replicas log nothing at all while frozen — not even the uvicorn access log. That rules out a slow await, which yields and would let other coroutines keep running and logging. Total silence means the event loop thread itself is blocked by a synchronous call, which no amount of Redis timeout tuning can address. The original 0.44.6.6 entry misread this.

Leading candidate, unconfirmed: core/background.py's loops are started with asyncio.create_task() in main.py, so they run on the loop thread, and at least two of its async def functions call the synchronous psycopg2 pool directly (authorize_master_key_on_all_nodes, sync_worker_code_on_all_nodes, both with get_db() as conn:) while also driving paramiko SSH to every node — including the one that just disappeared. Both are leader-singleton work, so a leadership change is exactly what triggers them, and losing the host causes one. The app log during the second test shows this family of code running and hitting the database mid-failover.

That fits every symptom, but it is a hypothesis. py-spy was added in 0.44.6.7 specifically so the next occurrence produces a stack trace instead of another inference. Until then, the operational remedy is podman restart on the affected app container — it restores service in ~10s.

0.44.6.5 — 2026-08-23

Packaging fix for the distributed image. No application behaviour changes.

Fixed

  • The HA Manager never reached a registry-pull install. haman/ was allowlisted into the ship bundle (mfconsole-ship/pack.sh) but listed in .dockerignore, and Dockerfile.ship never copied it — so the published image had no haman/ at all. services/ha_provisioner.py::_ship_checkout builds the payload for Enable HA by tarring the running image's own /app root, and a directory that isn't there is simply not shipped: no error, no warning, the wizard reports success and the hosts come up without the manager the KB tells operators to install from /root/mfcloud-master/haman/. An allowlist only warns about a listed path missing from source, never about a needed path being excluded downstream — the same silent failure mode agent_overlay/ hit before. Internal HA hosts were unaffected throughout (they get the directory from the source tree copied to them directly), which is why three releases of HA Manager work did not surface it.

haman/ ships as plain source on purpose: it runs on the host under its own systemd unit and the host's system Python, so a Cython .so built against this image's cp312 ABI would not import there.

0.44.6.4 — 2026-08-23

The inventory tree had no sort order at all, so clusters wandered.

Fixed

  • Inventory tree ordering. Datacenters, clusters and hosts were read with no ORDER BY, so the tree rendered in Postgres heap order — which changes every time a row is updated, since MVCC rewrites the tuple at the end of the table. Editing a cluster silently pushed it toward the bottom of the tree, and crs_last_pass_at made it continuous: a CRS-enabled cluster's row is rewritten on every pass, so the tree reshuffled itself with nobody touching it. All three queries are now explicitly ordered.

Added

  • Cluster reordering. Clusters have a real, persisted position (clusters.sort_order) and admins can move them with Move Up / Move Down in the cluster right-click menu (POST /api/inventory/clusters/{name}/move). Existing installs keep the order they already show — the migration seeds positions from the current physical row order rather than imposing a new one — and new clusters land at the bottom of their datacenter instead of at a random spot.

0.44.6.2 — 2026-08-22

The HA Manager could tell you the control plane was sick, but not what it looked like on the way down, and nothing it knew could leave the host. Both of those are now first-class: live rate telemetry with history, and a one-click diagnostics bundle.

Added

  • HA Manager — live telemetry. The manager's own view of the hosts was four snapshot polls with no history and no rates: disk percent and a 1-minute load average. It now runs a background sampler that takes two /proc snapshots 1s apart inside a single SSH round trip, so CPU, network and disk I/O are real rates, and keeps ~60 minutes of them in memory. Per-host cards chart CPU (iowait split out from total, because "busy" and "stuck on disk" need different responses), memory, network rx/tx and disk read/write, alongside load, PSI, root-filesystem fill, sensor temperatures and per-container podman stats. A host that stops answering keeps its charts — the outage is exactly when the run-up matters. The sampler only runs while a browser is polling it, so an unattended manager costs nothing; every sample is an SSH login on two peers, and a permanent sweep would bury the journal /api/pg/deadlocks greps. GET /api/metrics, GET /api/metrics/csv.
  • HA Manager — log downloads. The point of this UI is being reachable when the console is not, so evidence has to be able to leave the box. Download buttons on the log viewer (20,000 lines), the predictive-trigger log and the rolling-update log: GET /api/logs/download, /api/ml/logs/download, /api/update/log/download. system journal (everything) joins keepalived and ha-manager as a viewer target.
  • HA Manager — diagnostics bundle (GET /api/bundle). One zip covering the whole control plane: every host's container logs, keepalived / ha-manager / system journals, dmesg, podman ps, Patroni state, network and filesystem state, plus this UI's own status, PostgreSQL, hardware and telemetry views and a SUMMARY.txt that restates the scheduler and redis-master invariants. Secrets are masked by default — assignment-form password/secret/token keys, connection-string passwords, authorization tokens, and keepalived's space-form auth_pass, since keepalived.conf is in the archive by design.
  • Maintenance mode for control-plane hosts (deployment/keepalived/mfcloud-ha-maint.sh, 🔧 button in the HA Manager). Draining a host by hand meant systemctl stop keepalived, which is worse than it looks: it removes the VIP but fires no notify_backup, so the host keeps Redis mastership and its beat-primary flag. The cluster ends up with two schedulers, an orphaned Redis master nothing replicates from, and a new VIP holder that is a read-only replica. Maintenance mode instead fails the keepalived health check, producing an ordinary priority-loss transition that runs the hooks exactly as designed. It also hands over the Patroni leader — the DB primary does not follow the VIP, so moving only the VIP still leaves a failover to pay for at shutdown — and does that first, while the host is still healthy, because a switchover is the step most likely to refuse and failing it after the VIP has moved strands the cluster half-migrated. The flag lives under /etc, not /run: a host taken down for maintenance must come back up still drained rather than seizing the VIP before anyone has confirmed it is fit.
  • HA role reconciler (mfcloud-ha-reconcile.timer, every 30s). Enforces the invariant the whole HA design rests on — the VIP holder is the Redis master and the scheduler, everyone else is a replica that schedules nothing — from the single local fact of VIP ownership. Only one host can answer "yes" to that, so all three run it independently with no election, no quorum and no way to fight; the cluster-wide invariant falls out of three purely local decisions, which is also why it is safe during a failover. Logs only when it actually repairs something, so a non-empty log means real drift.

Fixed

  • Health-based VIP failover had never worked, on any host, silently. keepalived runs its track and notify scripts in the SELinux domain keepalived_t, which is denied execute on podman (container_runtime_exec_t), getattr on /usr/sbin/ip (ifconfig_exec_t) and read on the compose-labelled .env (container_file_t). Because a denied access(X_OK) makes bash report "command not found" rather than a permission error, every one of those failures looked like an ordinary non-zero exit.

The result: check_frontdoor.sh had never once succeeded, so all three hosts sat permanently at base priority −20 with Status = BAD. A sick host could not shed the VIP — it only ever moved when a host died outright or keepalived was stopped. The same denial is why notify_master's Redis promotion no-op'd, leaving the VIP holder on a read-only broker for hours, and why maintenance mode's flag changed nothing: the check was already failing, so failing it again moved no priority.

Fixed without widening SELinux policy or disabling it — granting a root daemon the right to execute the container runtime is a real privilege expansion, and this had a structural answer instead. keepalived's scripts now do only file writes and curl; everything privileged moved to mfcloud-ha-reconcile.sh, which runs from an ordinary systemd unit outside keepalived_t. notify_mfcloud.sh writes /run/mfcloud-ha/vrrp-state and stops; a new mfcloud-ha-reconcile.path watches that file so a transition is applied sub-second; the reconciler publishes a redis-fit verdict that check_frontdoor.sh reads, with a staleness check so a dead reconciler reads as unfit rather than asserting health forever. SELinux stays Enforcing.

Verified live: VRRP_Script(chk_frontdoor) succeeded → Changing effective priority from 120 to 140, the first successful health check in this cluster's history, followed by a full maintenance drain and return with the console serving throughout.

  • The front-door health check passed a VIP holder that could not accept writes. It asked Redis what role it held and, on slave, took a cheap PING branch on the assumption that a replica must be a backup host. The invariant runs the other way: the VIP holder is supposed to be master, and the failure worth catching is exactly a VIP holder that is not. On 2026-08-22 a notify_master promotion silently no-op'd; the host held the VIP with a read-only replica, every Celery enqueue failed READONLY, and this check reported it green for hours — because the broken state was the one it used to decide the check was unnecessary. It now branches on VIP ownership, the only question whose answer cannot be corrupted by the fault being tested for. Verified by injecting the fault on a live host: the old check exits 0, the new one exits 1.
  • promote_redis was fire-and-forget. It reported the exit status of podman exec, which says nothing about whether Redis changed role, and a container still coming up mid-failover gave a single shot nothing to recover with — it logged neither success nor failure. It now retries three times and re-reads the role to confirm, so a warning is a genuine "this host cannot serve as primary" signal.
  • The HA Manager's "Fail over VIP" button ran systemctl stop keepalived — the precise action that caused the outage above. Replaced by the maintenance button, which drains the host properly and verifies the handover.

  • One host's HAProxy was a single point of failure for the whole control plane. All three hosts carried DB_HOST=<one host's IP>, so every app replica reached PostgreSQL through that one host's HAProxy — even though HAProxy runs as a sidecar on all three precisely so it need not. Powering that host off took the database away from the other two: /api/login returned 500 (psycopg2.OperationalError ... No route to host), which the UI renders as "invalid credentials", and repeated retries then hit the 5/min login rate limit and turned into 429s. Now DB_HOST=haproxy — the compose service name, resolved per-host to that host's own HAProxy container. Identical on every host, and dependent on no peer.

  • A VIP failover locked every browser out of the console. With TLS_MODE=internal each host ran its own independent Caddy CA — same CN string, different keys — so after a failover the console was served by a CA the browser had never trusted. Caddy also sends Strict-Transport-Security: max-age=31536000, and HSTS removes the "proceed anyway" escape hatch, so the result was a hard block with the cluster perfectly healthy: curl -k 200 on every host, correct SAN, right MAC in ARP, and a browser that would not connect. All three hosts now share one CA (the one already trusted in the field, so nothing new had to be installed), with the previously-issued leaf certificates cleared so they re-issue under the shared intermediate. Confirmed by draining a host and watching the VIP's issuing-CA fingerprint stay byte-identical across the move.

  • A stray quote in the host-card renderer broke the HA Manager's entire <script> block, blanking the whole page — 0.44.6.1's fleet-warning work left + warnHtml' in render(). Caught before it reached any host.

0.44.6.1 — 2026-08-22

Makes 0.44.6.0's new HA behaviour observable, and fixes a health check that was lying about it.

Fixed

  • beat's health check reported healthy on a host that was not scheduling. It searched every process's cmdline for celery and beat — but CMD-SHELL puts the check's own command string into a process cmdline, and that string literally contains both words. The check therefore always found itself. Confirmed live: an idling container on a non-VIP host with only sh and sleep running reported healthy. The needles are now split (b'cel'+b'ery') and the checker skips its own pid and its parent shell. This is the second time this check has been wrong in the same direction — 0.44.6.0 fixed it matching beat alone, which the wrapper's own beat-entrypoint.sh path satisfies.

It now reports on agreement with this host's role rather than "is scheduling". Since beat follows the VIP, two of three hosts are correctly idle at any moment, and marking those unhealthy forever puts a permanent red in podman ps that operators learn to ignore — on the one service where a real fault matters. Unhealthy is reserved for genuine disagreement: scheduling without the flag (a second scheduler double-firing every periodic task) or holding the flag without scheduling (nothing periodic fires at all, HA monitor included).

  • Two ## 0.44.5.5 sections in this file. Concurrent work on the same day shipped two unrelated changes under one version number. Merged into a single section with a sub-heading each, rather than renumbering: both really were released as 0.44.5.5, and inventing a version that never deployed would make the history less accurate, not more.

Added

  • HA roles on haman's host cards. Redis mastership and the beat scheduler both follow the VIP as of 0.44.6.0, but nothing showed where they landed — the only way to check was podman exec ... INFO replication on each host by hand. Each card now carries a redis master · N replicas / redis replica badge and a scheduler / not scheduling badge.

The badges are coloured by whether the role agrees with VIP ownership, not by the role itself: the VIP holder must be the Redis master (every client writes to redis://<VIP>:6379, and a replica is read-only) and must be the host scheduling. A backup that is not scheduling is the expected steady state and renders grey, not red — colouring it red would train operators to ignore red on the one panel where red has to mean something.

  • Fleet-level HA invariants next to the VIP holder. Checks that cannot be seen from any single host's card: zero schedulers (nothing periodic fires anywhere, including tasks.ha_monitor.check_ha_status, so VM HA failover stops silently), more than one scheduler (every periodic task fires N times), and zero or multiple Redis masters. Each host card additionally spells out the two states that are actually broken — holding the VIP while read-only, and holding the VIP while not scheduling — in words rather than colour.

Note

Verified this release: the console and haman both load zero external assets — no CDN <script>/<link>, no Google Fonts, all CSS @imports relative, with chartjs, gridstack, inter and xterm vendored under static/vendor/. The install renders its own UI with no outbound internet.

0.44.6.0 — 2026-08-22

Closes the two components that were not actually HA in the "HA control plane". The VIP itself was already fine — keepalived VRRP with a health-tracking script, priorities 150/140/130 against a -20 penalty, so a sick master drops below its healthiest peer. What sat behind it did not.

Fixed

  • Redis was three independent, non-replicated instances behind one VIP. Every client — the three app replicas, the three workers, and every compute node's /etc/mfcloud-worker.env — addresses the broker as redis://<VIP>:6379. A failover therefore repointed all of them at a different, empty Redis: queued-but-unstarted Celery tasks vanished with the result backend, and the old master kept orphaned data. The compose comment called this acceptable because "tasks are re-triggerable from the UI" — which stopped being true once VM HA failover and backups depended on that queue. The VIP holder is now the Redis master and the other two replicate from the VIP, promoted and demoted by keepalived on every VRRP transition.

Replication rather than Sentinel is deliberate: Sentinel would force every client onto sentinel:// URLs, and the clients include the whole compute fleet, whose broker URL is written by services/ha_provisioner.py and healed by routers/hosts/repair.py. Pinning the master to the VIP leaves every one of those URLs byte-for-byte correct and touches no node.

  • beat was the only non-HA component left. Scheduling was pinned to one host by a static IS_BEAT_PRIMARY=true. Correct about the hazard — N beats means N independent firings of all 18 scheduled tasks, which thrashed Ceph VM placement on 2026-08-10 — but it meant that if that host died the VIP failed over, the console kept serving, and nothing periodic fired at all: no telemetry, no backups, and no tasks.ha_monitor.check_ha_status, so VM HA failover stopped working exactly when it was needed. The scheduler now follows the VIP.

  • check_frontdoor.sh did not check Redis. Now that the VIP holder is the Redis master, a host whose Redis is dead or read-only would have kept the VIP while every enqueue against it failed. The check is now role-aware: on the master it writes a real key (SETEX on scratch DB 15), because PING would pass on a read-only replica — the exact state that must not hold the VIP; on a backup it only requires PING.

That split is not cosmetic. The first version probed a write unconditionally, which passes only on the master — so every backup failed the check permanently just for being a backup, burning its -20 before it was ever needed and making the check useless for telling a healthy backup from a broken one. install-notify-hooks.sh refused to wire .117 because of it, which is exactly what that guard is for. A replica is checked with PING rather than master_link_status on purpose too: when a master dies every replica's link drops at once, and demoting all of them at the moment one needs to take the VIP is backwards.

  • beat's healthcheck would have reported an idling container as healthy. It matched b'beat' in any /proc/*/cmdline, and the new wrapper's own cmdline is sh /app/docker/beat-entrypoint.sh. Now requires both b'celery' and b'beat'.

Added

  • docker/beat-entrypoint.sh — supervises celery beat against a flag file instead of exec-ing it, so losing the VIP can stop the scheduler. Idles rather than exiting while not primary (an earlier exit 1 guard restart-looped 7 times in 12 seconds under restart: unless-stopped).
  • deployment/keepalived/notify_mfcloud.sh — the VRRP transition hook. Promotes/demotes Redis and writes/clears /run/mfcloud/beat-primary. Idempotent, and never exits non-zero: refusing the VIP because Redis didn't promote would trade a degraded broker for a total outage.
  • deployment/keepalived/check_frontdoor.sh — now version-controlled rather than existing only on the hosts.
  • deployment/keepalived/install-notify-hooks.sh — idempotent per-host installer. Backs up keepalived.conf, adds only the notify_* lines and script_user root (never rewrites priority / unicast_src_ip / unicast_peer, which are per-host and would break the election), refuses to wire a host whose health check fails, and refuses to reload a config keepalived itself rejects — a bad config here means no VIP anywhere.

Changed

  • IS_BEAT_PRIMARY is now a single-node override only. On HA it must be false on all three hosts; VRRP decides. services/ha_provisioner.py writes false to all three (it previously wrote true to host 1), and leaving it true on an HA host would pin a second scheduler there after it lost the VIP — the very failure the key was introduced to prevent.
  • The beat flag is bind-mounted read-only from /run (tmpfs), so a rebooted host comes up not scheduling until VRRP says otherwise, and the container can never promote itself. It remains the only mount beat gets; still no SSH key.

0.44.5.6 — 2026-08-22

Networking correctness pass — closes a handful of silent failure modes in IPAM/OVN port lifecycle and port-forward rules — plus a UI modernization pass on the Networks tab bringing it in line with the Host UI's shared modal components.

Fixed

  • Deleting an IP pool could hand its addresses to a second VM while the original VM's OVN port still held them. Pool deletion cascades its leases in the database, but never released the corresponding OVN logical switch ports first. DELETE /api/networks/ipam/pools/{id} now tears down each affected VM's OVN port in the background — mirroring the same "only on the VM's last active lease" check release_lease already used — before the pool disappears, closing the duplicate IP/MAC window.
  • Deleting a VM left its port-forward firewall rules running forever. VM delete only removed the port_forwards database rows, never the actual firewall-cmd NAT rule — a dangling forward that could misroute traffic to whichever VM later reused the freed IP. VM delete now schedules the same teardown routers/networks.py's own delete-port-forward route uses.
  • A firewalld reload or host reboot silently dropped every port-forward rule. apply_port_forward/remove_port_forward only ever touched the runtime rule set, never --permanent — unlike OVN's ACL self-heal, there was no recovery path. Both now apply/remove the permanent rule alongside the runtime one. Removal also now resolves the exact live firewalld rule instead of recomputing it from the VM's current DHCP lease, which could be stale (or, for a VM already deleted, nonexistent).
  • Removed two Celery tasks (create_network, forward_port in tasks/network.py) with no remaining callers anywhere in the app — superseded by the OVN/bridge and apply_port_forward paths.
  • create_port_forward/delete_port_forward now roll back and return a clean error on failure instead of an unhandled exception, matching every other handler in routers/networks.py.
  • Added the missing VLAN ID (0–4094) and MAC address format validators to NetworkCreate/IPAllocate — a malformed value used to reach the database/OVN layer before failing with a confusing downstream error.

Changed — Networks UI

  • Replaced the Networks tab's raw static modals (Security Group create/rule/ assign-VM, IP Pool create/allocate, L2 Bridge create, Router-connect, Port Group create) with the shared mfFormModal/mfPickModal components already standard on the Host UI.
  • Destructive deletes with real blast radius — OVN Logical Switch/Router, Security Group, L2 Bridge, IP Pool — now use the typed-confirm mfDangerConfirm dialog instead of a plain yes/no prompt. Switch and pool deletes fetch the actual affected pool/lease counts first and show them in the confirmation.
  • Added a filter box that narrows all 6 Networks tables by text match, an IPAM pool usage bar, and client-side CIDR/MAC validation on the forms above before they hit the server.

0.44.5.5 — 2026-08-22

Two unrelated changes shipped under this version — concurrent work on the same day, merged here rather than renumbered, because both really were released as 0.44.5.5 and inventing a version that never deployed would make the history less accurate, not more.

Offline console — vendored frontend libraries

The console no longer needs outbound internet to render its own UI. Every third-party frontend library is now served from the install itself.

Fixed

  • Telemetry charts, every terminal, and the Settings widget grid silently died on any install without internet access. Chart.js, xterm (plus its fit-addon), GridStack and the Inter webfont were all fetched at page load from cdn.jsdelivr.net and Google Fonts — unreachable on an air-gapped or egress-filtered management VLAN, which is the normal placement for a hypervisor control plane. Because the frontend guards these globals (typeof Chart === 'undefined' and friends), the failure mode was silence rather than an error: blank telemetry on hosts, VMs, ZFS, datastores and Ceph; a dead host terminal and VM console; dead Ceph and ZFS live logs; and a Settings page whose widgets could not be moved or resized — with nothing on screen explaining why. All four dependencies are now vendored under static/vendor/ and served locally.

Changed

  • Chart.js is now version-pinned. The page requested npm/chart.js with no version, so each browser load took whatever the CDN considered latest — an upstream release could have altered charting behaviour in production with no corresponding change in this repo. Frozen at 4.5.1, byte-for-byte what was previously being served. xterm 5.3.0, fit-addon 0.8.0 and GridStack 13.0.2 were already pinned and are unchanged.
  • Serving these ourselves also closes an unpinned third-party fetch into the page context of an admin console — the CDN tags carried no Subresource Integrity hashes.
  • Added scripts/update-vendor.sh to refresh the vendored set from the version pins at the top of that script. It regenerates the self-hosted Inter stylesheet and hard-fails if any remote URL survives the rewrite, so the offline guarantee cannot regress unnoticed on a later upgrade.

CRS predictive models

The CRS predictive models were measuring ~9x worse than doing nothing. The models were fine; the evaluation was broken.

Fixed

  • The validation split was an accidental host holdout, not a time split. train_and_store sliced the last 15% off the POOLED feature array and a comment called it "time-based" — but _build_features appends rows host by host, so on this fleet that slice trained on six hosts and validated exclusively on the two that happened to sort last by IP string. Those two sit at 34%/65% RAM against 5-14% for the training hosts, so the reported RAM MAE of ~37 was almost entirely that level offset rather than prediction error. The split is now taken per host, on the tail of each host's own timeline. Measured effect on the CPU model, on 7 days of real telemetry: -322% skill → +17%, i.e. from far worse than assuming no change to genuinely useful.
  • The training objective didn't match the reported metric. MAE was reported and gated on while the model optimised squared error. Switched to reg:absoluteerror — a further +6 points of skill on CPU (to +23%).
  • A single dropped telemetry scrape permanently disqualified a host from being predicted. _host_feature_row demanded 30 perfectly unbroken one-minute buckets, stricter than the _MAX_GAP_MIN tolerance training itself applies, so one missed sample meant no forecast at all — observed live with 192.168.14.12 returning nothing while both cluster peers had predictions. Inference now uses the same gap rule as training.

Added

  • Models are now trained on a delta target (change from now, with now added back at predict time), recorded per model as crs_ml_models.target_mode. Persistence becomes exactly representable, so a model starts at baseline parity instead of having to relearn each host's level — and a newly-added host at an unfamiliar load level can no longer be anchored to the wrong one, which is precisely what the split bug exposed. Existing rows default to 'level' and are still read correctly.
  • A promotion gate. predict_fleet no longer serves a model that fails to beat the naive "predict no change" forecast by CRS_PREDICT_MIN_SKILL (default 5%). Below-bar models are still trained, stored and shown, so an operator can see why they aren't in use. Measuring MAE without acting on it is what let a model 9x worse than persistence sit in the serving path: the number was on screen and correctly flagged, and it was consulted anyway. The cluster panel's model line now says "in use" or "not in use — below the 5% skill bar" alongside the existing skill percentage.
  • scripts/crs_model_eval.py — read-only offline harness that scores candidate split/target/objective combinations against the same real host_telemetry without touching crs_ml_models. It shares production's own feature builder so the two can't drift. Run it before changing anything in core/crs_forecast.py.

Note

On this fleet the RAM model lands at roughly baseline parity even fully fixed, because RAM barely moves in 15 minutes (naive baseline MAE 0.154 percentage points) — there is no signal to learn. It will therefore train, store, and be held back by the skill gate rather than served. That is the intended outcome, and CRS's trigger reads cpu_pred only regardless.

0.44.5.4 — 2026-08-21

CRS (Compute Resource Scheduler) correctness pass, plus the first UI answer to "CRS is enabled — why isn't it doing anything?".

Fixed

  • CRS picked the roomiest destination host, not the coolest one. _pick_target sorted candidates by free memory first and by load only as a tiebreak — but remaining_mem is an MB-resolution number that is essentially never equal between two hosts, so the load term was unreachable and the "prefer the coolest destination" logic (including the whole predictive-forecast tiebreak) never ran. CRS relieved a hot host by moving onto whichever host merely had the most RAM free, which is core/placement.py's question ("will a NEW VM fit"), not this one's. Destinations are now ranked by the axis actually under pressure, with free memory as the tiebreak.
  • CRS could migrate a VM onto a host with no room for it. The same function returned its top-ranked candidate unconditionally, ranking non-fitting hosts last rather than excluding them — so when nothing in the cluster had capacity it still handed back an over-committed destination. Hosts without room are no longer eligible at all.
  • A move was dropped when its first-choice destination failed the anti-ping-pong floor, even when another legal, meaningfully-cooler host was available. The accept loop now walks the ranked destinations instead of testing only the best one.
  • The VM-count fallback trigger counted powered-off VMs. A host with one running and five stopped VMs outranked a host running three, so CRS called the idle host "hot" and migrated its only running VM onto the host actually doing the work. It now counts running VMs only — the same recent-telemetry test the candidate filter already used.
  • Dismissing a CRS recommendation was effectively a no-op. The duplicate-suppression check only looked at pending rows, so the next sweep (5 minutes later) re-derived the identical move and queued it again. A dismissal now suppresses that VM for CRS_DISMISS_COOLDOWN_MIN (4h).
  • Nothing paced CRS passes against each other. CRS_MAX_MOVES_PER_PASS caps one pass, but a live migration routinely outlives the 300s sweep interval, and until it finishes the source host is still hot — so each pass accepted 2 more moves on top of the ones still running. Periodic passes now stand down while CRS_MAX_INFLIGHT_MIGRATIONS (2) migrations are already landing in the cluster.

Added

  • CRS "Last pass" line in the cluster panel. Every pass now records its own verdict and measurements (clusters.crs_last_pass_at / crs_last_pass_result) — the engine always computed them and then discarded them, so an enabled-but-idle CRS could previously only be explained by reading worker logs. The panel now distinguishes "balanced, CPU spread 4% against a 20% threshold" from the very different "a trigger fired but no eligible VM or destination was found".
  • "Preview next pass" dry run (POST /api/inventory/clusters/{name}/crs/preview). Runs the real decision engine with dry_run=True — same telemetry, triggers and ranking — but writes nothing, queues nothing and migrates nothing. On a cluster at Full automation it's the only way to see the next move before it happens.
  • First automated tests for the CRS decision engine (tests/test_crs_balance.py), covering the three fixes above whose failure mode was silent.

0.44.5.3 — 2026-08-21

Code-review follow-up on 0.44.5.2. The manifest force-sync that shipped there works, but only for the one stack it was written for — this generalises it before a second stack needs it and quietly doesn't get it.

Fixed

  • Builtin manifest fixes would have stopped reaching upgraded hosts after vLLM. 0.44.5.2's force-sync in core/app_stacks.py was a hand-maintained one-entry dict ({"vllm": _VLLM_YAML}). Because the seed INSERT is ON CONFLICT (stack_id) DO NOTHING, the next builtin whose template got fixed — Grafana, Portainer, any of the ~20 — would have shipped in the file and silently never reached a single already-live install, unless someone remembered to append to that dict. That is the same failure mode as the vLLM /dev/shm bug it was added to fix. The sync is now driven off the seed list itself, so every builtin is covered automatically and there is nothing to remember. Helm-backed builtins (manifest_template IS NULL) are skipped; the previous comment also claimed to sync helm_values_template, which it never did and which seed does not carry.
  • Concurrent cluster delete during a firewall reconcile returned 500 instead of 404. POST /{cluster_id}/reconcile-firewall dropped its not-found guard on the grounds that _require_healthy_cluster had already proved the row exists — but that helper opens and closes its own connection, so the proof does not carry into the following transaction. A DELETE /{cluster_id} (or the delete task finishing) landing between the two made c.fetchone()[0] raise TypeError. Guard restored.

Known issue (not fixed this pass)

  • rke2/Canal clusters permanently report Partial on firewall reconcile. _CNI_TRUST_IFACES in tasks/kubernetes.py requires both cni0 and flannel.1, but rke2's default Canal CNI creates flannel.1 and never cni0 — as that file's own comment notes. Every rke2 node therefore reports trusted=1 < total=2 forever, and the operator is told "Re-run once their CNI is up" on every bootstrap and every manual reconcile, for a CNI that is already up and a retry that can never change the outcome. Not fixed here because the constant is interpolated into the shell script executed over SSH on each node, and SSH-path changes need separate sign-off. The fix is to key the interface list off the engine, or report a terminal marker rather than a retry instruction.

0.44.5.2 — 2026-08-20

Closes the V41-G2 remainder: Ollama + Open WebUI, ComfyUI, and vLLM had never had a real Deploy click against a live cluster. Stood up a disposable single-CP K3s cluster and deployed all three for real. Ollama + Open WebUI and ComfyUI came up clean on the first attempt. vLLM did not — two real bugs found and fixed.

Fixed

  • vLLM catalog stack crash-looped on every deploy. Two independent causes, both in core/app_stacks.py's manifest template: (1) vLLM's multiprocess engine-core IPC ring buffer needs real /dev/shm, which the manifest never provisioned — containerd's default 64Mi immediately raised RuntimeError: Insufficient space in /dev/shm: 160 MiB required, 64 MiB free even for the smallest catalog default model (facebook/opt-125m). Added a dshm emptyDir{medium: Memory, sizeLimit: 1Gi} volume — a tmpfs size limit, not a reservation, so it costs nothing unless written. (2) On vLLM's CPU backend, --gpu-memory-utilization (despite the name) sets the fraction of system RAM to reserve and was left unset, defaulting to ~0.9 — on a node also running other pods this demanded far more memory than was actually free and refused to start. Pinned it to 0.4. Both fixed and live-verified: the OpenAI-compatible /v1/models endpoint responds, 0 restarts sustained. Existing installs get the fix via a new idempotent manifest_template force-sync in ensure_app_stacks_schema() (builtin catalog rows are already API-immutable by design — see _BUILTIN_LOCKED_FIELDS — so the file is the sole source of truth and a startup sync is safe).
  • Host KVM-DCTCK93's /var (VM image datastore) had no headroom. Unrelated to the above, hit while diagnosing it: /var was a fixed 80GiB LV with no unallocated space cushion, and a large thin-provisioned K8s worker disk (needed to work around the K8S-TMPL template's own gap, below) drove it to 100% full mid-deploy, pausing both test VMs on a QEMU nospace I/O error. Extended /var into the VG's already-unallocated free space live (80GiB → 165GiB, zero data loss, XFS online grow) — no other VM on the host was affected, but this was close to a real incident for everything else sharing that datastore.

Known issue (not fixed this pass)

  • K8S-TMPL's partition table ignores the requested disk_gb. Every K8s node deploy gets a fixed ~18GB LVM footprint (root=10G/swap=1G/ home=2G/var=5G) regardless of the VM's actual virtual disk size — growpart+resizefs are configured in the template's cloud-init but don't extend an LVM-on-partition layout on their own. kubelet's ephemeral-storage allocatable also doesn't refresh from a live resize without a k3s-agent restart. Worked around live for this pass (growpart + pvresize + lvextend + xfs_growfs + agent restart); the template itself still needs either a proper LVM --grow kickstart or a first-boot growpart/lvextend script in tasks/kubernetes.py's node provisioning so future K8s deploys don't silently get capacity-starved.

0.44.5.1 — 2026-08-20

Fixes thirteen findings from a code review of 0.44.5.0 — nine of them defects that release introduced. Three were live in the Atlas UI; the rest sat in the Kubernetes paths, which had never executed (no clusters exist on this deployment), so nothing had gone wrong yet. Two of them would have.

Fixed

  • Atlas: adding or removing a custom Field did nothing visible. 0.44.5.0's focus guard — added so a save couldn't rebuild the Inspector out from under someone mid-keystroke — treated any focused element in the panel as "user is typing", and browsers focus a <button> on click. The Fields + and × buttons live in that panel and do no DOM work of their own; they rely entirely on the re-render to show their result. So the PATCH succeeded, the row never appeared (or never disappeared), the key/value inputs never cleared, and operators clicked again and re-sent it. The guard now protects only text entries, which are the only controls that can hold unsaved keystrokes.
  • Atlas: Save could report the wrong outcome, or none at all. atlasSave() awaited a module-global holding the most recent PATCH from any node or field, so a stale false from an earlier rejected edit made Save show no toast (looking dead), and a stale true made it announce "Board saved." when that click saved nothing. It now discards any earlier result before blurring and reads back only what its own blur started; nothing to save is treated as success rather than failure.

  • Atlas: a tags edit didn't update an active search. tags is deliberately not in ATLAS_BOARD_FIELDS (it doesn't affect how a node draws), so a tags edit skipped the board rebuild — and with it the filter re-apply that lives there. A node could stay dimmed after being given the tag being searched for. Filter re-application is now its own step, run on every patch rather than riding along with a rebuild.

  • The pod-CIDR firewall grant could be removed with nothing narrower in its place. The narrowing script counted an interface as trusted without checking whether firewall-cmd --add-interface had actually worked. On a host where the interface was already bound to another zone (ZONE_CONFLICT) every add failed, the count still reached its threshold, the broad grant was removed, and the task reported success — silently dropping cross-node pod traffic on a working cluster. Every step is now verified by querying the resulting state (--query-interface, --query-source) rather than trusting an exit code, which also makes repeat runs clean since --add-interface on an already-trusted interface is non-zero on some firewalld versions.

  • Narrowing no longer runs before a node's CNI is fully up, or on rke2 at all. It previously required only flannel.1, which two cases break: cni0 is created by the bridge CNI plugin only when the first pod is scheduled, so a freshly joined worker would be narrowed while missing it and never revisited; and rke2's default CNI is Canal, which creates flannel.1 but no cni0 and routes local pod traffic over per-pod cali* veths that cannot be named in a firewalld rule. Both now leave the working (wider) grant in place and report the node as not-yet-narrowed, which the reconcile endpoint can retry. An rke2 cluster keeps the broad grant by design until there is a Canal-aware rule to replace it with.
  • A failing reconcile script was reported as an unreachable node. Any response without a recognised marker was labelled unreachable, sending operators to check SSH on nodes whose SSH was fine. Script and environment failures are now reported separately and carry the actual output into the event log.
  • HA promotion no longer destroys operator-added TLS SANs. 0.44.5.0 made the promotion retry "re-assert" config.yaml by stripping and re-emitting the keys it manages — which threw away any tls-san entries the operator had added themselves. After the restart the regenerated apiserver certificate would no longer cover those names, breaking every kubeconfig, ingress and CI job pointed at them. Existing entries are now preserved and unioned with the VIP. The code this replaced could not lose them.
  • HA promotion no longer corrupts config.yaml outright. In the same rewrite, a blank line or comment inside the tls-san list ended the list early and left the remaining items orphaned under no mapping key — invalid YAML that the engine refuses to start with, reproduced identically on all three retry attempts, leaving the control-plane down with no self-healing path. Blank lines and comments inside the list are now handled as part of it.

Changed

  • POST /{cluster_id}/reconcile-firewall now applies the k8s licence gate every other mutating Kubernetes endpoint already had, and uses one database connection instead of two (the second lookup's not-found branch was unreachable — the health check ahead of it already 404s).
  • reconcile_k8s_cluster_firewall gained a soft_time_limit, and the per-node budget is shorter now that a not-yet-ready CNI is an expected outcome rather than something to wait out. Nodes are walked sequentially with their own retry budgets, so a cluster carrying stale worker_ips entries could otherwise hold a worker slot for a very long time.
  • The jsonb-or-scalar control-plane list parsing is now a single _jsonb_list helper shared by _cluster_node_ips and _connect_any_cp, proven equivalent to both original spellings across the empty/null/legacy cases. scale_up_k8s_control_planes keeps its own spelling on purpose: its fallback applies only when the column isn't already a list, and changing that would alter behaviour in a working path for no functional gain.

Not fixed here

  • static/js/vm_settings.js offers Resize on a boot-position RDM passthrough disk, which the backend always rejects; the non-boot branch guards this case. From 0.44.4.9, not this work.
  • haman/haman.py's GPU parser is a copy of tasks/gpu.py's and the two have already diverged in where the onboard/BMC exclusion happens.

0.44.5.0 — 2026-08-20

Ships the 0.44.4.8 code-review fix set, which was written but never actually deployed: the three HA nodes went 0.44.4.7 → 0.44.4.9 on a separate release while 0.44.4.8's files sat undeployed in the working tree, so the nodes have been reporting a version whose changelog entry describes fixes their code didn't contain. Nothing in 0.44.4.8's entry changed — see it below for the full detail on all eighteen fixes. This release is that code reaching the nodes, plus the version bump it needs to get there.

Changed

  • Version bump is load-bearing, not cosmetic. templates/index.html cache-busts static assets as /static/js/atlas.js?v={{ app_version }}, and the nodes have been serving ?v=0.44.4.9 since this morning. Every browser that opened the Topology tab today holds a cached copy of the unfixed atlas.js — the one where clicking a node instantly deselects it. Without a new version in the query string those browsers keep that cached file and the fix is invisible no matter what is on disk.
  • The in-code references to when the pod-network firewall reconcile starts running automatically now name 0.44.5.0 rather than 0.44.4.8, so POST /{cluster_id}/reconcile-firewall's "clusters that predate it" is accurate against what is really deployed. Any cluster built before this release still carries the wide pod-CIDR grant and needs that endpoint run against it once.

0.44.4.9 — 2026-08-20

Fixed

  • Boot disk could never be resized from the VM Settings modal. _vmsRenderDisks() (static/js/vm_settings.js) rendered only a "Boot disk" tag for the boot disk row regardless of power state, never the Resize button every other disk gets — even though PATCH /{vm_id}/disks has no boot-disk restriction (only "VM must be powered off"). The boot disk is the one people most often need to grow. Now shows Resize alongside the tag when the VM is off, and "Power off to resize" when it's running, matching the existing non-boot-disk behavior.
  • Corrected a stale docstring on POST /{vm_id}/disks (routers/vms/settings.py) claiming it required a powered-off VM — tasks.settings.disks.add_disk has hot-attached via VIR_DOMAIN_AFFECT_LIVE for running VMs all along; only the boot-disk-resize gap above was real.

0.44.4.8 — 2026-08-20

Code-review pass over 0.44.4.2-0.44.4.5 (the Atlas board editor and the HA control-plane rework). Two of the findings are regressions those releases introduced; the rest are defects the same review turned up alongside them.

Fixed

  • Day-2 control-plane add always failed on bare-metal nodes. 0.44.4.5's new _wait_for_node_ready check was handed the caller's VM label and used it as the Kubernetes node name. Those are not the same thing: _resolve_node returns the bare IP as the label for source='host', and whatever vm_name the operator typed for source='existing', while k3s/rke2 name a node after the guest's HOSTNAME (this module passes --node-name nowhere). kubectl get node 192.168.7.42 is NotFound forever, so every bare-metal CP add burned the full timeout and then reported a failure even when the join had worked perfectly. Only source='new' lined up, and only incidentally — build_user_data writes hostname: {vm_name} into the cloud-init seed — which is why 0.44.4.5's live test, run entirely on provisioned VMs, passed. The check now matches on InternalIP via the existing _kubectl_get_nodes helper (the same reason scale_down_k8s_worker already resolved names through _find_node_name_by_ip), which works for all three node sources and drops a fragile inline jsonpath expression.
  • A slow join could leave a control-plane in the cluster but not in the database. The Ready-timeout raised before any bookkeeping ran, so a node that was already a voting member of the etcd quorum was left out of control_plane_ips/control_plane_vms/node_count and its VM stayed marked free for another cluster to claim — with the task then reporting Partial and setting the cluster back to healthy, pointing at nothing. This wait is defense in depth, not the authority on whether the join happened: bookkeeping now runs whenever the join script itself succeeded, and a timeout downgrades to a warning event naming the node to verify.
  • A non-fatal firewalld error silently skipped firewall-cmd --reload. The install-time firewall setup chained every step with &&, so one recoverable failure — a ZONE_CONFLICT from 0.44.4.4's new --add-source=10.42.0.0/16 on a host where that CIDR is already bound to another zone — short-circuited the chain before the reload, and the trailing || true hid it. The --add-port rules for 6443/8472/10250/2379-2380 stayed staged in the permanent config and never reached the runtime firewall, reproducing exactly the :6443 join failures those rules were added to prevent. Each step is now independent; only the "is firewalld present and running" guard still gates the block.

  • HA promotion's config retry didn't actually re-assert the config. 0.44.4.5 removed the early return on an existing tls-san: line but kept the grep-guarded append, so attempts 2 and 3 restarted the service without rewriting the file — leaving the one failure mode the retry loop was added to catch (a stale or half-written config.yaml) unrepaired. The managed keys are now stripped and re-emitted on every attempt, with the operator's other keys passed through untouched and the new file staged and moved into place only on success.

  • Atlas: a node click selected and then instantly deselected itself. The viewport's pointerup handler ran atlasDeselectAll() off a moved flag belonging to a gesture it never started — a node's pointerdown calls stopPropagation(), so the viewport's own pointerdown never ran, but the pointerup still bubbled. Selection had been an inline onclick (which fires after pointerup, and so survived) until 0.44.4.2 moved it into pointerup. Net effect: none of the Inspector editing UI that release shipped was reachable. The handler now ignores pointerups from gestures it didn't begin.

  • Atlas: at low zoom, a click was treated as a drag. The 4px drag threshold was compared against deltas that had already been divided by zoom, making it mean 4/zoom screen pixels — at atlasFitToView's lowest zoom (0.03) a 0.12px tremor counted as a drag, so the click never selected the node and a PATCH wrote a garbage position. The threshold is now compared in screen space; zoom is applied only to the movement actually given to the node.
  • Atlas: editing a name containing ' or \ corrupted it, worse each time. escapeJsAttr (which JS-escapes before HTML-escaping, correct only for a JS string nested inside an attribute) was used for plain value="..." attributes. A node titled Rafael's box rendered as Rafael\'s box, compared unequal on blur, and PATCHed the backslashed form back — gaining a backslash per edit. Those five call sites now use escapeHTML; escapeJsAttr remains where a JS string really is nested in an attribute.
  • Atlas: typing in one Inspector field could be discarded by the previous field's save. The PATCH's success handler replaced the whole Inspector innerHTML, so tabbing Title → Subtitle and typing had the live input removed mid-keystroke when the Title save landed. The re-render is now skipped while focus is inside the Inspector (the error path still re-renders unconditionally — a rejected edit must not sit in the form looking accepted).
  • Atlas: structured field values were flattened to strings. The fields column is JSONB and the schema is Dict[str, Any], so a value can be a number, bool, array or object; the editor String()-ed those into a text input and PATCHed the result back, permanently turning [80, 443] into "80,443" and an object into "[object Object]". Non-string values now render read-only as JSON.
  • Atlas: "Board saved." was shown before the save happened. The toast fired synchronously right after blur(), so clearing a title to empty produced a green success toast racing the red 422 from the same click, with nothing saved. Save now awaits the PATCH its own blur() triggers and stays quiet when that PATCH failed (the failure already surfaces the server's own error).
  • Atlas: an active search filter was silently dropped. atlasRenderAll() rebuilds the node layer's innerHTML, discarding atlasFilter's dimmed classes, so any selection, drag or field edit left the board showing the unfiltered set while the search box still read as filtered. The filter is now re-applied after a rebuild.
  • Atlas: a cancelled gesture left the board stuck. Neither the node drag nor the connector drag handled pointercancel, which arrives instead of pointerup when the browser takes the pointer away (touch gesture, contextmenu, capture loss) — leaving dragging true so the node followed the cursor with no button held, or the dashed draft connector stranded on a canvas stuck in its crosshair state. All three gestures now reset on cancel.
  • Atlas: spurious PATCHes on untouched fields. Nullable columns (subtitle, tags, edge label) arrive as null while an input always yields a string, so the === guard never matched and merely focusing and leaving a field wrote to the server; the Fields rows had no guard at all, so tabbing through N rows fired N PATCHes, each rewriting the whole fields jsonb. Both now compare with ?? ''.

Changed

  • Atlas edits that aren't visible on the board (tags, fields, parent) no longer trigger a full atlasRenderAll(). That rebuilt every node div, re-registered every pointer listener and forced a synchronous getBBox() layout per labelled edge for a change nothing on the canvas shows; the Inspector re-render alone covers it.
  • The initial-bootstrap control-plane join now carries a comment explaining why it still trusts the install script's exit code where the Day-2 path does not — bootstrap installs CP1 with --cluster-init from the outset and never calls _promote_cp1_to_ha, so the race that motivated the Day-2 wait cannot occur there.

Security

  • The pod CIDR is no longer blanket-trusted on cluster nodes. 0.44.4.4 put 10.42.0.0/16 into firewalld's trusted zone to unblock decapsulated cross-node pod traffic. That zone is ACCEPT-all, so the side effect was that any pod — holding a legitimate 10.42.x.x source address — could reach any port on any node, straight past the deliberately narrow --add-port allowlist written directly above it. A single untrusted tenant container was a full-node network client. The source grant only existed because flannel.1/cni0 don't exist yet at install time and so can't be named there. A new post-install reconcile now trusts those interfaces by name once the CNI has actually created them and then removes the CIDR-wide grant. It is dispatched automatically after every install path, and is idempotent.

The broad grant is removed only after flannel.1 — the VXLAN device where the decapsulated cross-node packet lands, i.e. the exact traffic 0.44.4.4 fixed — is confirmed trusted. A node whose CNI interfaces haven't appeared keeps its working (wide) rules and is named in the event log rather than being quietly cut off, so a slow or unusual CNI can never turn a security tightening into an outage.

Added

  • POST /api/kubernetes/{cluster_id}/reconcile-firewall (global_admin) — re-applies the pod-network firewall rules across every node of a cluster. New clusters do this on their own; this endpoint is for the clusters that predate 0.44.4.8, which were provisioned with the wide grant and had no way to ever lose it, and it doubles as the retry hook when a node's CNI wasn't up yet on the first pass. The task reports per-node which nodes it narrowed and which it skipped, and why.

0.44.4.7 — 2026-08-20

Fixed

  • beat's primary-election guard defaulted fail-open. docker-compose.yml's beat service refuses to run the Celery scheduler unless IS_BEAT_PRIMARY=true in its .env — the guard exists because two beat instances double/triple-fire every periodic task (real incident, 2026-08-10). Investigation confirmed both provisioning paths (.env.example for single-node, services/ha_provisioner.py per-host for HA) already set this key correctly and all 3 CP hosts' live .env are correct today — but the guard's default when the key is simply absent was true, meaning any future .env that lost the line (hand-edit, restored from an old backup, a host built outside either provisioning path) would silently become a second scheduler instead of silently refusing. Flipped the default to false (fail-closed) — a malformed .env now always idles instead of assuming primary. profiles: gating (the originally proposed fix) turned out not to work here: all 3 HA hosts share the identical compose file and --profile ha invocation, so a shared profile tag can't distinguish "the primary host" from "an HA secondary" the way it distinguishes "HA" from "single-node."

0.44.4.6 — 2026-08-20

Added

  • App Catalog: 8 new curated stacks, up from 15 to 23. Nextcloud and DokuWiki (Productivity), Paperless-ngx (Productivity — document management with OCR, deployed as a Pod with a bundled Redis sidecar since it hard-requires a broker and the catalog has no way to wire two stacks together), Prometheus (Monitoring, pairs with the existing Grafana entry), MariaDB and Metabase (Databases), and Pi-hole and Nginx Proxy Manager, seeding a new Networking category. Every entry's env vars and mount paths were checked against the image's own current docs before being written — this caught two things that would have otherwise shipped broken: Nextcloud's NEXTCLOUD_ADMIN_* auto-setup env vars silently do nothing without a full external database (SQLite installs go through Nextcloud's own first-visit wizard instead, same as Gitea/n8n's pattern), and Pi-hole v6 renamed WEBPASSWORD to FTLCONF_webserver_api_password and needs FTLCONF_dns_listeningMode=ALL to answer DNS queries that arrive over Pod networking rather than what it considers a "local" interface.
  • App Catalog UI redesign: single-column list → 2-column card grid. New .mkt-app-card component (components.css) mirrors the existing .k8s-card pattern (kubernetes.css) — a full-bleed top accent rule and hover-lift, but colored per category (MKT_CATEGORY_COLORS in marketplace.js) rather than by health state. Modal widened 560px → 760px to fit two columns comfortably.

0.44.4.5 — 2026-08-20

Follow-up to 0.44.4.4's 1→HA promotion note: that write-up said the path "completes successfully end-to-end" but was actually describing a false positive — a deeper live-testing pass found the real bug and a proper fix.

Fixed

  • A failed 1→HA control-plane promotion request permanently disabled itself from ever succeeding on retry. POST /{cluster_id}/control-planes set k8s_clusters.ha_enabled = TRUE unconditionally, synchronously, the moment a promotion was requested — not once _promote_cp1_to_ha (the step that actually converts control-plane 1's datastore from SQLite to embedded etcd) had actually succeeded. scale_up_k8s_control_planes only runs that conversion step when not ha_enabled, so after any first failed attempt, every later retry silently skipped it — joining new control-planes against a CP1 that was never actually running etcd. Every joiner failed identically and permanently with k3s's own "etcd disabled" fatal error, no matter how many times the operation was retried, and the earlier 0.44.4.4 write-up's "false failure that self-heals" read of this symptom was itself wrong — traced there to a genuinely separate, transient VM-boot hang on the lab host that happened to coincide with the real bug on the very first live-testing attempt. ha_enabled now only flips true inside the task, after control-plane 1 is confirmed listening on :2379 and kube-vip has deployed; the VIP lease itself still happens eagerly at request time (unchanged) so a retry reuses the same lease instead of leaking a second one.
  • _promote_cp1_to_ha itself now verifies its own result (etcd actually listening on :2379) rather than trusting that the write-config + restart + /readyz-answers sequence succeeding means the datastore migration took — /readyz answers just as fast from an apiserver that was never actually restarted, which is exactly what silently happened the first time this was live-tested. Retries the whole sequence up to 3 times, 10s apart, before giving up.
  • The control-plane join step now waits up to 90s for a joining node to actually show Ready in kubectl get nodes before trusting the join script's one-shot exit code as a hard failure — defense in depth for any other transient join-time race, independent of the ha_enabled fix above.

Live-verified clean end to end after the fix: a single-CP cluster promoted to 3-CP HA with both new control-planes joining successfully on the first attempt (2/2 joined, zero retries needed), ha_enabled correctly true only after kube-vip deployed, all 3 nodes Ready with the etcd role.

0.44.4.4 — 2026-08-19

Live-testing the V40 remainder against a real 3-node HA lab (single-CP, 3-CP k3s + rke2, LINSTOR CSI, MetalLB, quota 409, CP-kill VIP failover, Day-2 CP add/remove, 1→HA promotion — full checklist in planned.txt) found two real bugs, both fixed here.

Fixed

  • Firewalld silently dropped all cross-node pod/Service traffic on every HA or multi-node K8s cluster. The install step opened 8472/udp so the VXLAN envelope itself got through, but nothing ever trusted the decapsulated inner pod-to-pod packet once it landed on the receiving node's default firewalld zone — confirmed live via explicit "packet filtered" ICMP replies between two pods on different nodes, which surfaced as the apiserver returning a 502 while proxying to a webhook pod that happened to schedule on a different node from the one serving the request (hit first via MetalLB's own admission webhook, but this blocked any cross-node pod networking, not just MetalLB). Both k3s and rke2's firewall setup now also trust the shared k3s/rke2 default pod CIDR (10.42.0.0/16 — this codebase never sets --cluster-cidr) in firewalld's trusted zone.
  • A successful Day-2 control-plane add left the cluster permanently stuck out of healthy. _resolve_node (shared by the initial create-cluster bootstrap and the CP-add scale path) flips k8s_clusters.status to provisioning as a side effect whenever it provisions a new node — correct for the original bootstrap, which resets it to healthy on its own completion, but scale_up_k8s_control_planes never did the same. Every storage/networking/scale/CP endpoint gates on status == 'healthy', so a cluster that had just successfully grown from 3→5 control planes could never be touched again — not even to remove a CP or reconfigure storage — until this fix. Worker scale-up was unaffected (it calls the lower-level _provision_node directly, not _resolve_node).

Verified live (previously untested)

  • LINSTOR CSI: full end-to-end pass — a real Pod wrote and read back data through a PVC bound to a DRBD-replicated volume, not just "the CSI deployment rolled out." First time this path has been exercised against real registered satellites.
  • CP-kill VIP failover: clean ~4s recovery via kube-vip once isolated from an unrelated disk-exhaustion confound on the first attempt.
  • Day-2 CP add/remove, including the quorum-refusal path (a genuine simultaneous 2-of-4 CP loss correctly refused further removal with "Only 0/4 control-planes are Ready").
  • 1→HA promotion (_promote_cp1_to_ha) does complete successfully end-to-end, but the first attempt hit a transient race where the new control-planes' join attempts outran control-plane 1 finishing its SQLite→embedded-etcd migration — worth a closer look at add-CP retry/ readiness-waiting before fully trusting this path unattended.
  • rke2 3-CP HA smoke test: passes, same shape as k3s.
  • max_k8s_clusters quota 409: exact expected message on the second create attempt.

0.44.4.3 — 2026-08-19

Removed

  • Dead GET /api/networks/topology endpoint. Superseded by the Atlas board back when the Topology tab was rebuilt; no frontend code called it anymore. The _parse_vm_interfaces/_iface_matches_network helpers it shared with GET /{network_name}/summary are unaffected — that endpoint is still live and still uses them.

0.44.4.2 — 2026-08-19

Added

  • Atlas: drag, connect, and full editing. Nodes can now be dragged to reposition (position saves once, on release — not on every pointer move). Each node has a connect handle — drag from it to another node to draw a new connector, which opens the Inspector on the new connector so it can be labeled immediately. The Inspector is no longer read-only: Title/Subtitle/Status/Tags/Parent/Fields for a node and Label/Style for a connector all save in place (text fields on blur, selects and Fields rows immediately). This completes the interactive-editor loop the Topology tab rebuild set out to deliver.

0.44.4.1 — 2026-08-19

Fixed

  • Atlas: seeded boards could look empty even with data on them. The view always opened at a fixed pan/zoom regardless of where content actually sat, and Seed from Inventory laid every Cluster/Host/VM tier out in one unbounded row — a fleet with dozens of VMs produced a board thousands of pixels wide, most of it off-screen by default. The view now fits itself to whatever's actually on the board after every load and every seed, and seeding wraps into a grid instead of one long row.
  • Added "Clear Planning Nodes" — bulk-deletes every node not linked to real inventory (i.e. manually added via the palette) without touching anything Seed from Inventory created. Fixes not being able to individually grab nodes that got stacked on the exact same spot before drag-to-reposition existed.

0.44.4.0 — 2026-08-18

Added

  • Atlas: delete + Seed from Inventory. Selecting a node or connector in the Topology tab's Inspector panel now offers a Delete button (deleting a node also removes anything nested under it and any connectors touching it — the same cascade the database already enforced, now reachable from the UI). A new Seed from Inventory toolbar action auto-populates the board with real Clusters, Hosts, and VMs — connected Cluster → Host → VM — instead of building the same graph by hand. Safe to run repeatedly: already-seeded objects (matched by their real MFCloud id) are skipped, so re-running only adds what's changed in the fleet since the last seed.

0.44.3.9 — 2026-08-18

Added

  • Atlas canvas (step 2 of the Topology tab rebuild). The tab is now the interactive board itself — a palette of 12 tile types down the left, a pan/zoom canvas with SVG-drawn connector lines in the middle, and a read-only Inspector panel on the right that opens on selection. Click a palette tile to drop a node at the canvas center; wheel to zoom, drag empty canvas to pan. Node dragging, connector creation, and full editing/delete in the Inspector are staged for the next update — this pass is the rendering foundation.

0.44.3.8 — 2026-08-18

Added

  • Atlas backend (step 1 of the Topology tab rebuild). New atlas_nodes/ atlas_edges tables and GET/POST/PATCH/DELETE /api/atlas/* — a free-form infrastructure planning board (nodes aren't limited to real MFCloud-managed VMs/hosts), scoped per caller the same way GET /api/vms is (admin sees everything, tenant-scoped else by tenant, else by owner). Includes JSON and Mermaid board export. No frontend yet — this is backend-only, verified directly against the API; the interactive canvas UI ships in a follow-up.

0.44.3.7 — 2026-08-18

Removed

  • The Networks tab's "Topology Map" popup modal — fully superseded by the top-level Topology tab (same data, same card layout, no longer a popup). Removed the toolbar button, the modal markup, and showTopology() (infrastructure.js) — nothing else referenced them.

0.44.3.6 — 2026-08-18

Fixed

  • Found the actual cause of the Topology tab hanging on "Loading topology…" forever. Its container div used id="topology-container", which collided with the pre-existing #topology-container inside the Networks tab's "Topology Map" modal (infrastructure_modals.html, included earlier in the page). getElementById returns the first match in document order, so every load was silently writing correct data into that hidden modal's container instead of the visible tab — no exception anywhere, because nothing ever threw; the fetch, the parse, and the render all genuinely succeeded, just onto the wrong element. IDs are now namespaced topology-tab-container / topology-tab-empty.

0.44.3.5 — 2026-08-18

Fixed

  • Topology tab could get stuck on "Loading topology…" with no error shown. Its render step ran outside the fetch's try/catch, so any problem there (or a hung request) surfaced as an unhandled promise rejection instead of an on-screen message — invisible without opening the browser console. The whole load is now one guarded block with a 20s timeout, explicit handling of an expired/unauthenticated session (secFetch returns the raw 401 response rather than throwing), and every failure path now replaces the placeholder with a visible error and logs to the console.

0.44.3.4 — 2026-08-18

Fixed

  • Network topology/summary collapsed every VLAN sharing an OVS trunk bridge into one bucket. 0.44.3.3 fixed the default-NAT-network case but still matched a type='bridge' NIC by bridge name alone — several network rows share the same OVS trunk bridge (ovsbr0) and are only distinguished by their <vlan><tag id=.../></vlan> element, so every VM on that bridge (any VLAN, or untagged) showed up under every VLAN's card. Matching now requires the bridge and the VLAN tag to agree, shared by both GET /api/networks/topology and GET /api/networks/{name}/summary.

0.44.3.3 — 2026-08-18

Fixed

  • GET /api/networks/topology now reports real per-network VM attachment. It previously dumped every VM onto whichever network had vlan_id == 0 and showed nothing for every other VLAN — so the Topology tab and the Networks → "Topology Map" modal that share this endpoint both showed the same full VM list duplicated under multiple networks. It now parses each VM's stored domain XML for its actual <interface> source (network, bridge, or macvtap dev) and matches that against each network's name/bridge, the same DB-only approach GET /networks/{name}/summary already used — no new SSH/libvirt calls.

0.44.3.2 — 2026-08-18

Changed

  • Topology tab restyled to match the Networks → "Topology Map" card layout — one card per network with a VLAN badge, bridge name, and VM chips, replacing the Cluster→Host org-chart from 0.44.3.0/0.44.3.1. Same data source (GET /api/networks/topology) the existing modal already used, just promoted to a full tab instead of a popup.

0.44.3.1 — 2026-08-18

Added

  • Topology host boxes now list their bridges. Each host box shows the Linux/OVS bridges configured on it (from the same data the Networks tab's bridge list uses), so network layout is visible alongside CPU/RAM/VM state without leaving the Topology tab.

0.44.3.0 — 2026-08-18

A fleet topology view, and resource alerting grows a second, earlier severity level.

Added

  • Topology tab. A new admin-only view (Infrastructure → 🗺 Topology) renders the fleet as an org chart — MFCloud → Clusters → Hosts — with each host box showing live CPU/RAM bars and a running/stopped VM count, and each cluster box rolling those up. Clicking a box opens that cluster's or host's existing detail panel. It's assembled client-side from data the console already fetches elsewhere (inventory, host telemetry, VM list) — no new backend endpoint or schema change.
  • Two-level resource alerts. CPU and RAM alert thresholds (Settings → Platform Configuration) are now a Warning and a Critical value instead of one — both fire independently as separate webhook alerts (🟡 warning, 🔴 critical) so an operator gets an earlier heads-up before a host hits the threshold that used to be the only signal.
  • Stale Snapshot alert. A new Settings → Maintenance toggle flags VM snapshots older than a configurable number of days (default 7). The Snapshots list now shows each snapshot's age and marks stale ones with a ⚠ badge.

0.44.2.4 — 2026-08-18

Fixed

  • A VM that finished a live migration stayed defined — just powered off, owner blank — on the host it had just left. virsh migrate --persistent builds a fresh definition on the destination but leaves the old one on the source untouched; only cold migration was explicitly cleaning that up. The host detail panel's VMs tab lists every domain libvirt still knows about on a host regardless of which host the database currently attributes the VM to, so the leftover definition kept showing up under the old host indefinitely — clutter at best, and one accidental virsh start on the wrong host away from running the same disk in two places at once. Live migration now removes the source host's definition once the destination is confirmed to hold the current one, matching what cold migration already did. (Ghost entries left behind by migrations before this fix are not retroactively cleaned up.)

0.44.2.3 — 2026-08-18

A code review of the 0.44.2.2 hardening found its multi-host half incomplete: the proxy-trust fix was correct on a single-node install but could not work on the 3-host cluster, and two of the three places that set a password never received the bcrypt limit added to the third. This release finishes both.

Security

  • On a multi-host HA cluster, two of the three replicas mis-attributed every proxied request. The reverse proxy only runs on whichever host currently holds the cluster's floating address, and it reaches the other two replicas over their real LAN address — which those replicas had no way to recognize as a proxy. They therefore ignored the real client address the proxy passed along and used the proxy's own instead, collapsing the /api/login brute-force limit into one shared budget per replica, recording the wrong source address on every audit-log entry, and handing a self-enrolling node the wrong address to be reached back on (which then had to be corrected by hand). Because it depended on where the floating address currently sat, it moved between hosts on every failover — so it presented as intermittent rather than reproducible. The three cluster hosts are now recognized automatically from the addresses already configured for the cluster; there is no new setting to fill in.
  • Creating a user through the API enforced no password rules at all. The add-user endpoint accepted a one-character password, and equally accepted one past bcrypt's 72-byte limit — the flaw 0.44.2.2 fixed for the change-password flow but not here. All three places that set a password now share one definition of what a valid password is, so they cannot drift apart again.
  • An admin-issued password reset didn't respect that limit either, so a long generated passphrase either failed outright or was silently hashed from its first 72 bytes — leaving the temp password actually shared with the user unable to log in.

Fixed

  • setup.sh still printed a bare --profile ha up -d as the documented way to bring up HA by hand — which starts every replica on every host and hits the exact port-8000 collision 0.44.2.2's profile gate was added to prevent. It now prints the per-host sequence the in-app Enable HA wizard already ran.
  • An HA replica warned at every startup that its proxy settings needed attention — on all three hosts, including correctly configured ones. The check couldn't tell a primary from a secondary, so the warning was constant background noise. It now stays quiet unless the host genuinely can't identify its peers.
  • A TRUSTED_PROXY_CIDRS line left present but blank in .env disabled proxy trust entirely rather than falling back to the default — the opposite of what a blank value should mean. Blank now reads as unset, the same way the compose file has always read it.
  • Likewise, a blank REDIS_PASSWORD left the rate limiter's out-of-compose fallback building a password-less connection while the compose-started Redis still required the default one. 0.44.2.2 fixed this for an absent value; it now also covers a present-but-empty one.
  • An admin could issue a password reset against their own account, which immediately locked the session they were working in out of every page, with no visible way to recover short of signing out. Self-resets are now refused with a pointer to Change Password — and if an account is reset by someone else mid-session, the console prompts for the new password in place instead of showing a bare error.
  • A status callback arriving over a transport with no reportable peer address recorded the loopback address as the reporting node instead of leaving the previous value alone, which could route later operations for that VM to the wrong place.
  • A database failover during the acquisition of the startup lock — as opposed to during the work it guards, which 0.44.2.2 covered — leaked a database connection on every occurrence.

0.44.2.2 — 2026-08-16

Security

  • Direct-connect clients (bypassing Caddy) could spoof their source IP and defeat the /api/login brute-force limiter. uvicorn was launched with --forwarded-allow-ips=*, which trusts an X-Forwarded-For header from any direct peer — not just Caddy — and rewrites the request's client address before the app's own trust check ever runs. Since port 8000 is reachable directly on the LAN on every node, a caller hitting it directly could rotate a spoofed header on every request and bypass the 5/minute login throttle entirely, and have the spoofed address recorded in the audit log instead of their real one. The same gap let a direct caller forge the source IP behind the internal VM-status-callback and node-enrollment IP checks. Fixed by restricting --forwarded-allow-ips to the same trusted-proxy list the app's own IP resolver already used, and by routing the two internal endpoints through that same resolver instead of reading the headers unconditionally.
  • A hardened install (admin account deleted or renamed for security) would silently reintroduce the default admin password on the next restart. The bootstrap seed only checked for a literal admin username collision, not whether the row still existed at all — the same class of bug already fixed once for the default datacenter/cluster seed. Fixed by applying the same existence check.
  • A locked-out account reset by an admin kept its temporary password indefinitely. Unlike the first-login bootstrap password, an admin-issued reset didn't force a real password to be chosen afterward — worth fixing since a temp password is typically shared over chat or email and known to more than just the account owner. It now forces a change at next login, same as the bootstrap flow.

Fixed

  • A new password longer than bcrypt's 72-byte limit could leave an old password still valid after a "successful" change, or crash the request, depending on the installed passlib version. A new password now has to fit within that limit.
  • A .env with DEFAULT_ADMIN_PASS present but empty (rather than unset entirely) seeded the admin account with a blank password instead of falling back to the documented default.
  • The rate limiter's fallback Redis connection (only used when running outside the compose stack) didn't match the password the compose-started Redis actually requires, so every rate-limited route failed in that setup.
  • A database failover landing mid-startup could leave a stuck internal lock behind and crash-loop that replica indefinitely instead of recovering cleanly on the next attempt.
  • app-2/app-3 were missing an HA-only compose profile, so an unqualified --profile single (or bare up -d) on a single-node box swept them in too — both publish host port 8000, so they collided with app-1 and failed to start.

0.44.2.1 — 2026-08-16

Fixed

  • In-console Knowledge Base "empty iframe" hint gave instructions that always fail on a deployed host. It told operators to run cd mfcloud-kb && pip install -r requirements.txt && mkdocs build directly on the host to populate /kb, but a running host only ever receives the pre-built site/ output pushed to it — it has no mfcloud-kb source tree, so that command reliably failed with Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'. The hint now explains the real publish flow: build on a machine that has the source, then ship the resulting site/ to the host.

0.44.2.0 — 2026-08-15

Kubernetes Engine gets a real fleet console. The page was previously three stacked tables; it is now a health-first cluster overview with a per-cluster drill-down. Along the way, a set of shared layout styles that five views referenced but that had never existed in any stylesheet were written, so those views render as designed for the first time.

Added

  • Fleet health at a glance. The Kubernetes page opens with a rollup of the whole fleet — total clusters, how many are healthy, how many need attention (split into in-progress vs failed), total control-plane and worker nodes, and how many platform add-ons are configured.
  • Cluster cards. Each cluster is a card carrying its health as a colour rule and status dot, its engine, whether it is highly available, whether any node runs on bare metal, its API endpoint, its control-plane/worker shape, and its add-ons. Clusters still provisioning pulse so an in-flight build is visible without opening the tasks drawer.
  • Search, health filters, and a list layout. Clusters can be filtered by name, endpoint, tenant, or backend, narrowed to a single health state, and switched to a dense table for large fleets. The layout choice is remembered.
  • Cluster detail view. Selecting a cluster opens a page with four tabs: Overview (configuration, control-plane members, and a timeline of the cluster's recorded activity), Nodes (the live node list, now including kubelet version and OS image), Add-ons (storage and load balancer, with their settings and any error text), and Workloads (the manifests and catalog apps running on that cluster specifically).
  • Unconfigured add-ons advertise themselves. A cluster with no storage provisioner or no load-balancer provider shows that as an actionable placeholder rather than an empty cell, and explains that configuration becomes available once the cluster is healthy.

Fixed

  • Five views were rendering unstyled. The Kubernetes, Containers, App Catalog, Automation, and VM Catalog views all marked up card, toolbar, and data-table styles that were not defined in any stylesheet, so they drew as bare boxes with no panel framing, padding, or header treatment. Those shared styles now exist and all five views pick them up.
  • Cluster names are escaped before display. Names were written into the cluster table without escaping, inconsistent with the rest of the page.
  • Layout toggle was invisible in the light theme. The card/list switch used symbol characters that the browser rendered from a colour-emoji font, which ignores the theme's text colour, leaving the selected icon at roughly four percent contrast against its own background in light mode. It is now drawn as a proper icon that follows the theme.
  • Node tracking drift is easier to read. Nodes the console believes it added but the live cluster no longer reports, and nodes the live cluster reports that the console has no record of, are now clearly labelled in the node list instead of sharing styling with ordinary status text.

0.44.1.9 — 2026-08-15

Console UX pass. Every remaining native browser dialog is gone, and the in-app dialogs they were replaced with are keyboard- and screen-reader- accessible.

Changed

  • No more browser-chrome pop-ups. 100 confirm(), 94 alert() and 7 prompt() dialogs were still being served by the browser itself — 201 in total. They could not be styled, ignored the light/dark theme, rendered differently on every OS, blocked the whole page while open, and could not show any detail beyond a single line of text. All of them now use the console's own dialogs and toasts.
  • Destructive prompts state what they will actually do. Confirmations now carry a real title naming the object ("Delete Tenant", "Evict Host", "Wipe Storage Pool"), an action-specific button instead of a generic OK, and a red treatment reserved for irreversible operations. Deleting a cluster or a datacenter now requires typing its name, and lists what will be released before you commit — neither was possible in a browser dialog.
  • Safer defaults on destructive dialogs. Keyboard focus lands on Cancel, not on the destructive action, so a reflexive Enter no longer confirms a delete.
  • API keys get a proper reveal. A newly created key is shown in a dedicated dialog with a Copy button, replacing a prompt() box that was being used purely because its text happened to be selectable — a stray Enter used to dismiss the only chance to capture the credential.

Accessibility

  • Visible keyboard focus everywhere. Only buttons had a focus ring before, so tabbing through the sidebar, inventory tabs, links and form fields left no indication of position (WCAG 2.4.7).
  • Dialogs are now real dialogs. All five in-app dialog types announce themselves as modal, keep Tab cycling inside while open, close on Escape, and return focus to whatever opened them (WCAG 2.1.2, 2.4.3).
  • Skip to main content link as the first tab stop, so keyboard users can bypass the masthead and inventory tree (WCAG 2.4.1).
  • Global search has a real label instead of relying on placeholder text, which disappears on the first keystroke and is not reliably announced.
  • Dialog animations respect the operating system's reduce-motion setting.

Fixed

  • Dialogs built their markup from inline styles, which silently opted them out of both the density toggle and the light theme, and pinned the destructive red to one fixed value that did not adapt between themes. They now use the same design tokens as the rest of the console.
  • Multi-line confirmation messages kept their line breaks instead of collapsing into a single run-on paragraph.

0.44.1.8 — 2026-08-14

Hardening pass over the 0.44.1.6/0.44.1.7 multi-cluster Ceph, OVN Central and VM Catalog work. Two full code-review passes over the same tree found 23 issues; this release fixes all of them. Ten traced to a single root cause: clusters.ceph_cluster_id had no write path outside startup — it was populated only by the init_db() backfill and one narrow self-heal, never by Ceph bootstrap or compute-cluster creation, and never cleared. So the link was legitimately unset on live systems and could dangle permanently after a cluster destroy, while each new reader treated "can't resolve" as "no Ceph".

Fixes — Ceph HA and data integrity

  • Ceph storage fencing could silently refuse to fence, leaving a dead node's VMs unrecoverable. When a compute cluster had no resolved Ceph link — the normal state for any cluster created since the last restart — _ceph_monitor_ip() returned "can't fence", so HA marked the node Fence-Failed and refused to restart its VMs anywhere. Unfencing hit the same dead end, so a recovered node's blocklist entry was never lifted. It now falls back to the sole registered cluster exactly as the VM-build, migration and V2V resolvers already did; only a genuinely ambiguous multi-cluster setup declines.
  • Destroying a Ceph cluster left every compute cluster pointing at the deleted row, with no way to recover in-app. Re-bootstrapping produced a new id while every link still referenced the old one, and because both backfills only fill unset links they skipped it forever — breaking fencing, Ceph-backed VM deploys, storage migration, V2V and capacity reporting until someone edited the database by hand. Destroy now clears the link as part of the same transaction, and a startup migration repairs deployments already broken by the previous behavior.
  • A console upgrade could fail to start on any deployment that had registered the same Ceph cluster twice. The new uniqueness constraint was added without checking for existing duplicates; the resulting error aborted schema initialization, which the app startup does not survive. Duplicates are now detected and reported with the constraint skipped, and the statement is checkpointed so no other failure can take the console down on upgrade.
  • A connection leak on every Ceph-backed VM build in a multi-cluster deployment could eventually exhaust database connections fleet-wide.

Fixes — reporting and alerting

  • Storage Capacity omitted Ceph entirely for a Ceph-backed cluster whose link wasn't resolved, showing it as having essentially no storage.
  • One unreachable Ceph cluster hid every other cluster's capacity. The rollup aborted on the first failure, and because cluster order isn't stable the reported figure varied between refreshes. Each cluster is now polled independently, and the database connection is released before the polling starts.
  • The Ceph health webhook could still miss a real state change. 0.44.1.7 fixed the case where a poll fails; a poll that succeeds with an incomplete response (manager mid-failover) still recorded the placeholder state and suppressed the next genuine transition to HEALTH_ERR.

Fixes — VM Catalog

  • Opening a blueprint pinned to a non-default Ceph storage tier and saving it silently erased that setting. The storage-tier dropdown was a fixed single entry that was never populated from the cluster, so selecting anything else left it blank on save. Both the blueprint editor and the deploy dialog now list the cluster's real pools, and a tier the cluster no longer reports is preserved rather than dropped.
  • The "CSV Template" button returned an authentication error instead of a file — it was a plain link, which carries no credentials.
  • A CSV row with more fields than the header returned a server error instead of a clear per-row message. Commonly hit by a stray trailing comma or an unquoted comma inside a value.
  • Clearing a numeric field on a blueprint and saving returned a server error instead of a validation message.
  • Bulk-deploy uploads are now size-bounded and row-capped while reading, rather than after parsing the entire file.
  • Bulk deploy no longer risks timing out mid-batch. Each VM creation was blocking on the outbound notification webhook — up to five seconds per VM when that endpoint is slow or unreachable, and it stalled the whole console while it waited. Notifications are now sent in the background, which speeds up every VM deploy, not just batches.

Fixes — OVN Central and console

  • Deploying OVN Central to a host without firewalld was reported as a failure even though OVN installed correctly and was serving, leaving the central address unrecorded. The firewall step is now best-effort and only runs where firewalld is active; the post-install verification remains the real check.
  • A failed "Deploy Central" request gave no feedback at all — the dialog looked untouched, inviting a retry that could repoint the control plane a second time.
  • A pooled database connection was held open across the reachability probe of an unreachable OVN central, which could starve the connection pool across repeated attempts.
  • The OVN panel issued a request non-admins are never permitted to make on every refresh, filling browser and server logs with denials.
  • Reconnecting a VM console could show a false "connection lost" banner over a console that was actually connecting, because the previous session's teardown event arrived after the new one had started.

Internal

  • Storage pool names from external clusters are now escaped before rendering in the deploy, migrate and catalog dropdowns.
  • Ceph maintenance mode went from three database lookups and three remote command invocations per request to one of each, without losing its fail-fast behavior.
  • Removed an unused Ceph resolver, replaced two throwaway tuple wrappers with named values, and corrected comments that described behavior the code no longer had.

0.44.1.7 — 2026-08-14

Fixes

  • 0.44.1.6 regression: a ZFS/mixed-tagged compute cluster's dashboard Storage Capacity could show the shared Ceph cluster's total/used bytes folded in. clusters.ceph_cluster_id is deliberately backfilled onto every compute cluster while a single Ceph cluster is registered (so a VM that explicitly overrides to Ceph storage can still resolve which physical cluster to use, regardless of that cluster's default backend) — but _cluster_backed_by_ceph was treating the FK's mere presence as proof the cluster's primary backend is Ceph, inflating the scoped rollup for any cluster tagged something else. Reverted that check to storage_backend == 'ceph' only.
  • destroy_cluster_task's OSD-row cleanup still wasn't cluster-scoped. 0.44.1.6 left DELETE FROM ceph_osds unconditional with a comment explaining cluster_id wasn't populated yet — true when that comment was written, stale by the time it shipped (the same release populates it on every insert and backfills existing rows). Scoped it via the same cluster the ceph_clusters row deletion already targets.
  • Ceph health-monitor webhook could miss a real state change after a transient poll failure. A single network blip against one cluster's dashboard API got recorded as UNKNOWN, which then suppressed the next genuine health-change alert (the "never fire on first-seen UNKNOWN" guard treated the blip the same as a fresh unknown cluster). A failed poll now leaves the last known state untouched instead of overwriting it.

0.44.1.6 — 2026-08-14

New Features

  • Multi-cluster Ceph support. ceph_clusters previously had no uniqueness constraint, but every Ceph route in the app read it with LIMIT 1 — a second cluster would have been invisible to the console the moment it existed, and Destroy on either cluster wiped the DB registration row for both (DELETE FROM ceph_clusters had no WHERE). Added UNIQUE constraints on cluster_name/monitor_ip, populated the previously-unused ceph_osds.cluster_id, and added a new clusters.ceph_cluster_id FK linking each compute cluster to the Ceph cluster backing it (self-healing backfill, so every existing single-cluster deployment resolves identically to before — zero behavior change unless a second cluster is actually registered). Threaded real cluster identity through all ~44 call sites: status/telemetry/pools/OSDs/ maintenance/mgr-failover endpoints, the background health poller, fleet inventory & storage-capacity rollups, and — the correctness-critical paths — HA fencing, storage migration, VM disk placement, V2V import, and the K8s/LXD Ceph-CSI bootstrap flows, so a fence/migration/deploy always targets the Ceph cluster that actually backs the node/VM in question instead of an arbitrary row. The Ceph modal now lists every registered cluster and lets an operator switch between them; the Deploy modal, Storage Migrate modal, and V2V's target-storage picker list real Ceph pools instead of a hardcoded "vms" option.

Fixes

  • Destroy Ceph Cluster wiped the wrong cluster's registration. Scoped the DELETE FROM ceph_clusters to the cluster actually being torn down — see above. This was a live data-loss trap the moment a second cluster existed, independent of the rest of multi-cluster support.
  • Ceph pool-create's client.libvirt cap rebuild was pulling every compute cluster's assigned pool name fleet-wide, not just the ones on the cluster being modified — harmless with one Ceph cluster, but would have granted (or collided) caps across clusters once a second one exists.

0.44.1.5 — 2026-08-12

New Features

  • VM Catalog: network + storage location chosen at deploy time. The deploy dialog (click a blueprint's card) is now a full dialog instead of a 2-field prompt: target host/cluster (unchanged), a real Network dropdown (/api/networks, VLAN/OVN/security-group shown inline), and a Storage Backend picker (local/ZFS/Ceph/NFS) with the matching location field — ZFS pool, Ceph tier, or NFS datastore — shown conditionally, same pattern as the main Deploy VM dialog. The blueprint still sets sane defaults for all of these (used as the dialog's starting values, and still what CSV bulk-deploy rows use unchanged); the operator can now override network and storage per single-VM deploy without editing the blueprint itself. Blueprint editor gained matching Ceph pool / NFS datastore default fields (previously only ZFS pool was settable — Ceph/NFS blueprints silently fell back to defaults before this).

Fixes

  • VM Catalog deploys ignored VLAN tagging on non-default networks. create_new_vm needs vlan_id resolved separately from network_name (resolve_vlan_id(), same helper routers/iac.py uses) — the catalog's deploy endpoints never did this, so any network override would have silently landed on VLAN 0 (untagged) instead of the intended network. Only became reachable once network selection shipped above; fixed before it could bite anyone.

0.44.0.16 — 2026-08-12

Fixes

  • Open VM console looked frozen/black after the VM live-migrated out from under it. Live migration destroys the source VM's QEMU process right after cutover, which kills the proxied TCP connection an already-open console was riding on. The RFB client's disconnect event fired, but the handler only updated an 11px status pill in the toolbar — the actual video canvas kept showing whatever frame was on screen at the moment of disconnect, indistinguishable from a live, working console. Added a visible overlay ("Connection lost — the VM may have migrated to another host") over the console canvas on any unexpected disconnect, with its own Reconnect button, instead of relying on the easy-to-miss status pill alone. The underlying migration self-heals (VNC listen address, task tracking) were already correct — this was purely the frontend not surfacing the disconnect over the frame that mattered.

0.44.0.15 — 2026-08-12

New Features

  • VM Catalog: OS badges on blueprint cards. Each card now shows a small color-coded badge auto-detected from the blueprint's template name (Rocky/RHEL/CentOS/Fedora/Ubuntu/Debian/SUSE as a monogram, Alpine as a mountain triangle, Windows as a four-pane flag) instead of a plain generic icon — no external icon assets loaded, pure client-side regex match. Falls back to a generic penguin badge for unrecognized Linux templates, or the blueprint's manual icon field for anything else.

0.44.0.14 — 2026-08-12

Fixes

  • VM Catalog single-deploy: target host was a free-text field. Replaced with the same picker the main Deploy VM dialog uses — "🎯 Automatic", one "🎯 Automatic — <cluster>" per eligible cluster, then individual hosts, filtered to the blueprint's OS class. No more typing an exact hostname to pin placement. mfFormModal (the shared small-form modal helper) gained a type: 'select' field kind to support this — reusable by future callers instead of forking a one-off modal.

0.44.0.13 — 2026-08-12

New Features

  • VM Catalog. New "🗂 VM Catalog" tab (internal-team tool, separate from the existing LXD image "Catalog" tab): admins curate one-click blueprints (template + vCPU/RAM/disk/OS/network fixed, picked from the live Content Library so a typo can't reference a nonexistent template), and anyone can deploy a VM by clicking a blueprint's card and naming it. Also supports a CSV bulk path (vm_name,blueprint[,target_host][,network_name] rows, up to 200 per upload) — rows left on automatic placement are spread across the cluster together via the same DRS-lite batch-placement logic the V2V multi-VM import uses, instead of every row landing on the same host. One bad row (unknown blueprint, name collision, no host fits) fails just that row; the rest of the batch still deploys. New vm_catalog_blueprints table; new routers/vm_catalog.py (/api/vm-catalog/*) reuses create_new_vm verbatim — no changes to quota/RBAC/DRS-lite/staging.

Fixes

  • VM create could crash on an OVN-backed network when called outside the main /api/vms/create route. create_new_vm unconditionally queues a background_tasks.add_task(...) OVN port/security-group sync when the target network resolves to a switch or security group — three other callers (routers/iac.py's Terraform/Ansible endpoint, the AI Assistant chat panel's VM-deploy tool in core/chat_tools.py, and the new VM Catalog above) didn't pass a background_tasks object, which would have raised AttributeError on first use against an OVN network. All three now pass one through (the chat panel builds and runs one by hand, since a WebSocket message has no FastAPI request/response cycle to do it automatically). Found while building VM Catalog, before it shipped.

0.44.0.12 — 2026-08-12

Fixes

  • "Deploy Central" wizard's node field was wrongly restricted to existing compute-fleet hosts. It was a <select> populated only from /api/hosts (compute nodes), so a freshly-created dedicated VM — the normal, KB-recommended target — could never be selected, since VMs aren't tracked as compute nodes at all. Now a free-text IP field with existing compute nodes offered only as convenience suggestions, not a restriction. Found live the first time it was actually used.

0.44.0.11 — 2026-08-12

New Features

  • OVN Central deployment automation — UI (3 of 3, complete). New "Deploy Central" tab in the OVN section's New Object wizard: node picker pre-filtered to exclude nodes already registered as chassis, version field pre-filled from the fleet's current pin, and the replace-existing-central confirmation surfaced as a plain confirm dialog when the API's guardrail trips. Deploy progress (Installing → Verifying → Live/Error) now shows as a banner on the existing OVN Central Health panel rather than a separate polling UI, since that's where the wizard already points the operator after queuing. Closes out the 3-phase OVN Central management build.

0.44.0.10 — 2026-08-12

New Features

  • OVN Central deployment automation — API (2 of 3). New POST /api/networks/ovn/central/deploy queues bootstrap_ovn_central (shipped last release), enforcing all three guardrails before anything is queued: version pin required (explicit or falls back to the already-pinned fleet ovn_version, never repo-latest), refuses a node already registered as an OVN chassis (checked by tunnel_ip, not hostname), and refuses to silently replace a currently-reachable central — that needs an explicit replace_existing:true. No UI yet; this is callable but not clickable.

0.44.0.9 — 2026-08-12

New Features

  • OVN Central deployment automation — backend (1 of 3). New tasks.ovn.bootstrap_ovn_central Celery task installs and configures the OVN control plane (ovn-northd + NB/SB) on an already-reachable node, mirroring bootstrap_ovn_chassis's proven install technique. Unlike chassis bootstrap, the version pin is required, not optional — this closes the exact gap that caused the 2026-08-03 outage (a fresh install independently resolving repo-latest). Settings only get written after the new central is verified to actually answer ovn-nbctl/ovn-sbctl show, never on a completed-but-unresponsive install. No API endpoint or UI wired up yet — those are separate follow-up phases; this task can't be triggered by anyone yet.

0.44.0.8 — 2026-08-11

New Features

  • OVN Central Health panel now shows explicit ovn-northd service status, not just NB/SB DB liveness. Confirmed live against ovn-central-01: ovn-northd.service is a single systemd unit that forks and monitors three children (northd itself, plus one ovsdb-server per NB/SB), so a plain systemctl is-active can't tell "everything's fine" apart from "one specific child died but the wrapper unit never flapped." The new check counts each monitored child's own (healthy) status instead. Data cleanup from testing this against the real fleet: 3 unreachable/decommissioned chassis removed, and KVM-CEPH-01/02/03 + KVM-DCTCK93 chassis now correctly identify themselves to OVN (a stale external-ids:hostname=TMPL-el9 left over from template cloning was corrected on the CEPH nodes) — chassis drift on the real fleet is now zero.

0.44.0.7 — 2026-08-11

Fixes

  • OVN chassis registration no longer produces permanent false-positive drift. bootstrap_ovn_chassis now explicitly pins external-ids:hostname on the chassis instead of leaving OVN's Southbound DB to auto-detect the box's own OS hostname — previously the two could never agree (the register-chassis picker submits IPs, and a node cloned from a golden template may not even have a unique OS hostname), so the OVN Central Health panel's chassis-drift check would flag a healthy, correctly-bound chassis as drifted forever. Found live while validating the health panel above against real data. Only affects chassis registered/re-registered after this fix — existing chassis are unchanged. The "Register OVN Node" picker also now shows each node's real hostname (falling back to its IP) instead of a bare IP list.

0.44.0.6 — 2026-08-11

New Features

  • OVN Central Health panel + config UI. Networks → OVN now shows a live, read-only status strip for the OVN control-plane node (ovn-central-01): reachability, NB/SB database liveness, installed-vs-pinned OVN version (the same class of mismatch behind the 2026-08-03 ARP-recursion outage), and chassis drift between ovn-sbctl show's ground truth and the app's own ovn_chassis table — plus a collapsible raw ovn-nbctl/ovn-sbctl show dump so engineers no longer need to SSH in just to eyeball state. Fetched on-demand (tab open / manual Refresh), never polled, since each check is a live SSH round-trip. Every command it runs is a show/list/version query — nothing here can mutate NB or SB state. Settings now also has an OVN Central widget (ovn_central_ip + ovn_version), replacing the podman-exec-a-python-snippet workflow the KB previously documented for pointing MFCloud at the central VM. Deployed to all 3 CP hosts and verified healthy/auth-gated; live behavior against the real ovn-central-01 still pending hands-on verification.

0.44.0.5 — 2026-08-10

New Features

  • mfcloud boot splash for BIOS-firmware VMs. New "Show mfcloud boot splash" checkbox in the Create VM wizard (Firmware section). When a VM deploys on BIOS/SeaBIOS firmware, its console shows the mfcloud logo during POST via QEMU's native boot-splash support. UEFI/OVMF VMs are unaffected — OVMF's boot logo is compiled into the firmware binary with no runtime override, so the option is skipped (with a build warning) for those. The splash image is staged separately per node at /usr/share/mfcloud/bootsplash.jpg (see deployment/node-minimal-setup.sh); a node missing it just shows the default firmware screen instead of failing the deploy. Applies to newly created VMs only — not yet exposed on the VM Settings edit path for existing VMs.

0.44.0.4 — 2026-08-10

New Features

  • Hardware Lifecycle Manager: cluster rolling firmware campaign. Click a cluster, pick which out-of-compliance firmware components to bring into line, and mfcloud walks every node one at a time — evacuate, update, verify, rejoin — with no further clicks. A new fleet-compliance view surfaces firmware version drift across a cluster before you start. A node carrying a local-storage VM halts its own turn for manual handling instead of riding the slower unattended live-migration fallback; insufficient cluster capacity aborts the whole campaign rather than guessing. Campaign state lives in the database and advances via a beat-scheduled sweep, so a campaign survives a master/worker restart mid-run. Code-complete but not yet run against real hardware — see patch_note/design_artifacts/HARDWARE_LIFECYCLE_MANAGER_DESIGN.html for the full design and current validation status.

0.44.0.3 — 2026-08-10

New Features

  • Windows golden-template workflow. Converting a Windows VM to a template now verifies its guest agent is actually running before allowing the conversion — blocks with a clear message (override available) instead of silently baking a template with no IP/heartbeat/quiesced-snapshot support, since the source VM is gone once conversion completes and there's no second chance to check. Windows templates are now badged separately in the Content Library.
  • Port groups now enforce their configured security group + IP pool at deploy time. Deploying a VM onto a network-backed port group with those fields set now automatically attaches it, allocates its IP, and applies the security rules — previously this required a manual assignment step after every deploy.
  • Per-cluster dedicated LINSTOR storage, matching the existing per-cluster Ceph pool isolation. A cluster bound to a LINSTOR resource group now auto-provisions each VM's replicated volume into it instead of requiring an operator to pre-create the resource by hand.
  • Billing rollup now reports estimated runtime-hours per cost center alongside the existing allocated vCPU/RAM/disk figures, from a new power-state history sampler.

0.44.0.2 — 2026-08-10

Fixes

  • CRS now enforces a VM's require_labels placement constraint. Previously request-time only and never persisted, so CRS could migrate a VM onto a host missing a capability label it was originally required to land on. Now persisted as vms.require_labels at creation and enforced as a hard destination filter in both the periodic/maintenance-exit rebalancer and the maintenance-exit VM-count restorer.
  • CRS: exclude now renders a badge in the VM list, matching the existing HA: local-storage styling — previously enforced server-side with no visual indicator.
  • Pending CRS recommendations older than 24h are now auto-expired by a new hourly sweep instead of sitting in the approval queue indefinitely.

0.44.0.1 — 2026-08-10

Fixes

  • Host detail panel's Containers tab: tenant isolation. GET /api/hosts/{ip}/summary now scopes its container query the same way list_containers does — it was returning every tenant's container names/owners for a host to any authenticated user, unlike every other container-listing endpoint.
  • Host-panel container rows now escape onclick target attributes with escapeJsAttr instead of escapeHTML, matching the Containers tab's own table.
  • Host-panel container right-click menu now reads from the panel's own freshly-loaded data instead of a cache that's only populated by visiting the main Containers tab first — it could silently do nothing otherwise.
  • Dragging a container folder onto a different host's root row is now rejected with an error instead of silently reparenting it back under its original host (folders can't move across hosts).
  • Container console resize (drag-to-resize, fullscreen) now reaches the backing LXD exec PTY, not just the xterm.js grid — full-screen terminal apps (htop, vim, …) render at the correct size again.
  • Fullscreen container console modal no longer clamped to 92% viewport height.

0.44.0.0 — 2026-08-10

Big release — everything shipped since 0.43.0.0, the last version pushed to Docker Hub. Per-patch detail for each item below lives in its own dated entry further down this file (0.43.0.1 through 0.43.9.3).

New Features

  • Traffic Control — bandwidth/QoS management (Phases 1-3). NIOC-style rate limiting: Settings → Traffic Control caps for local/NFS/cloud backups and storage migrations, per-VM NIC Inbound/Outbound Mbps caps via libvirt <bandwidth>, and a Ceph mClock QoS profile selector.
  • CRS Predictive Trigger (ML). XGBoost-based fleet load forecasting on top of the existing reactive rebalancer — 6-hourly retrain, baseline-MAE comparison ("N% better than naive"), and a live-predictions view on the haman dashboard.
  • Hardware Lifecycle Manager: out-of-band Redfish/iDRAC telemetry. New Network, Storage/RAID + physical-disk health, and DIMM slot population views on the host Hardware tab, plus a Ceph "Manager (mgr) Daemons" control with one-click promote.
  • Networks tab: tree navigation + object detail panes. L2 Bridges, Portgroups, Cluster VLAN Fabric, OVN Logical Switches/Routers/Chassis, Security Groups, and IP Pools now open in the same tabbed detail-pane shell used by VMs/hosts/datastores instead of being unclickable flat lists.
  • Virtual SCSI controller management (VirtIO SCSI / LSI Logic Parallel / LSI Logic SAS / BusLogic, multiple per VM) and a VM Settings redesign — Hardware/Options/Advanced tabs are now collapsible per-device rows instead of one long form.
  • Configuration Parameters — raw QEMU -set key=value launch-arg passthrough for advanced tuning, gated behind a dedicated can_edit_advanced_config permission.
  • Selectable vNIC link speed (1/10/25 Gbps), so guests report a real speed instead of virtio-net's default "Unknown!".
  • Host identity/inventory polish — hosts now show detected hostname alongside IP everywhere, plus a Dell hardware-vendor badge.
  • Maintenance-exit VM-count restoration — exiting Maintenance now deterministically pulls back the same number of VMs a host ran before.
  • Two-tab Tasks / Cluster log view on the Recent Tasks drawer, backed by a new cluster_log audit table — MFCloud's first daemon-level audit trail.
  • Display Timezone setting and a Settings-page Software Update widget.
  • LXD container parity — Monitor/Logs tabs, a resizable/fullscreen console, host-scoped container folders (New Folder/Subfolder/Delete/ drag-and-drop, mirroring the VM folder tree), and a per-host "Containers (N)" tab.

UI Improvements

  • VM and Host Monitor tabs redesigned into multi-panel dashboards; the standalone Performance page was retired in favor of the same range picker built directly into both.
  • Host detail panel gained VMs and Monitor tabs; HA Cluster status moved off a floating overlay onto the Global Summary tab; alarms moved into the Recent Tasks drawer (floating bell removed).
  • Dark-mode native <select> popups fixed app-wide; Networks-tab badge/table cleanup; Traffic Control's Network Adapters modal widened and its bandwidth inputs properly labeled.

Security Fixes

  • OVN security-group ACL rules were silently no-ops — the ACL builder matched the wrong port field for ingress vs. egress; Security Groups now enforce as configured.
  • HA failover split-brain protection — a bounded lock now prevents two overlapping monitor cycles from fencing the same node and restarting its VMs twice.
  • iDRAC/BMC credentials encrypted at rest (AES-256-GCM); Redfish/ iDRAC connections now use TLS certificate pinning, matching the app's existing SSH host-key pinning.
  • Login audit log IP spoofing — closed a header-trust gap that let a caller hitting port 8000 directly spoof the logged source IP of a login attempt.

Reliability

  • Celery beat was running duplicated across all 3 HA nodes instead of primary-only — real damage confirmed (a Ceph cluster's VM placement thrashing under repeated concurrent CRS rebalance passes). Fixed with a durable IS_BEAT_PRIMARY runtime guard.
  • Ceph dashboard now survives its seed monitor node being unreachable — falls back through other cluster members and self-heals monitor_ip to whichever peer is actually active.
  • AI chat panel no longer hangs forever on a backend error.
  • Database-pool startup hardened against container-recreate races; a fleet-wide compute-node worker crash loop (missing core.fencing in the node code-sync allowlist) fixed at the source.

Bug Fixes

  • VM live-migration now warns immediately instead of silently doing nothing when there's no eligible target host.
  • Fixed stale cached VM memory/vCPU counts persisting after a hardware change via VM Settings.
  • Disk/NIC/SCSI-controller hot-attach no longer requires powering off a running VM first; CD/DVD eject/attach now reaches a running guest immediately instead of only on next reboot.
  • Add Host for Ceph now installs python3-jinja2 on every node, not just the bootstrap node.
  • Container folders: create/delete moved from inline icon buttons to right-click menus, matching the VM folder tree's existing pattern.

0.43.9.2 — 2026-08-10

Fixes

  • 0.43.9.1's container "New Folder" solved the wrong problem, and its Containers-tab list fix showed the wrong scope. "New Folder" ran mkdir -p inside a container's own guest filesystem — the actual ask was an organizational folder to group containers under a host (put-container- in-folder), not a Linux directory. Removed the guest-exec mkdir feature entirely (core/lxd/_instances.py::exec_run, the /mkdir route, its schema) and replaced it with container_folders — host-scoped (containers aren't reliably cluster-scoped the way VMs are; see core/schema.py's comment), nestable, with the same New Folder/New Subfolder/Delete/drag-and-drop UX the VM folder tree already has. Folders render per-host in the sidebar tree (Host → Folder → Container) and are assigned by dragging a container onto a folder row. Separately, the Containers-tab list fix landed as one flat table of every container across every host — not what was asked, and not how the VMs view works either (VMs have no flat table). Reverted that table; container names/status now show in the sidebar tree (unchanged) and, new in this release, in each host's own detail panel under a "Containers (N)" tab (GET /api/hosts/{ip}/summary now includes a containers list; mirrors the existing "VMs (N)" tab exactly, hosts.js's _hostContainersHtml).

0.43.9.1 — 2026-08-10

Added

  • LXD container detail panel: Monitor and Logs tabs. Previously the container detail panel (opened from the tree or the Containers tab) only ever had a Summary tab — no way to see CPU/RAM history or an audit trail without leaving the panel. Monitor embeds the same chart the "📊 Stats" modal already had, live-polled every 5s against the existing /telemetry/history endpoint; Logs reuses the VM audit-log endpoint/renderer unchanged (it already works for containers — task history is keyed by name, not by VM-vs-container).
  • Container console: resizable + fullscreen. The console modal was effectively stuck at the shared modal default of 460px wide — a leftover max-width:920px never took effect because no explicit width was ever set, so it silently fell back to the smallest modal size in the app. Now opens at 90% of the viewport, can be dragged to any size (CSS resize: both), and has a fullscreen toggle. xterm.js now fits its container instead of a fixed 80×24 grid (FitAddon + ResizeObserver, same pattern already used by the ZFS live-log and Ceph telemetry terminals).
  • Container "New Folder" quick action. Runs mkdir -p inside a running container via a new one-shot, non-interactive LXD exec (core/lxd/_instances.py::exec_run, distinct from the interactive exec the console uses) — POST /api/containers/{host}/{name}/mkdir.

Fixes

  • Containers tab listed zero containers even with running containers on the host. The tab's own container table never actually existed in the template — _renderContainerTable() was fully implemented and already wired into the tab's load path, but had no #containers-tbody element to write into (a gap left by an earlier per-row-button-wall removal), so it silently no-op'd every time. Container names were only ever visible via the sidebar tree, which hits the same API through a different code path. Restored the table; clicking a name opens the same detail panel the tree does.

0.43.9.0 — 2026-08-10

Fixes

  • 0.43.8.9's IS_BEAT_PRIMARY guard stopped the duplicate scheduler but turned every HA secondary's beat container into a permanent restart loop. The refusal branch did exit 1, and beat still has restart: unless-stopped — confirmed with a throwaway container running the identical guard logic: 7 restarts in 12 seconds, forever, with the "REFUSING TO START" line re-logged on every single retry. The scheduler itself never ran (the guard's actual job), but this is a much noisier steady state on HA-02/HA-03 than intended. The refusal branch now idles (exec sleep infinity) instead of exiting, so a non-primary host's beat container logs the refusal exactly once and sits calmly at a single "Up" — re-verified with the same throwaway-container test: 0 restarts. Its healthcheck will correctly report unhealthy there (no beat process running); that's the expected, permanent signal for "not the primary," not a fault to chase.

0.43.8.9 — 2026-08-10

Fixes

  • 0.43.8.6's duplicate-beat fix had no way to stay fixed — and it already recurred. That entry stopped two already-running duplicate beat containers on HA-02/HA-03, but nothing stopped them from coming back: beat had no profiles: gate in docker-compose.yml, so a bare podman-compose up -d (no service filter) on any host starts every gate-less service, beat included. Confirmed live 2026-08-10: exactly that happened again, beat was healthy on all 3 nodes simultaneously before this was caught a second time. Added a runtime guard instead of relying on operator discipline alone — beat's command now checks a new IS_BEAT_PRIMARY env var (.env, defaults to true) and refuses to actually start the scheduler unless it's explicitly true, regardless of which compose invocation started the container. HA-02/HA-03's .env now set it to false; the primary's stays true. Verified by hand: starting beat on either secondary now exits immediately with a clear "REFUSING TO START" log line instead of silently scheduling anything.

0.43.8.8 — 2026-08-10

Fixes

  • 0.43.8.5's CRS self-telemetry contamination fix didn't actually exclude the contaminating rows. core/crs_forecast.py's _fetch_series added a JOIN compute_nodes to restrict training to registered hosts, but core/schema.py permanently seeds compute_nodes with its own ('localhost', 'Active', '__control_plane__') sentinel row so unrelated lookups can resolve the master — the join still matched host_ip='localhost' and let the control-plane's inflated self-sample straight through. Every 6-hour retrain since 0.43.8.5 continued training on it. Filters out the __control_plane__ sentinel cluster explicitly instead of relying on join membership alone.

0.43.8.7 — 2026-08-10

Fixes

  • AI chat panel could hang forever on a genuine backend hiccup, with zero trace anywhere. routers/chat.py's background turn-runner caught WebSocketDisconnect but silently swallowed every other exception (except Exception: pass) — including anything a tool call raised that wasn't HTTPException/KeyError/TypeError/ValueError (a dropped DB connection during a Patroni failover, an SSH timeout reaching a hypervisor node, etc.; confirmed a real path via lifecycle.py's VM-creation license check re-raising a raw exception). The frontend only re-enables its input on the server's done envelope, which was never sent, and the socket stayed open so the 5s auto-reconnect never kicked in either — the panel just sat on "thinking" until the engineer reloaded the page. Now logs the exception and sends error+done so the panel recovers instead of hanging.

0.43.8.6 — 2026-08-10

Added

  • CRS Predictive Trigger (ML): baseline-comparison MAE + live predictions. A raw MAE number (e.g. 1.6%) means nothing without a reference point — is that actually better than doing nothing? crs_ml_models gained a baseline_mae column: a naive "predict no change" forecast's MAE on the same held-out validation split the model's own MAE comes from, computed at every retrain. The console and haman dashboard's ML sections now show both ("N% better than baseline", or a warning if the model isn't beating the naive forecast). Also added a live predictions view (haman) — current CPU/RAM vs. the model's live 15-min-ahead forecast for every predictive-enabled cluster's hosts, refreshed on demand, so an operator can see the model producing sane numbers continuously instead of only finding out if/when it crosses the trigger threshold.

Fixes

  • beat was running on all 3 HA nodes instead of just the primary. Celery has no distributed-beat lock in this stack — two schedulers each fire every periodic task independently, which is exactly what happened. Confirmed real damage: tasks.crs.rebalance_all_clusters (5-min cadence) fired repeatedly within seconds of itself across the 3 uncoordinated schedulers, thrashing a Ceph cluster's VM placement (one 3-host cluster's VM-count spread swung 9/1/19 → 9/1/18 → 14/15/0 → 5/14/10 within about 2 minutes of observation) — far more churn than CRS_MAX_MOVES_PER_PASS=2 per pass should ever produce. No HA/fencing or ZFS-replication damage found on audit (no fencing fired, no Patroni failover occurred, no ZFS replication is even configured on this fleet). Stopped the duplicate beat containers on the 2 secondaries; beat runs on the primary only going forward, per docker-compose.yml's own comment on that service.

0.43.8.5 — 2026-08-10

Fixes

  • CRS Predictive Trigger (ML) training data was contaminated with control-plane self-telemetry. core/crs_forecast.py's _fetch_series pulled from host_telemetry with no host filter at all — that table also carries tasks/telemetry.py's own master self-sample (host_ip='localhost', written independently by whichever of the 3 HA nodes runs the beat/worker, so in practice up to 3x the row volume of any single real host under one shared key). That's console-infra load, not hypervisor/VM load, and it measurably doesn't look like the real fleet: confirmed live 2026-08-10, its CPU reading's stddev over 24h ran 4-9x any actual cluster's (8.6 vs 0.9-2.3). Restricted training to currently- registered compute_nodes only.
  • Predicted CPU/RAM had no upper clamp. predict_fleet() floored predictions at 0 but never capped the high end, so an XGBoost regressor extrapolating past its training range could emit e.g. 137% into a recommendation's pred_hot_cpu/pred_spread_pct text — looks like a bug to whoever reads it. Clamped to [0, 100] on both ends, matching how live cpu_pct/ram_pct are already bounded.

0.43.8.4 — 2026-08-09

Added

  • "Manager (mgr) Daemons" control on the Ceph Administration tab. Lists every mgr daemon and which host it runs on, active vs. standby, with a "Make Active" button per standby. Follow-up to 0.43.8.0/0.43.8.1's seed-failover fix: Ceph itself has no "become active" command for a specific daemon, only ceph mgr fail <current active>, which lets a standby take over — deterministic with exactly one standby (the normal setup here), best-effort otherwise. New GET /api/ceph/mgr-status and POST /api/ceph/mgr/promote in routers/ceph/maintenance.py, gated on Storage Admin for the promote action.

0.43.8.3 — 2026-08-09

Fixes

  • VM Summary panel kept showing a VM's old memory/vCPU count after a hardware change via VM Settings, even after a reboot. Two compounding bugs: apply_vm_settings (tasks/settings/hardware.py) correctly rewrote the libvirt XML on the compute node, but never updated the console's cached vms.vcpus/vms.memory_mb row — and the Summary endpoint (routers/vms/core/telemetry.py) read that stale DB row first, falling back to a live libvirt query only when the DB value was zero, so a nonzero-but-wrong cached value could never be corrected by the live read. Summary now prefers the live libvirt value (DB is fallback-only, e.g. when the host is unreachable), and apply_vm_settings re-syncs the DB row on every successful settings apply so the cache stops drifting going forward.

0.43.8.2 — 2026-08-09

Added

  • DIMM slot population map, read via iDRAC Redfish. Third addition to Hardware Lifecycle Manager's out-of-band telemetry (after Network and Storage/RAID in 0.43.7.9): a new Memory (iDRAC) section on the host Hardware tab showing every physical memory socket (Systems/{id}/Memory) — populated vs. empty, capacity/type/speed/ manufacturer/health per installed module, plus a "N of M slots populated / total GB" summary. GET /api/hosts/{ip}/memory follows the same dynamic-discovery convention as the other Redfish endpoints, so a BMC that only lists populated sockets (confirmed live behavior on one Dell iDRAC generation) degrades gracefully rather than reporting a wrong empty-slot count.

0.43.8.1 — 2026-08-09

Fixes

  • Add Host for Ceph never installed python3-jinja2, so every node joined after the first couldn't run cephadm shell. bootstrap_ceph_task installs python3-jinja2 on the first (bootstrap) node — a fix from a prior incident — but add_ceph_host (routers/ceph/hosts.py, used for every subsequent node) had its own separate package list that never picked up the same fix. This stayed hidden as long as ceph_clusters.monitor_ip kept pointing at the original bootstrap node; it surfaced the moment the active mgr (and now, the new peer-failover fallback from 0.43.8.0) sent cephadm-shell traffic to a node added later. add_ceph_host's install list now matches bootstrap's.

0.43.8.0 — 2026-08-09

Fixes

  • Ceph dashboard went dark when the seed monitor node was down for maintenance, even though the cluster itself was healthy. core/ceph_client.py already followed the dashboard's own HTTP 303 mgr-failover redirects, but that only works once some node answers — if ceph_clusters.monitor_ip itself was fully unreachable (host rebooting, network down), there was nothing to redirect from and every call failed outright with a raw connection error, both for a fresh client and for one already cached from before the node went down. CephApiClient now falls back through the other members of the same cluster (compute_nodes.cluster_name) when its seed node won't answer at all, and self-heals ceph_clusters.monitor_ip to whichever peer actually responded — so the console's "Primary Monitor" tracks the real active mgr after a failover instead of needing a manual DB edit.

0.43.7.9 — 2026-08-09

Added

  • Out-of-band Network and Storage/RAID health, read via iDRAC Redfish. Extends Hardware Lifecycle Manager's existing BMC telemetry (system, power, temps, fans, PSUs) with two more read-only views on the host Hardware tab: Network (iDRAC) — per-NIC link status/speed/health from Systems/{id}/EthernetInterfaces, out-of-band so it still answers even if the host OS is down (unlike the existing SSH/ethtool Physical NICs table) — and Storage Health (iDRAC) — RAID controllers, physical drives (health, media type, predicted-failure), and virtual disks/volumes from Systems/{id}/Storage, the first place this console shows any disk health data at all (the existing Unallocated Disks table is SSH/lsblk, capacity only, no health). Both new GET /api/hosts/{ip}/network and GET /api/hosts/{ip}/storage endpoints discover resource paths dynamically from /redfish/v1 like every other Redfish call in core/fencing.py/routers/hosts/hardware.py, so they work against any Redfish-compliant BMC, not just Dell iDRAC.

0.43.7.8 — 2026-08-08

Fixes

  • CRS Predictive Trigger (ML) was live in the UI but non-functional. The frontend toggle/status card (static/js/hosts.js) and DB schema (crs_ml_models table, clusters.crs_predictive_enabled column) had already been deployed, and the Celery beat schedule had already been firing tasks.crs.train_predictive_models every 6h — but the task itself, the core/crs_forecast.py XGBoost forecaster, core/crs.py's predictive-trigger hook into the rebalance pass, and the routers/inventory.py API (crs_predictive_enabled param plus the /crs/predictive/status and /crs/predictive/train endpoints) had never shipped. Beat had therefore been failing on an unregistered task every 6 hours since the schedule went live, and clicking the UI toggle 400'd with "No fields to update." Deployed the missing backend pieces and added xgboost (plus scikit-learn, which xgboost's sklearn-API wrapper needs at runtime but doesn't pull in itself) to requirements.txt.

0.43.7.7 — 2026-08-08

Fixes

  • Fleet-wide compute-node worker crash loop. core/fencing.py (added earlier today for Redfish/iDRAC TLS pinning) was never added to WORKER_CORE_MODULES (core/config.py), the curated list of core/ files pushed to enrolled KVM nodes. tasks/hardware.py imports it at module scope, and Celery imports every task module at worker startup — so as soon as any node received a code sync (Repair Agent or a fresh provision) after today's Redfish change went out, its worker crash-looped forever on ModuleNotFoundError: No module named 'core.fencing', with "Repair complete" still reported since the restart step doesn't check whether the resulting process actually stayed up. All 7 enrolled compute nodes were affected simultaneously; manually recovered by pushing the missing file and restarting each node's mfcloud-worker directly. Fixed at the source by adding fencing to WORKER_CORE_MODULES so future syncs include it.

0.43.7.6 — 2026-08-08

Security

  • Login audit log could be spoofed with a fake source IP. /api/login built its logged client_ip by trusting the X-Real-IP/X-Forwarded-For headers unconditionally. Port 8000 is reachable directly on the LAN on every node (bypassing Caddy), so a caller hitting it directly could set those headers to any value and have it recorded as the source of a failed (or successful) login in the cluster log — observed live: failed admin login attempts logged from an address nothing on the LAN was using. The brute-force rate limiter was never fooled by this (it already only trusts the header from a verified proxy peer); only the audit-log field was. Fixed by reusing that same trust-checked resolver for the logged IP.

0.43.7.4 — 2026-08-08

Security

  • Security-group ACL rules were silently no-ops. The OVN ACL builder matched on the wrong port field for ingress vs. egress rules (core/ovn.py) — every "block this traffic" rule created through the Networks tab effectively matched nothing. Fixed; existing security groups now enforce as configured.
  • HA failover split-brain risk. check_ha_status (Celery beat, every 60s) had no protection against two overlapping runs both fencing the same dead node and restarting its VMs on two different survivors. Added a bounded lock around the monitor cycle and around each VM's restart dispatch.
  • iDRAC/BMC passwords are now encrypted at rest, matching every other stored credential in this app (AES-256-GCM). Existing plaintext rows keep working until next saved.
  • Redfish/iDRAC connections now use TLS certificate pinning (trust-on-first-use + SHA256, mirroring this app's existing SSH host-key pinning) instead of accepting any certificate.

Fixes

  • Throttled local backups (Traffic Control rate limit) could leave an orphaned qemu-img process running against a resumed VM's disk on a Celery timeout, corrupting the backup. Fixed for both backup and storage-migration convert jobs.
  • NIC bandwidth permission check could be bypassed on a transient read error; fixed to fail closed.
  • Reactivating a host that had been fenced now waits for stale VM definitions to be confirmed cleaned up before marking it Active again (previously fire-and-forget); the manual Power-On action is now blocked for a Failed/Fence-Failed host for the same reason.
  • Assorted Networks-tab tree/detail-pane fixes: breadcrumb sync on network-object clicks, stale data after Release/Allocate/Connect actions, and a DOM-id collision between the classic and inline Security Group/IPAM views.
  • Several smaller permission-ordering, NULL-handling, and validation consistency fixes across the NIC bandwidth and OVN switch/router APIs.

0.43.7.3 — 2026-08-08

Fixes

  • Traffic Control — Network Adapters modal's bandwidth column unreadable after 0.43.7.2 — the "Current Adapters" table's edit-row Bandwidth cell used placeholder="In"/"Out" to label its two inputs, but a placeholder never renders once an input has a value, and both always default to 0 — so in practice it showed two identical, unlabeled spinners. Replaced with real <label>s above each input (static/js/vms/vms-networking.js::editNICRow). Also widened the modal (620px → 780px, templates/components/vms/modals_hardware.html) — the table's existing Speed column had been getting crushed to an unreadable sliver ever since the Bandwidth column was added in 0.43.7.0, with no room left for 7 columns (#, Network, MAC, Type, Speed, Bandwidth, Actions) at the old width.

0.43.7.2 — 2026-08-08

Fixes

  • Traffic Control (0.43.7.0) — NIC bandwidth caps missing from the VM Settings hardware editor, Settings-page card disappearing for existing admins — two gaps found after the 0.43.7.0 rollout, both UI-only (the backend/API from 0.43.7.0 was already correct):
  • The Inbound/Outbound Mbps fields only reached the standalone Network Adapters modal (static/js/vms/vms-networking.js). The VM Settings → Hardware tab's inline adapter row — the primary way to edit a NIC — is rendered by a separate file, static/js/vm_settings.js, that 0.43.7.0 never touched, so Save/Add there silently reset any bandwidth cap back to 0. Added the same Limit In/Out fields to _vmsRenderNicRows/vmsAddNicRow/vmsSaveNicRow and the matching markup in templates/components/vms/modals_hardware.html.
  • The Settings page's widget grid (GridStack) persists each admin's drag/resize layout to their browser's localStorage. Any admin who'd already customized that layout before 0.43.7.0 shipped had a saved layout that didn't know about the new Traffic Control widget — GridStack's load() removes DOM widgets it can't find in the saved layout by default, so the card was silently deleted from the page for those admins (present in the served HTML, gone after GridStack ran). static/js/admin.js's initSettingsGrid() now calls load(saved, false), which only repositions/resizes widgets already known to both sides instead of pruning ones that aren't — a saved layout can no longer delete a widget added after it was written.

0.43.7.1 — 2026-08-08

Fixes

  • Networks tree/table decluttering — a Cluster VLAN Fabric entry is stored as the same underlying networks row as a plain portgroup, so it was showing up twice: once in the flat Portgroups list/tree (with no cluster context) and again in the Cluster VLAN Fabric table/tree (with its actual cluster, subnet, and trunk status). It now only appears in the latter, more specific one. Also: a cluster whose hosts have no bridges configured (nothing networking-relevant to show) now starts collapsed in the Networks sidebar tree instead of expanding every host row by default — still one click away, just not competing for attention with clusters that actually have something to show.

0.43.7.0 — 2026-08-08

New Features

  • Traffic Control — bandwidth/QoS management (Phases 1-3) — NIOC-style rate limiting for the three biggest uncontrolled bandwidth consumers, shipped in three layers:
  • Backup / migration rate caps — three new Settings → Traffic Control fields (tc_backup_limit_mbps, tc_cloud_backup_limit_mbps, tc_migration_limit_mbps, all 0 = unlimited by default). Local/NFS backups and storage migrations pass a shared -r <bytes/sec> flag into their qemu-img convert (new mbps_to_bytes_per_sec helper, tasks/utils.py); cloud backups get equivalent wall-clock pacing on the S3 upload loop (tasks/cloud_backup.py). The local-backup cap carries a suspend tradeoff (the VM is paused for the whole convert), surfaced as an inline warning in the settings card and enforced with a 50 Mbps floor when set above 0.
  • Per-VM NIC bandwidth caps — Add/Edit Network Adapter gained Inbound/Outbound Mbps fields, applied via libvirt's <bandwidth> element (tasks/network.py) and shown per-NIC in the adapter list. Changing a NIC's limit requires can_manage_networks (the existing Network Admin role, or Super Admin) — same gate and error message as every other Networks route; resending an unchanged limit (e.g. while editing VLAN on an already-capped NIC) doesn't require the permission.
  • Ceph mClock QoS profile — Ceph Administration tab gained a cluster-wide osd_mclock_profile selector (balanced / high_client_ops / high_recovery_ops), gated on can_manage_storage (routers/ceph/maintenance.py). Scoped to the profile switch only — the legacy osd_max_backfills/osd_recovery_max_active knobs from the original design were verified live to be no-ops under Ceph's mclock scheduler (the modern default since Quincy) and were deliberately left out rather than shipping non-functional controls.

Design: patch_note/design_artifacts/TRAFFIC_CONTROL_DESIGN.html. Phase 4 (host-level HTB via AWX node provisioning) is infrastructure-side, not app code, and remains unbuilt.

0.43.6.1 — 2026-08-08

Fixes

  • Network object detail panes always show their tab strip — the shared object-detail shell (openObjectDetail/openDetailPanel, static/js/detail-panel.js) used to hide the tab row entirely for an object with only one tab, so clicking an L2 Bridge, OVN logical switch/router, OVN chassis, or Cluster VLAN Fabric entry in the Networks tree landed on a bare Summary pane with no visible menu at all — inconsistent with every other object type in the console (VM, host, container, datastore, …), which already had 2+ tabs and so always showed the strip. Now every object's tab strip is always visible, even a single-entry one.

0.43.6.0 — 2026-08-08

New Features

  • Networks tab: tree navigation + object detail panes — the sidebar's Networks inventory view (the 🌐 tab) now lists every network object in a browsable tree instead of a flat, unclickable per-host network list: each host gets an "L2 Bridges" sub-folder (bridges are genuinely host-scoped), and a new top-level "Network Objects" folder holds Portgroups, Cluster VLAN Fabric, OVN Logical Switches/Routers/Chassis, Security Groups, and IP Address Pools (cluster/global-scoped, so they don't belong under any one host). Clicking any leaf opens an inline Summary pane — icon, status badge, key/value cards, and extra tabs where relevant (VMs for portgroups, Firewall Rules/Assigned VMs for security groups, Leases for IP pools) — reusing the same openObjectDetail/.mfd-*/.vmsum-* shell already used for VM/host/datastore detail views, so the new tree leaves match the rest of the console instead of introducing a new visual language. Destructive actions (Delete/Remove) reuse the existing confirm()-guarded functions from the Networks tab's CRUD modals — no duplicate delete logic. New file static/js/networks/tree_detail.js; tree-building changes in loadInventory() (static/js/main.js). The existing toolbar/tables "Networks" page (top context tab) is unchanged and still the primary create/bulk-manage surface — the tree is an additional drill-down path alongside it.

0.43.5.0 — 2026-08-06

New Features

  • Selectable vNIC link speed (1/10/25 Gbps) — the Add/Edit Network Adapter UI (both the standalone Network Adapters modal and the VM hardware editor's inline NIC rows) now has a Link Speed dropdown, so a guest's ethtool/Device Manager reports a real speed instead of virtio-net's default "Unknown!". Implemented via libvirt's qemu:override (tasks/vm/xml.py), which sets QEMU's virtio-net-pci speed/duplex device properties directly — libvirt's own <link> element has no speed attribute, so this is the only way to make virtio-net report a real link speed. Each such NIC gets a self-generated MAC and a stable ua-nic-<mac> alias so edit_nic/remove_nic can find and update or clean up its override entry later. Falls back gracefully (retries the defineXML without the override) if a node's libvirt is too old to support it, rather than failing the NIC add/edit outright. Scope: only NICs added or edited going forward get a speed — existing VMs' NICs are untouched, and the initial deploy-wizard NIC isn't wired up yet (planned follow-up).

0.43.4.9 — 2026-08-06

Fixes

  • Networks tab visual cleanup — the VLAN tag chip in the Open vSwitch Portgroups and Cluster VLAN Fabric tables was a solid-fill blue pill with white text (background:var(--accent) inline styles) that stood out against every other badge in the console, which reads the shared soft-tint .badge family instead. Replaced with a new badge-accent variant (tables.css) for tagged VLANs, and a badge-muted "Untagged" label — not the meaningless "VLAN 0" — for the default network. Same swap for the IPAM pool table's logical-switch reference chip, previously a hand-rolled inline pill duplicating what .badge already provides. The L2 Bridges table picked up matching cleanup while in the area: its badge-secondary class was referenced but never defined in CSS (silently rendered unstyled), now badge-muted; bridge status went from plain colored text to a real status badge; Delete buttons switched from inline red-outline styles to the shared .btn-danger class already used everywhere else. Toolbar reduced to one primary action (Create Portgroup) instead of two competing blue CTAs, and its 7 buttons regrouped by function (Layer-2/OVS, Topology, OVN, Security, IPAM) instead of an arbitrary left-to-right order.

0.43.4.8 — 2026-08-06

New Features

  • Performance page retired — its 8-range picker (30s/1m/5m/15m/30m/1h/6h/24h) is now built into the VM and Host Monitor tabs directly: CPU%/RAM% read a live 2s poll (ported from Performance's ring-buffer mechanism), the other charts index-slice the existing history fetch with a 20s auto-refresh so nothing visibly freezes next to the live charts. static/js/performance.js and its template are deleted, along with the nav tab. New onLeave tab callback in the shared detail-panel shell (static/js/detail-panel.js) stops a Monitor tab's live poll on tab-switch or navigating to a different object — a fixed dispatcher, not a mutable slot, since a naive "last poller wins" design silently leaks a poll interval whenever a sidebar tree click opens a second object before the first one's tab is ever revisited (freshly-opened objects always land on Summary, not Monitor). Containers, which also used Performance's target dropdown, have no Monitor tab of their own yet and lose their only range-picker view — known, accepted gap; they keep their existing separate Stats popup.

Fixes

  • VM Monitor's Disk Latency chart now shows an explicit "not available on this range" message on 6h/24h instead of a blank line — the rollup endpoint has no latency columns (only the live/raw endpoint does), a pre-existing backend gap this surfaces honestly rather than silently.

0.43.4.7 — 2026-08-06

New Features

  • Host Monitor tab redesign — the Host Detail Panel's Monitor tab (click a host → Monitor) is now a Quick Overview stat-tile strip (VMs, Logical CPUs, RAM%, CPU%, IO Wait%) + 5 single-purpose charts (CPU%, RAM%, IO Wait%, Network Rx/Tx with min/max/current, Disk Read/Write throughput), replacing a single combined CPU/RAM/IOWait chart. Preserves the existing Live/6h/24h range picker — all 5 charts now respond to range changes, where previously only one did. New host_telemetry.disk_rd_bps/ disk_wr_bps/net_rx_bps/net_tx_bps columns, populated by summing each host's own VMs' already-computed per-VM rates in tasks/telemetry.py (not a new host-level probe) — reflects VM-attributable I/O only, not true physical-NIC/disk-controller saturation. The 6h/24h rollup also now includes IO Wait, closing a gap where it previously only ever showed on the Live range.

0.43.4.6 — 2026-08-06

New Features

  • VM Monitor tab redesign — the VM Detail Panel's Monitor tab (click a VM → Monitor) is now a multi-panel dashboard instead of one badge strip + a combined chart: a Quick Overview stat-tile strip (Uptime, CPU MHz per core, RAM MB, CPU%, RAM%) plus 4 single-purpose charts — CPU%, RAM%, Network Rx/Tx (with a min/max/current stat row), and Disk Read/Write Latency. New vm_telemetry.disk_rd_lat_us/disk_wr_lat_us columns, derived in tasks/telemetry.py from libvirt block-stats time counters that were already being fetched but never read. New vms.last_started_at, stamped on a create/start Success task callback, powers the Uptime tile — pre-existing VMs show — until their next boot event (no way to backfill history that was never recorded). No "CPU Ready" panel: that's an ESXi CPU-scheduler metric with no KVM equivalent, so Disk Latency fills that slot with real data instead of a fabricated number. MFCloudTelemetry .buildOptions() gained a real auto-scale y-axis mode (yMax: false) for metrics with no fixed ceiling, reused by the new Network/Latency charts.

0.43.4.5 — 2026-08-06

Fixes

  • Performance tab target dropdown popped open light-themed — <select> popup lists (and other browser-native chrome: scrollbars, checkboxes, date pickers) ignore custom CSS on the closed control and instead follow the color-scheme property, which the console never set. Added color-scheme: dark to :root and color-scheme: light to the existing [data-theme="light"] override in base.css — fixes every native <select> in the app, not just Performance's target picker.

0.43.4.4 — 2026-08-06

New Features

  • Maintenance-exit VM-count restoration — exiting Maintenance now deterministically pulls back the same number of VMs a host was running before it went into Maintenance, picked randomly from anywhere in the cluster (not necessarily its original VMs). Previously, exiting Maintenance only ran CRS's load-based rebalance, which might move nothing back at all if the cluster's CPU/RAM formula didn't judge it "imbalanced." New compute_nodes.pre_maintenance_vm_count column, stamped on entry (api_host_maintenance) and consumed on exit by new core.crs.restore_vm_count() (routers/hosts/core.py, core/crs.py). Respects the same safety filters as CRS's other moves (CRS: exclude tag, cooldown window, local-disk storage, DRBD replica-node placement); works regardless of clusters.crs_enabled since this is maintenance-lifecycle symmetry, not opt-in load balancing.

Fixes

  • CRS VM-count trigger could be skewed by inactive hosts — count_of (added in 0.43.4.x's CPU+RAM+count rebalance work) tallied VMs from every host in the cluster with no Active-status filter, unlike the CPU/RAM axes. A Failed/Maintenance host with stale VM rows still assigned to it could dominate the hot-host count and mask a real imbalance between the actually-Active hosts for that pass. Now scoped to Active hosts only, matching cpu_of/ram_of (core/crs.py).
  • Corrected CRS_DESIGN.html's "Known limitations" section, which still claimed VM-count wasn't a balancing trigger — it is, as a fallback-only third axis.

0.43.4.3 — 2026-08-06

New Features

  • Hosts are labeled by hostname, not just IP — the inventory tree, Hosts table, host detail panel (title + new "Hostname" row), and Ctrl+K search now show a host's detected hostname wherever it's known, with the IP kept as a secondary detail (tree tooltip, table subtitle, "Address" row) rather than hidden. IP remains the internal identity everywhere — filters, routing, SSH — this is display-only. Detected via SSH hostname -f at Add Host time and self-heals lazily the first time a host's Summary panel is opened, same pattern as the existing manufacturer probe. New admin action, Node Ops ▾ → Detect Host Names, sweeps every already-enrolled host missing one in a single click instead of opening each Summary panel by hand (POST /api/hosts/backfill-hostnames). Hosts with no detected hostname yet keep showing their IP everywhere, unchanged (routers/inventory.py, routers/hosts/core.py, routers/hosts/status.py, services/host_tasks.py, static/js/main.js, static/js/hosts.js, static/js/search.js).

0.43.4.2 — 2026-08-06

Improvements

  • Dell vendor badge moved from the inventory tree to the host's Summary panel — the badge introduced in 0.43.4.0 crowded the sidebar tree next to already-tight IP labels; it now shows as a "Manufacturer" row in the host detail panel's Summary tab (HOST card), alongside Hypervisor OS and BMC/iDRAC. Falls back to the raw dmidecode string as plain text for unrecognized vendors instead of only showing something for Dell. Still shown in the Global Summary tab's Hosts/Control-Plane table, which has the room for it (static/js/hosts.js, static/js/main.js).

0.43.4.1 — 2026-08-06

Improvements

  • Alarms moved into the Recent Tasks drawer; the floating bell is gone — the standalone alarm bell + popup panel (bottom-right, overlapping the Recent Tasks drawer) is replaced by a new "Alarms" tab inside the drawer itself, next to "Tasks"/"Cluster log". A small warning-colored chip in the drawer's always-visible header shows the nagging (critical/warning) alarm count at a glance — click it to jump straight to the Alarms tab — so there's one notification surface instead of two competing corner widgets. Same data source and dismiss/clear behavior as before (GET /api/alarms, per-browser dismissal in localStorage); static/js/alarms.js now renders into the drawer instead of injecting its own floating DOM (static/js/tasks-drawer.js, templates/components/dashboard.html).

0.43.4.0 — 2026-08-06

New Features

  • Hardware-vendor badge on hosts (Dell) — a host whose chassis reports as Dell (dmidecode -s system-manufacturer) now shows a small Dell EMC badge next to it in the inventory tree and the Hosts/Control-Plane table. Detected automatically at Add Host time (services/host_tasks.py) and backfilled the first time an already-added host's Summary panel is opened (routers/hosts/status.py) — no manual step for existing hosts. Purely cosmetic identification, same idea as the App Catalog's per-appliance logos: static/js/main.js's hostVendorBadgeHtml() falls back to a plain text pill if a vendor's logo asset is ever missing, so this never renders a broken-image icon. New compute_nodes.manufacturer column (core/schema.py); exposed via /api/hosts, /api/inventory, and /api/hosts/{ip}/summary.

0.43.3.2 — 2026-08-05

Bug Fixes

  • Host detail panel's VMs tab had no way to filter, sort, or right-click a VM — the per-host "VMs (N)" tab (opened by clicking a host in the inventory tree) rendered a plain static table with no search box, no column sorting, and no context menu. Added a search box (name/IP/owner)
  • status dropdown filter, click-to-sort column headers, and a Guest IP column (vms.guest_ip, already populated by the existing sweep_guest_ips beat task but not previously surfaced here). Row right-click now opens the same full VM action menu used by the main inventory tree and VM leaves (routers/hosts/status.py, static/js/hosts.js).

0.43.3.1 — 2026-08-05

New Features

  • Configuration Parameters (raw QEMU launch-arg passthrough) — VM Settings → Advanced gained a "Configuration Parameters" row for raw -set key=value QEMU launch overrides via libvirt's qemu:commandline passthrough. Gated behind a new dedicated can_edit_advanced_config permission (not just VM-write) — global_admin/local admin always have it, everyone else needs it granted explicitly via a custom role in Identity → Roles. Offline-only (these are QEMU launch args, only read at process start) and applying any parameter "taints" the domain (live migration may become unsupported). New endpoint PUT /api/vms/{id}/advanced-config (routers/vms/settings.py, tasks/settings/advanced_config.py).
  • VM Settings Options and Advanced tabs redesigned into collapsible device rows — matches the Hardware tab's layout from 0.43.3.0: Boot Options, Machine Type, Secure Boot, and Guest Notes on Options; CPU Affinity, GPU Assignment, and PCIe Passthrough on Advanced. Each row shows a live summary (e.g. "UEFI · Boot: Hard Disk · Delay: 0ms", "No GPU assigned") when collapsed.

Bug Fixes

  • Ejecting/attaching CD/DVD media on a running VM silently didn't reach the live guest — set_cdrom (tasks/settings/disks.py) only ever called defineXML(), which rewrites the persistent (INACTIVE) libvirt config but never the actually-running QEMU process. The UI reported success and the media appeared ejected, while the running guest kept the old disc mounted until its next reboot. Now also calls updateDeviceFlags(..., VIR_DOMAIN_AFFECT_LIVE) when the VM is running, so the change reaches the live guest immediately.
  • Adding a disk, SCSI controller, or network adapter always required powering off the VM — add_disk, add_scsi_controller (tasks/settings/) and add_nic (tasks/network.py) hard-refused whenever dom.isActive(). All three now attempt a live hot-attach (attachDeviceFlags(..., VIR_DOMAIN_AFFECT_LIVE)) on top of the persisted config when the VM is running; a hot-attach QEMU/libvirt rejects (e.g. IDE, which never hot-plugs) doesn't undo the persisted change — the device still appears after the next reboot instead of failing outright. Removing, resizing, or editing existing hardware still requires powering off.

0.43.3.0 — 2026-08-05

New Features

  • Virtual SCSI controller management — VM Settings can now add/remove SCSI controllers on a VM independently of disks, choosing a model per controller (VirtIO SCSI, LSI Logic Parallel, LSI Logic SAS, BusLogic). Multiple controllers with different models can coexist on one VM — e.g. an LSI Logic controller keeping a V2V-imported boot disk bootable before virtio drivers are injected, alongside a VirtIO SCSI controller for new, fast disks. Adding a SCSI-bus disk with no controller picked now explicitly creates a VirtIO SCSI controller if the VM has none, instead of relying on libvirt's implicit (and machine-type-dependent) default. New endpoints POST/DELETE /api/vms/{id}/controllers (routers/vms/settings.py, tasks/settings/controllers.py); removal is refused server-side if a disk is still attached to that controller.
  • VM Settings Hardware tab redesigned into collapsible device rows — CPU, Memory, each Hard Disk, each SCSI Controller, each Network Adapter, Video Card, and each CD/DVD drive now render as a single-line row that expands to its full edit form, with one "Add New Device" menu replacing the previously scattered per-section Add buttons. Network adapters are now editable inline (network/VLAN change, remove) without leaving the Settings modal — the standalone NIC Manager dialog is unchanged and still used from other entry points.

0.43.2.2 — 2026-08-05

Bug Fixes

  • Settings page "Update Now" button never showed failure/timeout feedback — pollForConsoleRestart() (static/js/main.js) hardcoded the top banner's button id, so the Settings-page trigger (admin.js) polled the wrong element and got stuck on "Restarting…" forever if an update ever stalled, with zero indication to the operator. Both callers now share one triggerConsoleUpdate() helper parameterized by button id, so this can't drift out of sync again.
  • VM Monitor chart's live x-axis labels ignored the Display Timezone setting — refreshAll() (static/js/main.js) still used the browser's local time instead of the shared mfFormatTime() helper every other timestamp on the page already uses.
  • mfLoadDisplayConfig() fired an authenticated request on every page load, including the login screen — now guarded the same way alarms.js already guards its own startup call, instead of firing before a token exists.

Reliability

  • DB pool init's startup retry loop could stall the event loop — core/schema.py's init_db() (added in 0.43.0.1) used a blocking time.sleep() inside FastAPI's async lifespan(), capable of stalling the event loop up to ~45s on a DB-unreachable start. init_db() is now async and awaits asyncio.sleep() instead.

Housekeeping

  • static/js/containers/snapshots.js's timestamp formatter now calls the shared mfFormatDateTime() helper instead of a hand-rolled duplicate.
  • SettingsModel.timezone (models/schemas.py) now validates against zoneinfo — an invalid IANA zone name is rejected with a 422 instead of being silently saved.

0.43.2.1 — 2026-08-05

Bug Fixes

  • Clicking a host in the inventory tree didn't open the host detail panel at all — setFilter() (static/js/main.js) had a dedicated inline-detail branch for VM and container leaves (openVmSummary/openContainerSummary, filling the main pane) but hosts fell through to the generic "scope the Summary tab's tables" branch instead. The richer host detail (Summary/VMs/ Monitor tabs, shipped in 0.43.2.0) was only reachable via the tree's right-click "Summary" action, as a centered popup — inconsistent with every other leaf type. openHostSummary() now renders inline (openObjectDetail instead of openDetailPanel) and a host leaf click opens it directly, matching VM/container behavior exactly.

0.43.2.0 — 2026-08-05

New Features

  • Host detail panel gained a VMs tab — clicking a host in the inventory tree opens the same shared tabbed detail panel as everything else, now with a "VMs (N)" tab listing every VM on that host: live Running/Stopped state (cross-referenced against libvirt, not just the DB), vCPU, memory, and owner, each row clickable through to that VM's own detail panel. GET /api/hosts/{ip}/summary (routers/hosts/status.py) now returns the per-VM list alongside the existing capacity/OS summary; a VM libvirt doesn't report (e.g. mid-migration) still shows with an "Unknown" state instead of silently vanishing from the list.

0.43.1.2 — 2026-08-05

Quality of Life / UI Improvements

  • HA Cluster status moved off a floating overlay onto the Summary tab — static/js/ha_status.js used to render a fixed-position 🧬 FAB (bottom-left) that opened a floating panel over whatever page content was underneath it. It's now a plain widget card (#cpt-ha-widget) alongside Host Availability / Host Health / Alert Summary on the Global Datacenter Summary tab — same primary/replica-per-host data, no more overlap. Still hidden entirely on a non-HA (single-node) console.

0.43.1.1 — 2026-08-05

Bug Fixes

  • VM live-migration silently did nothing with an empty target-host list — submitVmMigrate() (static/js/vms/vms-operations.js) returned early with no toast/error whenever the target <select> had no options, which happens whenever activeNodes (hosts with status === 'Active') doesn't include any host other than the VM's own. The most common cause: putting a host into KVM-level Maintenance (POST /api/hosts/{ip}/maintenance, distinct from Ceph's own OSD-flag maintenance toggle) removes it from that pool. Working as intended once there's nowhere else to go — the bug was the operator getting zero feedback. Now warns immediately when the Migrate modal opens with no eligible target, and again if Migrate is submitted anyway.

0.43.1.0 — 2026-08-05

New Features

  • Two-tab "Tasks" / "Cluster log" view on the Recent Tasks drawer — the expanded drawer is now a real two-tab datagrid instead of a stack of compact cards. The Tasks tab shows Start Time / End Time / Node / User name / Description / Status, sourced from task_history (which gained a host_node column, populated from the reporting compute node's IP on real VM-lifecycle callbacks, or the serving CP replica's own identity for control-plane-only actions). The Cluster log tab (admin-only, new GET /api/logs/cluster) is backed by a brand-new cluster_log table — MFCloud had no daemon/service-level audit trail before this, only the per-VM task_history. It now captures login success/failure (routers/auth.py, previously unlogged entirely) and the terminal outcome of every task (one line per completion, not per progress tick, via a new core/cluster_log.py helper) with node/service/PID/severity — retained 30 days by the existing reaper loop.

0.43.0.1 — 2026-08-05

New Features

  • Display Timezone setting — System > Settings > Platform Configuration gained an IANA timezone picker (e.g. America/Chicago) controlling how already-UTC timestamps render across the console. Storage and scheduling stay UTC-only on purpose (core/schema.py, tasks/app.py) — this only affects display, via two new shared helpers in main.js (mfFormatDateTime() / mfFormatTime()) threaded through the pages that render timestamps.
  • Software Update widget on the Settings page — a manual counterpart to the existing top-nav update banner, so an admin can check for and trigger a console update from System > Settings without waiting for the banner.

Bug Fixes

  • Settings page widgets rendered as raw HTML text — the GridStack layout persistence added for the Settings widget grid called save() with no arguments, which defaults to saveContent: true and snapshots each widget's live innerHTML into localStorage. On the next page load, GridStack's default content renderer writes that saved string back with textContent (an XSS guard, not innerHTML), dumping the widget markup onto the page as literal visible text instead of parsed elements. static/js/admin.js's initSettingsGrid() now saves position/size only (save(false)) — these widgets are static markup from view_settings.html, not GridStack-managed content — and strips any legacy content field from a previously-saved layout on load so browsers with the already-corrupted layout in localStorage self-heal.
  • DB pool init could crash-loop on a container recreate — right after a forced redeploy, a replica's container could start before db was resolvable in the container network's internal DNS; a single failed connection attempt used to crash the whole app on startup, and the orchestrator restarting it hit the same race every time (seen live 2026-08-05: a replica crash-looped for ~48 minutes, flapping the HA dashboard as HAProxy round-robined onto the dead replica). core/schema.py's init_db() now retries with backoff (8 attempts, capped at 10s) before giving up.

0.43.0.0 — 2026-08-05

Consolidates the internal 0.42.9.6 → 0.42.9.18 iterations (never individually published) into this release, the first pushed to Docker Hub since 0.42.9.5.

New Features

  • GPU-as-a-Service (V41-G1), passthrough v1 — host GPU inventory/scan (gpu_devices table, tasks/gpu.py::scan_host_gpus via lspci -k -nn) and VM GPU assignment (PUT/DELETE /api/vms/{id}/gpu) reusing the existing PCI-passthrough XML path, with a 1-GPU-1-VM availability check. Whole-GPU VFIO passthrough only for v1 — SR-IOV slicing deferred.
  • Hard/soft VM placement affinity rules — a new affinity_rules table lets any affinity_group tag be flipped from the existing soft anti-affinity spread into hard anti/affinity enforcement in core/placement.py, with CRUD at /api/affinity/rules and a management table on the IAM dashboard.
  • Tenant node-count footprint cap — a fifth per-tenant quota ceiling (max_nodes) alongside the existing vCPU/RAM/K8s-cluster limits.
  • ZFS: child datasets (POST /{pool_id}/dataset), a dedicated "ZFS Local Storage" toolbar button/panel (Pools / Import / Replication), and ARC memory usage telemetry on the datastore summary panel.
  • ~17 new App Catalog appliances (Django, Drupal 7/9, BookStack, Ansible, Canvas LMS, and others).

Quality of Life / UI Improvements

  • Deploy Virtual Machine rebuilt as a step wizard — a guided left-rail flow (VM Identity → Source & Placement → Storage → Hardware → Provisioning → Network → Ready to Complete) replacing one long scrolling form, and no longer requires a password or SSH key to submit.
  • Container Summary panel brought to parity with the VM Summary panel — Related Objects (Cluster/Host/Networks), a live NIC hardware list, Snapshots count, Tags (new tag editor), Notes, and live IP addresses read from LXD instance state.
  • Related Objects / Snapshots / Tags / Notes cards added to the VM Summary panel, now that it's the primary way to inspect a VM.
  • Removed the flat "Virtual Machines" and "LXD/Incus Containers" tables — the sidebar tree already lists every VM/container, and clicking a leaf now opens an inline Summary panel instead of jumping to a table. Import VM (V2V) and Scan VM Storage moved into the header's Actions dropdown.
  • Ceph modal rebuilt as a 5-tab panel (Summary, Nodes, OSDs, Health & Monitoring, Administration) instead of 9 stacked sections; Networks page toolbar reduced from 9 buttons to 7 with OVN object creation consolidated into one modal and Security Groups split into 2 tabs; DRBD storage tab moved to the shared ⋯/right-click context-menu pattern already used by ZFS pools and NFS datastores.
  • VM list "vCPU" column renamed to "CPU"; Memory column now shows allocated GB instead of just usage %.

Performance

  • /static assets (all ~60 JS/CSS files) now serve with a one-year immutable Cache-Control header instead of being re-validated on every page load — safe since every reference is already cache-busted with ?v={{ app_version }}.
  • zfs_live_log / ceph_live_log WebSocket handlers ran blocking DB/SSH calls directly on the async event loop, able to stall every other request on the server while a log stream was connecting; moved to a background thread.

Security Fixes

  • Cross-tenant VM deletion via a self-issued API key — an unscoped api_keys fallback on the internal task-update endpoint let any authenticated user delete another tenant's VM inventory record; removed, the endpoint now only accepts the shared INTERNAL_SECRET header or a registered compute-node IP.
  • Missing ownership check on VM→template conversion — any VM-manage user could convert another tenant's VM into a template; now gated by the same tenant-isolated ownership check every sibling VM endpoint already uses.

Bug Fixes

  • VM-to-OVN dataplane wiring was silently incomplete even with a correctly paired northd/controller — tap interfaces are now bound to br-int with a UUID-based logical switch port identity, plus an ACL reapply race condition fixed with a Postgres advisory lock. Verified live end-to-end (VM-to-VM traffic, Security Group enforcement).
  • Ceph RBD-backed VMs could not live-migrate ("Migration without shared storage is unsafe") — new Ceph disks now use cache='none' instead of writeback.
  • VM list State badges/HA tags rendered as unstyled plain text (wrong CSS classes); Memory column always showed 0% (RAM usage was never populated).
  • Detail-panel backdrop click skipped cleanup, leaving WebSocket connections (ZFS Live Log, Ceph cluster stream) open after dismissal.
  • Proxmox V2V connection form didn't trim the API token secret.

0.42.9.5 — 2026-08-03

Fixed

  • Three more Docker Scout findings on the published console image — CVE-2025-47273 / CVE-2026-59890 (setuptools 70.3.0) and GHSA-6v7p-g79w-8964 (msgpack 1.1.2), all three reported against pip/_vendor/bom.cdx.json inside /opt/venv. Root cause: pip itself vendors msgpack (for its CacheControl HTTP cache) and records setuptools 70.3.0 as its own build-tool provenance in that bundled SBOM — true even of the newest pip release (26.2), so no version bump could ever clear it. pip was only ever needed to populate /opt/venv during the builder stage and is never imported at runtime; mfconsole-ship/Dockerfile.ship now deletes pip (binaries + package + vendor bundle) from /opt/venv right after it's copied into the runtime stage, leaving the top-level setuptools/wheel packages (which something might still need via pkg_resources) untouched.

0.42.9.4 — 2026-08-03

Fixed

  • CVE-2026-69247 (CVSS 8.2, High) in the bundled Python cryptography package — Docker Scout flagged cryptography 49.0.0 (affected range >=44.0.0,<50.0.0) in the published console image shortly after the 0.42.8.14 push. requirements.txt's floor bumped to >=50.0.0. Also pinned the separate system-level cryptography copy in mfconsole-ship/Dockerfile.ship (installed to defeat scanners picking up the unused RPM-installed version) — that layer was Docker-cached and unpinned, so it could silently keep serving whatever version was resolved the first time it was ever built, independent of requirements.txt.

0.42.8.14 — 2026-08-03

Fixed

  • OVN networking was non-functional for live traffic fleet-wide — root-caused to a version mismatch between ovn-northd (the control-plane compiler) and ovn-controller (the per-chassis agent). Each ovn-controller release reserves a different, fixed number of physical OpenFlow tables for the logical ingress pipeline; a newer northd compiling a bigger pipeline than an older controller's table budget causes a legitimate ARP-responder loopback packet to get misrouted back into the ingress pipeline forever, dropping all traffic between VMs on the same logical switch. Fixed by aligning the control plane to the release already running fleet-wide. Verified against real VM-to-VM traffic on the same chassis and across separate physical hosts. Security Group ACL enforcement — previously never validated against live traffic — is now confirmed working end-to-end through the real API (rule applied → traffic blocked; rule removed → traffic restored).
  • Security Group ACL removal could silently leave a stale rule in place when two VMs' group-membership changes landed close together — a non-atomic read-then-delete against OVN's shared ACL table with no locking. Now serialized per logical switch with a transaction-scoped Postgres advisory lock, safe across all 3 HA app instances.

Added

  • OVN chassis bootstrap can pin an exact release (new ovn_version setting) instead of always installing whatever's newest in the repo — the version-mismatch bug above happened because a freshly-provisioned node silently grabbed a newer release than the rest of the fleet.
  • Security Group ACL self-heal sweep — a new 5-minute background pass reconciles every VM's actual OVN ACLs against what its group membership says they should be, closing a gap where a missed background task (app restart, transient OVN failure) could leave permanent, silent drift.

0.42.8.13 — 2026-08-03

Added

  • Submit Firmware Update form in the host Firmware tab (Hardware Lifecycle Manager). Component picker pre-filled from the inventory already shown, image URL + target version fields, and a live status hint showing whether the host is in Maintenance or the selected component qualifies for the BMC-self exception — before you click Submit, not after a 409. Type-to-confirm (mfDangerConfirm) gate before submission, since this is a real Redfish SimpleUpdate call, not a simulation.

0.42.8.12 — 2026-08-03

Added

  • Hardware Lifecycle Manager (HLM), Phase 1-3: Dell iDRAC/Redfish firmware inventory and updates, no separate vendor management console required. New "Firmware" tab in the host Hardware Health modal lists installed firmware (BIOS, iDRAC, NICs, PERC/HBA, disks, PSU, ...) via the standard DMTF Redfish UpdateService — vendor-neutral discovery, same pattern already used for host health. POST /api/hosts/{ip}/firmware/update submits a real SimpleUpdate job and a new 30s beat sweep (tasks/hardware.py) polls it to completion without blocking the request. Hard safety gate: submission is rejected (409) unless the host is already in Maintenance (evacuated), with one narrow, server-verified exception for updates that target the BMC/iDRAC itself — those only reboot the out-of-band controller, never the host OS or its VMs. Live-validated end to end against real VxRail hardware: iDRAC firmware 7.00.00.172 → 7.00.00.173, checksum-verified against Dell's published SHA-256, submitted, polled through completion, version confirmed afterward — host stayed Active throughout, its VMs untouched. See patch_note/design_artifacts/HARDWARE_LIFECYCLE_MANAGER_DESIGN.html.

0.42.8.11 — 2026-08-02

Fixed

  • Node-side worker crash-looped with ModuleNotFoundError: core.zfs_node_setup on every EL9 (Rocky 9) compute node, mfs690 included — confirmed live via its systemd journal (mfcloud-worker.service, restart counter over 100). core/config.py's WORKER_CORE_MODULES (what should get pushed to nodes) already listed zfs_node_setup, but mfconsole-ship/build_cython_node.py — the separate script that Cython-compiles the Python-3.9-ABI bundle EL9 nodes actually receive — has its own hardcoded module list that was never updated to match. The module compiled fine and looked correct in WORKER_CORE_MODULES, but its .so never existed in the shipped image's node_bundles/el9/ for any node to receive; the push silently found nothing to send. Added the missing entry, plus cross-reference comments in both files so the next single-file addition to WORKER_CORE_MODULES doesn't silently drift the same way.

0.42.8.10 — 2026-08-02

Fixed

  • Single-server console refused to start after rotating any secret (SystemExit: 2 from core.auth.verify_secret_key_cluster_consistency, requiring a manual UPDATE settings ... WHERE key='secret_key_fingerprint' to unblock) — the cluster-consistency check that catches a real multi-node HA problem (one replica shipping a different SECRET_KEY than its peers, live incident 2026-07-17) doesn't distinguish "another replica disagrees" from "no replicas exist, the operator just rotated their own key while reusing the same persistent database" — the far more common case for a single-server install (e.g. re-running setup.sh --force). It now checks ha_cluster_hosts first: with no registered peers, a mismatch self-heals by adopting the current key as the new fingerprint instead of refusing to start; the strict refuse-to-start behavior is unchanged once real HA peers are registered, which is the only scenario the check exists to protect.

0.42.8.9 — 2026-08-02

Fixed

  • Single-server installs could fail to recreate app-1 with "no such host: registry.mfcloud.io" — MFCONSOLE_IMAGE was only ever set as a shell export scoped to install.sh's own run, never persisted into .env. Any later docker compose command (a plain up -d --force-recreate, a reboot, an operator re-running compose by hand) started a fresh shell with no such export, fell back to compose.prod.yml's hardcoded default — which still pointed at the old self-hosted registry, dead since the 2026-07-20 move to the public Docker Hub org. install.sh now writes MFCONSOLE_IMAGE into .env so every future compose invocation resolves it the same way, and the fallback default itself now points at docker.io/mfconsole/kvm-manager instead of the dead host, in case .env is ever missing the line entirely.
  • watchtower sidecar (0.42.8.8's "Update Now" button) crash-looped on current Docker Engine — its bundled client defaulted to Docker API 1.25, which Engine 28+ rejects outright ("Minimum supported API version is 1.40"). Pinned DOCKER_API_VERSION explicitly in compose.prod.yml.

0.42.8.8 — 2026-08-01

Added

  • "Update Now" button on the Docker Hub update banner — the console can now pull and install a new release itself instead of just linking out to Docker Hub. Backed by a new watchtower sidecar in compose.prod.yml (the registry-pull topology only — not present when running from source): it's the ONLY container in the stack with docker.sock access, deliberately kept out of the public-facing app container (an RCE there would otherwise mean full host takeover, since docker.sock access is host-root-equivalent). Watchtower only acts when the button's click hits its token-gated internal HTTP API (POST /system/update-console, Super Admin only) — never on a timer, and scoped by label to just app/app-1/worker/beat so it can never touch the database, Redis, or LINSTOR. The token is generated by setup.sh alongside the other secrets. The button shows live progress through the brief app-1 restart and reloads the page once the new version answers.

0.42.8.7 — 2026-08-01

Fixed

  • Single-server (all-in-one) installs could end up with an unrestricted VNC console, or a broken one — three compounding bugs, all specific to the topology where the console itself runs from the public mfconsole/kvm-manager Docker Hub image on the same host it also manages as a compute node:
  • The firewalld setup step in node provisioning declared success unconditionally (;-joined commands with no real exit-status check), so a node where firewalld was masked — common when Docker/Podman has been set up on the box — silently stayed inactive while the log claimed it was enabled. VNC (5900-6500) was left open on the host's public interface with no restriction at all.
  • LXD install failed on EL9 nodes with Unable to find a match: snapd — dnf install epel-release snapd in one transaction fails because dnf resolves against repos enabled before the command runs; EPEL's repo file isn't visible to the solver within that same call even though epel-release is being installed by it. Now split into two dnf calls.
  • Once firewalld actually runs, the VNC restriction rule that allowlists CONTROL_PLANE_IP never matched traffic from a co-located master container — Docker delivers container→host-owned-IP traffic locally with no NAT, so it arrives with the container's bridge-internal source IP instead. Unpatched, enabling firewalld on this topology would have broken the console instead of just leaving it open. The VNC restriction step now also detects a co-located master container (publishing port 8000) and allowlists its bridge subnet. Also hardened setup.sh: a manually-entered CONTROL_PLANE_IP is now validated against the host's actual interfaces instead of being accepted unchecked — a wrong value here fed silently into the firewall allowlist, sshd's penalty-box exemption, and the worker callback pin.

0.42.8.6 — 2026-08-01

Fixed

  • Ceph pool creation could silently revoke a live VM's RBD access — POST /api/ceph/pools rebuilt client.libvirt's cephx OSD caps from only the pools formally assigned to a cluster (clusters.ceph_pool_name) plus the pool being created, so a pool created and used by a VM disk without ever being assigned to a cluster would lose access the next time a different pool was created — the VM then failed to boot with a silent Permission denied. The rebuild now first reads client.libvirt's current caps (ceph auth get) and unions in whatever it already has, so the cap set can only grow, never silently shrink.
  • Alarms: a dead Celery worker could vanish from the dashboard after 24h, even on a still-active node — the "silent worker" alarm's 24h floor (added to stop a decommissioned/re-IP'd node's frozen last_seen from becoming a permanent ghost alarm) applied unconditionally, so a node that's still enrolled and Active but has a stuck worker (e.g. the 2026-07-31 Redis-auth incident) dropped off the alarm list exactly when it had been broken longest — the Hosts alarm block only covers SSH/libvirt-level unreachability, not a dead Celery process. The floor now only suppresses the alarm for nodes that are no longer Active.
  • Host repair: a Redis password containing &, |, or \ could corrupt the worker env file — the auto-repair broker-URL patch built its sed replacement directly from CELERY_BROKER_URL (which embeds REDIS_PASSWORD); those characters have special meaning to sed's substitution syntax (whole-match reinsertion, delimiter, escape) and weren't escaped for it, unlike the existing shell-level shlex.quote(). A password containing one could write a garbled broker URL or break the sed script outright — caught by the existing readback check (so it failed loudly, not silently) but still left the node stuck. The replacement text is now escaped for sed separately from the literal value used for comparison/append.
  • XSS: several list views were vulnerable to attribute-breakout via names read from external systems — Ceph pool names, LINSTOR/DRBD resource names, NFS datastore names, ZFS pool/host names, network names, LVM pool names, and cluster-VLAN names were interpolated into inline onclick="fn('...')" handlers with escaping that only handled the JS string delimiter (backslash/single-quote), not the HTML attribute delimiter (double-quote) — or, in several spots, HTML-escaped before JS-escaping the delimiter, which doesn't protect against breakout either (the browser HTML-decodes the attribute before the JS engine ever sees it). A crafted name — e.g. from a pool or resource created outside this app's own name-validated create endpoint — could break out and run arbitrary JS for anyone viewing that list. All 9 call sites across ceph_core.js, drbd_core.js, nfs.js, zfs.js, main.js, lvm.js, and vlan_fabric.js now go through one shared, correctly-ordered escapeJsAttr() helper in api.js.

0.42.8.5 — 2026-07-31

Fixed

  • Emergency Host UI (:9443) unreachable on every enrolled compute node — services/node_provisioner.py's firewall setup (run on both Add Host and Repair Worker) opened 8443/tcp but never 9443/tcp, the port mfcloud-agent.service actually listens on (agent_overlay/mfcloud-agent .service). 8443 is genuinely used too, just by LXD's cluster API, not the Emergency UI — the two got conflated. The service itself was always installed and running correctly; the firewall silently dropped every inbound connection to it. Found live: 192.168.12.10:9443 timed out from the console despite ss -tlnp on the node showing it listening. Fixed the port list, and corrected mfcloud-kb/docs/reference/ports.md, which documented the same wrong port. Existing nodes enrolled before this fix need a Repair Worker run (or a manual firewall-cmd --add-port=9443/tcp --permanent --reload on the node) to pick up the corrected rule — it isn't retroactive.

0.42.8.4 — 2026-07-31

Fixed

  • Docker Hub update-available banner never fired — tasks/update_check.py's beat-scheduled check (every 6h) queried Docker Hub for the mfconsole/ mfconsole repo, but the console has shipped as mfconsole/kvm-manager since the move to public Docker Hub distribution (see install.sh, DOCKERHUB_OVERVIEW.md). The lookup 404'd silently every cycle (caught by the sweep's own error handling so it never disrupted beat), so Super Admins were never notified a newer image existed. Fixed the repo name in the check itself and in the banner's "pull the latest image" link, which was pointing at the same wrong Hub URL.

0.42.8.3 — 2026-07-31

Fixed

  • VM Settings: Q35→i440FX machine switch failed ("itco model of watchdog is only part of q35 machine") — libvirt materializes q35's implicit iTCO watchdog (and AHCI controller) in the domain XML; the chipset conversion left them behind and i440FX has no equivalent, so the redefine was rejected. The reverse direction had the matching bug waiting (i440FX's explicit IDE controller has no q35 equivalent). The conversion now drops the source chipset's implicit devices and lets libvirt regrow the target chipset's own set on redefine.

0.42.8.2 — 2026-07-31

Fixed

  • VM Settings: UEFI→BIOS conversion failed after toggling Secure Boot off ("cannot use feature-based firmware autoselection when firmware autoselection is disabled") — the Secure Boot on/off paths write a <firmware><feature/> block that is only legal while UEFI autoselection is enabled; the BIOS conversion removed the autoselect flag and loader but left that block (and the SB-only SMM feature) behind, so libvirt rejected the redefine. The BIOS conversion now strips both.

0.42.8.1 — 2026-07-31

Fixed

  • VM Settings: BIOS→UEFI conversion failed on VMs deployed without ACPI ("unsupported configuration: UEFI requires ACPI on this architecture") — BIOS i440FX Linux guests are deployed without <acpi/>, and the firmware conversion never added it, so libvirt rejected the redefine and rolled the entire settings change back. The UEFI and Q35 conversions now ensure <features><acpi/> the same way fresh deploys do.
  • VM Settings: bogus "Secure Boot needs UEFI firmware + Q35 machine" error on VMs that already had both. The Secure Boot=On → UEFI+Q35 forcing was one-way in the dialog: selecting BIOS or i440FX while Secure Boot was already On (pre-checked from the VM's live state) submitted a contradictory payload, and the backend honored the firmware downgrade first — stripping the UEFI config Secure Boot needed, then failing with advice the operator had already followed. Fixed in all three layers: the dialog now flips Secure Boot off when BIOS/i440FX is picked, the submit path never sends SB=on with a non-UEFI/non-Q35 combo, and the node task treats secure_boot=on as authoritative (forces UEFI+Q35 server-side).
  • VM Settings: UEFI VMs could show as BIOS in the dialog — firmware detection only recognized an explicit <loader> element, not the <os firmware='efi'> autoselect form, so a converted VM could be misreported as BIOS (and a blind re-Apply of that state would have silently stripped UEFI). Secure Boot detection likewise now recognizes the <firmware><feature name='secure-boot'> autoselect form.

0.42.8.0 — 2026-07-31

Added

  • LVM-thin local storage backend — a new storage_backend=lvm option alongside local/ZFS/Ceph/DRBD/NFS: a raw block device attached directly to the VM (no filesystem layer), thin-provisioned so it supports snapshots. Per-node, not shared — same operational model as ZFS pools. New Storage → Local LVM-Thin tab (create/list/delete pools, reusing the existing physical-disk inventory's usage classification) and a matching option in the VM deploy modal. POST /api/storage/lvm/create mirrors routers/storage/linstor.py::create_drbd_pool's hardened SSH sequence (stale-VG cleanup, force-override, refuses to wipe in-use volumes) minus the LINSTOR-specific registration, since these pools are never DRBD-replicated. v1 scope is VM disks only — no K8s CSI, no replication, single-disk-per-pool (multi-disk VGs are a possible fast-follow).

Fixed

  • Rate limiter lost its Redis connection after the previous release wired requirepass into Redis — core/ratelimit.py's slowapi Limiter had its own hardcoded, unauthenticated redis://redis:6379/2 connection string, separate from CELERY_BROKER_URL, that nobody updated. Every request behind a rate limit (including /api/login) started failing with a 500. RATELIMIT_STORAGE_URI is now set explicitly alongside the broker URLs (same host/auth, so it also follows the CONTROL_PLANE_IP VIP correctly on the 3-host HA stack instead of only ever reaching one replica's local Redis — a pre-existing bug independent of the auth regression). Found via local smoke-testing while building the LVM feature above, before it reached any real deployment.

0.42.5.18-beta — 2026-07-31

Added

  • Ceph & LINSTOR datastore Summary panels (V38.1-C3-adjacent feature-parity gap) — clicking a Ceph pool or LINSTOR/DRBD resource name now opens the same tabbed Summary panel VMs/hosts/NFS datastores already have, listing every VM whose disk actually lives there (matched against stored domain XML, same pattern as the NFS datastore summary). The Storage → Ceph tab also gained a Storage Pools table — previously the tab only had the OSD-add form, with no pool visibility at all.
  • Dedicated Ceph pool per cluster — POST /api/ceph/pools creates a new RBD pool and extends client.libvirt's cephx caps to cover it (previously every Ceph-backed VM fleet-wide shared one hardcoded vms pool); a cluster can now be assigned its own pool (clusters.ceph_pool_name, editable from the cluster Edit modal) and VM deploys onto that cluster resolve to it automatically.
  • Container telemetry in the Performance tab (closes the V38.1-C3 remainder) — LXD containers now appear as a third target kind alongside VMs/hosts, sourced from the existing container-telemetry beat with dedup'd polling; 6h/24h ranges are greyed out for containers pending a rollup.
  • Kubernetes Day-2 control-plane add/remove (closes the V40 "day-2 CP" remainder) — POST/DELETE /api/kubernetes/{id}/control-planes grow an HA cluster to 3/5/7 CPs or safely remove one (etcd-quorum-checked, drain + delete-node), including promoting a legacy single-CP cluster to HA on the fly. Not yet live-tested against a real k3s/rke2 HA cluster — see the task docstrings in tasks/kubernetes.py for the specific risk callout on the k3s SQLite→etcd live migration step.
  • Redis/Celery broker HA (closes the "Highly Available Message Broker" roadmap pillar, Redis-based rather than the originally-scoped RabbitMQ swap) — Redis now runs independently on all 3 control-plane hosts instead of one, following the existing CONTROL_PLANE_IP keepalived/VRRP VIP (active-passive; a failover starts with an empty broker, tasks are re-triggerable from the UI). See deployment/REDIS_HA.md.

Fixed

  • Redis had no authentication. .env.example documented REDIS_PASSWORD as required ("task injection = root on nodes") but nothing actually wired it into docker-compose.yml or CELERY_BROKER_URL — the broker was reachable, unauthenticated, by anything on the LAN. setup.sh now generates it like every other secret; found while implementing the Redis HA change above (deploying to 2 more hosts without fixing this would have tripled the exposure instead of closing it).

0.42.5.17-beta — 2026-07-30

Removed

  • Discord Bot integration, shipped earlier today in 0.42.5.16-beta below. Working end-to-end (Gateway connection live, tool-calling confirmed), but the local Ollama model's ~40s+ per-turn latency made it impractical for Discord's message-passing UX, so it's being pulled rather than left half-usable. Removed core/discord_bot.py, the leader-only background task hook in core/leader.py, the GET/PUT /api/settings/discord-bot endpoints, the Settings → Discord Bot panel, and the discord.py dependency.

0.42.5.16-beta — 2026-07-30

Added

  • Discord Bot — two-way AI Assistant chat (Settings → Discord Bot): @mention the bot in a channel or DM it directly to chat with the same tool-calling assistant the console chat panel offers (VM/K8s/backup diagnostics, VM deploy, power actions, app deployments). Runs as a leader-only singleton background task (core/discord_bot.py) using Discord's Gateway (an outbound-only WebSocket, like the existing webhook alerts) — no public inbound endpoint required, and no synchronous response deadline, unlike the Microsoft Teams equivalent that was evaluated and deferred for exactly those two reasons. Mutating actions (VM deploy, power actions, app deploy) use the same server-enforced propose/confirm safety gate as the console panel — real clickable Approve/Cancel Discord buttons, not a conversational "yes" the model itself could type. No per-Discord-user MFCloud identity exists; every interaction runs under a fixed service (global_admin) identity, so access is controlled by who can reach the bot in Discord, not per-user RBAC.

0.42.5.15-beta — 2026-07-30

Added

  • Microsoft Teams alert webhook, alongside the existing Discord/Slack one — a separate webhook_url_teams setting (Settings → Platform Configuration) since Teams' Workflows webhooks expect a different JSON payload shape ({"text": ...}) than Discord's ({"content": ...}). Both can be configured at once; every existing alert source (CPU/RAM threshold, VM/host anomaly, storage forecast, Ceph health) now fans out to whichever of the two are set, independently.

0.42.5.14-beta — 2026-07-30

Added

  • In-console AI Assistant reaches parity with mfcloud-mcp: the chat panel's tool set grew from 9 to 19 tools, adding VM power actions (get_vm_power_state, propose_vm_power_action/confirm_vm_power_action), backup visibility (list_vm_backups), and full Kubernetes App Catalog deployment (list_kubernetes_clusters, list_app_catalog, list_app_deployments, get_app_deployment_log, propose_app_deployment/confirm_app_deployment) — everything the external mfcloud-mcp MCP server already offered. Every mutating action keeps the same server-enforced propose/confirm safety gate as VM deploy: a real Approve/Cancel card in the UI, not a conversational "yes".

Fixed

  • GET /api/vms/{id}/summary had no tenant isolation — any authenticated user could pull another tenant's full VM summary (power state, guest IPs, disks/NICs) by guessing a VM name. Now scoped the same way every other per-VM route is (admin / owner / same-tenant).

0.42.5.13-beta — 2026-07-29

Added

  • Proxmox VE import (V2V) — a third "From Proxmox (Live Copy)" tab alongside the existing "From Datastore" and other live-copy V2V Import options. Browse a Proxmox host's stopped guests over its REST API (API token auth) and pull one straight in; disk conversion runs over SSH + qemu-img on the source Proxmox host (dir/NFS, LVM/LVM-thin, ZFS, and Ceph RBD-backed disks all supported) and then goes through the same virt-v2v conversion, driver injection, and placement pipeline the other live-copy path already uses. v1 scope: single-VM only — batch import (like the other tab's multi-VM placement table) is a planned fast-follow. ⚠️ Not yet validated against a real Proxmox host — please test carefully before relying on it, and report anything that doesn't match a real Proxmox deployment's storage layout.

0.42.5.12-beta — 2026-07-29

Fixed

  • noVNC console broken after VM migration: a live migration transfers the currently-running domain's XML as-is — if that VM's <graphics> listen address was ever anything other than 0.0.0.0 (e.g. a VM that reached inventory outside the normal deploy flow), it stayed that way on the destination host too, and the console proxy (which always connects to the compute node's real IP, never localhost) could never reach it. Both the live and cold migration paths now force the destination VM's VNC listen address to 0.0.0.0 before/during the move — self-healing the misconfiguration on every migration instead of only at first deploy.

0.42.5.11-beta — 2026-07-29

Fixed

  • Edit Hardware context-menu item was still masking a real 0 vCPU/RAM as "unset" for stopped containers — the 0.42.5.10-beta fix only covered the running-container branch of the context menu. Caught by testing the 0.42.5.10-beta fix batch locally before this went further.

0.42.5.10-beta — 2026-07-29

Fixed

  • OVN/macvlan container-NIC bypass: creating an LXD container with both an OVN IP pool and a macvlan-backed network selected silently routed the container's traffic through macvlan (bypassing the OVN logical switch's ACLs entirely) while the API still reported it as OVN-bound with a real IPAM-allocated IP/MAC. The combination is now rejected with a clear error.
  • Placement: score_hosts() was probing every label-matching host for live capacity before filtering by OS class, wasting a live qemu+ssh round trip on hosts that could never be selected for the VM being placed.
  • LXD NIC management: remove_nic could silently clear an instance's stateful flag and could wrongly reattach the default profile; both add_nic and remove_nic now use LXD's ETag/If-Match support so a concurrent NIC change on the same instance is rejected and retried instead of silently clobbered.
  • V2V debug mode: a cancelled import with V2V_DEBUG=1 set now cleans up its staging directory like every other exit path, instead of leaving a large (up to tens of GB) source copy behind.
  • Hardened silent-failure handling in the SSH-penalty-exempt/VNC-firewall repair helpers and the LXD-evacuation alarm query — both now log instead of quietly degrading.
  • Assorted V2V batch-import and dashboard-widget correctness/cleanup fixes from a full code-review pass (Chart.js instances leaking on repeated tab opens, a modal-form Enter key submitting unedited fields, a container hardware-edit dialog treating a real 0 as unset, redundant /api/alarms polling).

0.42.5.9-beta — 2026-07-29

Changed

  • Host Health honeycomb (Compute Overview widget) now shows a small IP label under each hex cell, so a bad host can be identified at a glance instead of hovering over every hex to read its tooltip. Shows the last two octets (e.g. 14.11) rather than the full IP — the default text-truncation would otherwise clip from the end and hide exactly the octets that distinguish one host from another.

0.42.5.8-beta — 2026-07-28

Fixed

  • Root cause of the intermittent "0 VMs" / "awaiting host data" dashboard blips: compute nodes' OpenSSH PerSourcePenaltyExemptList only contained the floating control-plane VIP, which is never the actual source IP of an outbound connection — each of the 3 console replicas connects from its own real interface IP. So none of the 3 replicas were actually exempt from OpenSSH 9.8+'s brute-force penalty box, and their redundant polling of the same compute nodes periodically tripped it, causing real, intermittent qemu+ssh failures (not a client-side rendering bug — see 0.42.5.7-beta above for a related but separate fix). routers/hosts/repair.py::_ensure_penalty_exempt() (run on every Repair Worker action) only ever wrote the single VIP; its sibling function, _ensure_vnc_firewall_restricted(), already correctly pulled every real HA host IP from ha_cluster_hosts — the two had drifted. Now both agree. Retroactively fixed on all enrolled compute nodes the same day.

0.42.5.7-beta — 2026-07-28

Fixed

  • The Summary tab's VM/host stats and Overview widgets could occasionally drop to "0 VMs" / empty for one poll cycle (e.g. Active Virtual Machines, Top VMs by CPU/Memory %, CPU Usage Share). Cause: refreshAll() treated ANY non-array response from /api/vms (or non-{hosts:[...]} response from /api/hosts) as an empty list — including a transient error body that still parsed as valid JSON, which a request racing a backend restart could plausibly return. Now only a recognizably well-formed response overwrites the cached VM/host list; anything else keeps the last known-good data and logs a console warning instead of silently blanking the dashboard.

0.42.5.6-beta — 2026-07-28

Fixed

  • Renaming the console root label (sidebar tree root) reverted to the container hostname on the next page load. PUT /api/system/console-name persisted CONSOLE_NAME to .env but never updated the already-running process's environment, so os.getenv("CONSOLE_NAME") kept returning nothing until the container was recreated — falling back to socket.gethostname() (a random container ID like 3ed2785f3a3a). Now also sets os.environ["CONSOLE_NAME"] in the same request so the rename takes effect immediately.

0.42.5.5-beta — 2026-07-28

Fixed

  • The Host Availability rings, Host Health honeycomb, and CPU Usage Share pie visibly reset/regrew every ~3 seconds on the Summary tab. Cause: the underlying Chart.js instances were destroyed and recreated on every background poll tick even when the numbers hadn't changed, replaying the entry animation each time. Now only rebuilds when the data actually changed, and the CPU Usage Share pie updates its existing chart in place instead of tearing it down.

0.42.5.4-beta — 2026-07-28

Fixed

  • The new Host Health honeycomb and Host Availability "Up" ring were an overly bright neon green — switched both to the theme's existing darker --success-solid token (already used for solid-fill buttons) instead of the badge/text-tuned --success green. Same swap for the "Top VMs by Memory %" bars.
  • Switching the sidebar scope (selecting a different host/cluster/datacenter) now immediately repaints the Summary tab's overview widgets instead of waiting up to 3 seconds for the next background poll.

0.42.5.3-beta — 2026-07-28

Added

  • The Summary tab now opens on a real monitoring-style overview instead of just the old stat cards and host tables. New widgets: Host Availability (Up/Down/Unreachable/Maintenance donuts), a per-host Health honeycomb (colored by status and CPU/RAM thresholds), an Alert Summary (Critical/Warning/Info, from the same counts the Alerts panel uses), Top VMs by CPU % and by Memory % ranked lists, and a CPU Usage Share pie for the top 6 VMs. Everything is real data already available client-side or one existing endpoint away — no new telemetry collection was added.
  • Datastores now have an Overview tab alongside the existing per-VM Summary view (Storage tab, or the dashboard Datastores view — click a datastore name). Shows capacity, IOPS, throughput, availability history, and disk utilization. VM/host counts are real (drawn from the same domain-XML disk match as the existing Summary tab); capacity, IOPS, throughput, and history are clearly labeled as sample data until per-datastore telemetry sampling exists.

0.42.5.2-beta — 2026-07-28

Added

  • V2V Import now supports migrating multiple VMs from other hypervisor platforms in one go. The live-copy import tab's guest picker now lets you check 2 or more VMs at once. With multiple guests selected, pick an MFCloud placement cluster and the console automatically decides which KVM host each VM lands on — scoring free RAM/CPU headroom the same way "Deploy to cluster" does for new VMs, and spreading the batch across hosts rather than stacking them on one. A per-VM preview (target name, OS type, and assigned host) is shown before anything imports, and any row's assigned host can still be overridden by hand. Single-VM imports, and the file-based (OVA/VMX-on-datastore) import path, are unchanged.

0.42.5.0-beta — 2026-07-27

This release rolls up everything since 0.42.3.7-beta into one build — the patch versions in between (0.42.3.7 through 0.42.3.14) were incremental local milestones that never individually shipped. Their detail stays below.

Added

  • LXD containers now have an Edit Hardware action. Previously CPU/memory limits could only be set at create time — now right-click a container (in the Containers tab or the sidebar tree) and change them live, no restart needed.
  • LXD containers now have a NIC manager. Add or remove network adapters on an existing container — previously only the single NIC set at create time was possible. The network dropdown offers the same choices (LXD- managed networks and macvlan port groups) as Create Container.
  • LXD containers can now attach to a macvlan-backed port group directly through the Create Container UI, not just LXD-managed networks like lxdbr0. New networks.macvlan_parent field lets a bridge-backed port group point at a host-side interface (a VLAN sub-interface, or the plain physical NIC for untagged/bridged attachment) that containers attach to via a macvlan NIC — closing a gap where the only way to get this networking shape was to call LXD's API directly.
  • NFS Datastores now show which VMs are actually on them. Clicking a datastore name in the Storage tab or the dashboard Datastores view opens a summary panel listing every VM with a disk on that share.

Changed

  • Replaced several plain browser pop-up boxes with proper on-screen forms — tenant/cluster resource quotas, cluster storage backend, renaming a datacenter, a cluster's HA failover timing, tagging a host's capabilities, and evicting a Ceph node all got a real form instead of chained prompt() boxes. Destroying a Ceph cluster is now one styled confirmation requiring you to type DESTROY, instead of two stacked pop-ups.
  • Combined the Datastores toolbar's three storage buttons into one — "Manage NFS," "Manage ZFS," and "LINSTOR / DRBD" opened the same window on different tabs, so now it's a single "Manage Storage" button.
  • Fixed a button getting cut off in the Datastore Manager's Enterprise Pools list — "Fix Permissions" and "Unmount" now live behind a single "⋯" menu / right-click, matching hosts and ZFS pools elsewhere.
  • Fixed hard-to-read status badges and action buttons across Storage — LINSTOR/DRBD replication status, and the bright-green primary buttons ("Create Policy," "Format & Create Array," "Deploy Replicated Volume," etc.) all used low-contrast colors in dark mode; both now use the same soft-tinted / darker-green style as the rest of the console.
  • Light theme cleanup — telemetry chart colors, capacity-bar tracks, RAM/disk mini-cards, floating-panel shadows, and sidebar/tab/task-drawer hover highlighting all had leftover dark-theme styling that did nothing (or looked broken) in light mode; all now follow the current theme.

Fixed

  • Right-clicking a container in the sidebar tree did nothing (the browser's own context menu appeared instead) — the tree's context-menu dispatcher had no branch for container leaves. Right-click now shows the same menu as the Containers tab table (Console, Stats, Snapshots, Edit Hardware, NICs, Migrate, Power, Delete).
  • K8s Scale Workers could fail with a raw qemu-img error when a requested worker disk_gb was smaller than the chosen template's actual virtual size — now checks the template's real size first and raises a clear error naming the minimum disk_gb required, for every create_vm call, not just K8s workers.
  • K8s cluster node table had no visibility into DB/live drift — nodes the live cluster reports that MFCloud never provisioned, or nodes MFCloud believes exist that the live cluster no longer reports, are now flagged ("⚠ untracked" / a synthetic "Missing" row) with a way to clear stale bookkeeping.
  • LINSTOR CSI cluster delete left every node behind as an OFFLINE satellite — cluster delete now best-effort deregisters each node's LINSTOR satellite before dropping the DB row, instead of requiring manual cleanup afterward.

0.42.3.14-beta — 2026-07-27

Fixed

  • Light theme cleanup — several places still used colors leftover from the dark theme design that were never updated. Telemetry charts (Performance tab, VM/Container/Ceph monitoring) had washed-out axis labels and a nearly invisible "current value" readout on light backgrounds — chart colors now follow the current theme. CPU/RAM/disk capacity bars and modal progress bars had an invisible empty track. The RAM/disk telemetry mini-cards used the old bright green/yellow instead of the readable darker versions. A few floating panels (search results, task log, AI chat) had a much heavier drop shadow than everything else. Hover highlighting on the sidebar, inventory tabs, and task drawer silently did nothing in light mode since it was a "lighten a dark background" effect with nothing to lighten.

0.42.3.13-beta — 2026-07-27

Changed

  • Fixed the same hard-to-read green buttons everywhere in Storage. "Create Policy," "Format & Create Array" (ZFS), "Format & Add to Ceph Cluster," "Deploy Replicated Volume," "Extend Existing Pool," and "Finish Setup" (DRBD) all used the same overly bright, low-contrast green with white text as the LINSTOR/DRBD status badges fixed earlier — now they use a darker green that's easy to read in dark mode while still standing out as the primary action button.

0.42.3.12-beta — 2026-07-27

Changed

  • Combined the Datastores toolbar's three storage buttons into one. "Manage NFS," "Manage ZFS," and "LINSTOR / DRBD" all opened the exact same Storage window, just landing on a different tab — now it's a single "Manage Storage" button, and you pick NFS/ZFS/DRBD from the tabs inside as before. The separate Ceph HCI Cluster button stays on its own, since it opens a different, more specialized screen for managing the whole Ceph cluster.

0.42.3.11-beta — 2026-07-27

Added

  • NFS Datastores now show which VMs are actually on them. Clicking a datastore name in the Storage tab or the dashboard Datastores view opens a summary panel listing every VM with a disk on that share (VM, host, disk path) — previously the only way to check was to hunt through each VM's settings individually.

0.42.3.10-beta — 2026-07-27

Changed

  • Fixed a button getting cut off in the Datastore Manager's Enterprise Pools list. The row for each storage pool had two buttons ("Fix Permissions" and "Unmount") that no longer fit in the window and were getting clipped off the edge. Both actions now live behind a single "⋯" menu button (right-click the row also works) — same style used for hosts and ZFS pools elsewhere in the console.

0.42.3.9-beta — 2026-07-26

Changed

  • Fixed hard-to-read status labels on the LINSTOR / DRBD replication tables (node connection status, storage pool health, and the "Replication State" column on the Datastores dashboard). These used solid bright green/yellow badges with white text, which is very low contrast in dark mode — now they use the same soft-tinted badge style used everywhere else in the console.

0.42.3.8-beta — 2026-07-26

Changed

  • Replaced several plain browser pop-up boxes with proper on-screen forms. Setting a tenant's or cluster's resource quota no longer chains 2-3 separate gray text-entry pop-ups — it's one clean form now. Choosing a cluster's storage backend is now a dropdown of valid choices instead of a box where a typo would just get rejected. Renaming a datacenter, setting a cluster's HA failover timing, tagging a host's capabilities, and evicting a node from a Ceph cluster all got the same treatment.
  • Destroying a Ceph cluster is safer to confirm. This used to be two stacked pop-ups (a yes/no, then "type DESTROY"). It's now a single styled confirmation box, and the Destroy button stays disabled until you actually type DESTROY into it.

0.42.3.7-beta — 2026-07-25

Fixed

  • K8s Scale Workers could fail with a raw qemu-img error when a requested worker disk_gb was smaller than the chosen template's actual virtual size — qemu-img resize refuses to shrink a disk without --shrink (which we never want to pass; truncating a live filesystem corrupts the guest), so the operator saw an opaque subprocess exit-code failure instead of an actionable message. tasks/vm/storage.py now checks the template's real size with qemu-img info before attempting the resize (zfs/nfs/local backends — the ceph backend already handled this via its own grow-only comparison) and raises a clear error naming the minimum disk_gb required. Applies to every create_vm call, not just K8s workers.
  • K8s cluster node table had no visibility into DB/live drift — MFCloud had no way to flag a node the live cluster reports that MFCloud never provisioned (manual kubeadm join, or a scale-up whose DB commit never landed), or the opposite: a worker MFCloud believes it added that the live cluster no longer reports (crashed VM, failed join never rolled back). GET /{cluster_id}/nodes now cross-references worker_ips/ control_plane_ip against the live kubectl get nodes snapshot and annotates each row — the Nodes modal (static/js/kubernetes.js) shows an "⚠ untracked" badge for the former and a synthetic "Missing" row for the latter, with a "Clear" action to drop the stale bookkeeping entry. Additive only — never touches k8s_clusters state itself.
  • LINSTOR CSI cluster delete left every node behind as an OFFLINE satellite — a full cluster delete tore down the k8s engine on each node but never told the LINSTOR controller, so LINSTOR-backed clusters had to be cleaned up by hand (Storage → LINSTOR → node delete) after every teardown. DELETE /{cluster_id} now best-effort deregisters each node's satellite (LinstorClient.delete_node) before dropping the DB row — tolerant of an already-dead cluster or missing LINSTOR config, same as the existing per-node uninstall loop.

0.42.3.6-beta — 2026-07-25

Fixed

  • VM consoles fleet-wide could fail to connect ("Connection error") — every VM's XML correctly asks libvirt to bind VNC to all interfaces, but libvirt only honors that at define-time if the node's own /etc/libvirt/qemu.conf agrees; on affected nodes it silently fell back to loopback-only, so the console proxy could never reach it no matter what the VNC firewall rule allowed through. Node enrollment (services/node_provisioner.py) and the fast Repair Worker path (routers/hosts/repair.py) now both assert qemu.conf's vnc_listen = "0.0.0.0" and restart libvirtd, mirroring the existing VNC-firewall-rule re-assertion pattern — every node, fresh or repaired, gets this from now on. Doesn't retroactively fix already-running VMs on already-affected nodes (a VM's QEMU process keeps whatever address it bound to at its own start time) — those need a restart or a Repair Worker run against the node followed by a VM restart.

0.42.3.5-beta — 2026-07-24

Fixed

  • Sidebar inventory tree (VMs/Hosts/Storage/Networks folders) reset to fully expanded on every browser reload — the tree's collapse/expand state was only ever carried across the 3s polling rebuild in memory, never persisted, so a manually-collapsed folder snapped back open the moment the page was refreshed. Collapse/expand choices are now saved to localStorage (static/js/main.js) keyed by tree node id and re-applied both on the initial page load and on every subsequent tree rebuild — a folder now stays collapsed across reloads until explicitly re-expanded.

0.42.3.4-beta — 2026-07-23

Fixed

  • "Enter Maintenance" could silently leave VMs stuck, and permanently lock a compute node's row for every future admin action against it — the endpoint held one open DB transaction for the entire request, including a live SSH capacity probe of every other host in the cluster (core/placement.score_hosts) that can run for several seconds against a slow or unreachable node. If the client/proxy gave up waiting mid-probe, the cancelled request left that transaction open, and the connection leaked back to the pool "idle in transaction" — blocking any later Maintenance/ Activate/Reactivate call for that same host indefinitely, with no error surfaced anywhere until someone found and killed the stuck DB session. The endpoint now uses short, separately-committed connections instead of one request-long transaction, so nothing is held open across the slow probe — a cancelled request just cancels, it doesn't wedge the row.
  • Postgres now also enforces a 2-minute idle_in_transaction_session_timeout fleet-wide (previously 0/disabled) as a backstop for this class of leak in general, not just this one endpoint.

0.42.3.3-beta — 2026-07-23

Added

  • "Migrate VMs Back" failback action — a node returning from an HA fence no longer leaves its VMs stranded on their emergency host forever. Each failed-over VM now remembers the host it fenced away from (vms.ha_failed_over_from); once that host is healthy again, its right-click menu shows "↩ Migrate VMs Back (N)" so an operator can live-migrate them home in one click — offered, never automatic, since the operator is best placed to judge whether a just-returned node is actually ready for load. The flag clears on any explicit migration, so manually placing a VM elsewhere doesn't keep nagging to send it back to a home you moved it away from on purpose.

Fixed

  • HA fence could fail with a spurious "No module named 'services'" on a node's first-ever fence event — core/fencing.py's Ceph blocklist fence re-imported services.ceph_tasks on every call instead of once; the import is now resolved eagerly at module load and cached, removing that class of failure. A node with no BMC/iDRAC configured (nested/lab hosts) also no longer attempts and logs a power fence that was never going to work — it now checks BMC availability up front and goes straight to the Ceph storage fence.

0.42.3.2-beta — 2026-07-22

Fixed

  • Storage migration to/from Ceph could hang indefinitely — the disk-convert helper behind Move Disk read qemu-img convert's progress from stdout only after its stderr had been fully drained, a deadlock risk any time the child writes enough to stderr to fill the pipe buffer before finishing. Local/ZFS/NFS moves rarely write to stderr so this stayed hidden, but the Ceph (RBD) path routes through librbd/librados, which is chatty enough to trigger it — a migration to or from Ceph HA storage could sit stuck at "Running" forever instead of completing or failing. stderr is now drained concurrently on a background thread so it can never block the progress read.
  • ISO/large file upload showed no progress — the file browser's Upload button (Datastores → Browse) sent the file via fetch(), which gives no upload-progress signal; a multi-GB ISO would sit behind a static "Uploading…" toast for however long the transfer took, indistinguishable from a hang. Now uses XMLHttpRequest with upload.onprogress to drive a live percentage bar and transfer rate, and turns red with the actual error detail (e.g. a Caddy 413) instead of a bare "Upload failed" on failure.
  • Large ISO uploads rejected outright — Caddy's shared request_body cap was 2GB, applied to every route behind it including the upload endpoint, so anything past 2GB never reached the app. Raised to 20GB.

0.42.3-beta — 2026-07-21

Added

  • Resizable inventory sidebar — drag the right edge of the datacenter tree (the Hosts / VMs / Storage / Networks navigator on the left) to widen it. Previously fixed at 280px, so VM and host names with an IP suffix (e.g. MF-VDI-04 (192.168.x.x)) were clipped by the tree's ellipsis with no way to see the full name. Width persists across sessions (200–640px range).

Fixed

  • Cluster/folder row action icons stranded far from the name — the ⓘ / ✎ / 📁+ / 🗑 buttons on cluster and folder rows used margin-left:auto (and the row label used flex:1 1 auto), both of which pin trailing content to the row's far right edge. That was invisible at the old fixed 280px width but left a large empty gap after short names once the sidebar could be widened. Icons now sit immediately after the name.

0.42.2-beta — 2026-07-21

Image hardening pass — no application code changes, console behavior is identical to 0.42.0-beta. Ships from a new public registry (see below).

Security

  • Shipped image CVEs cleared (21 → 0) — the published mfconsole/mfconsole Docker image carried 8 HIGH / 10 MEDIUM / 3 LOW vulnerabilities in OS-level Python packages (cryptography, urllib3, setuptools, jwcrypto, requests, idna) pulled in transitively by the base image's package manager — never imported by the running app, but visible to any image scanner (Docker Scout, Trivy, etc.) now that the repo is public. Dockerfile.ship's runtime stage now patches these in place and clears the stale package metadata so scans come back clean.

Changed

  • Distribution moved to a public Docker Hub org — mfconsole/mfconsole replaces the prior self-hosted, token-gated registry.mfcloud.io. No docker login is required to pull. See mfconsole-ship/README.md for the access-control tradeoff this implies.
  • One-command install — the deploy-files bundle now includes install.sh, which wraps .env setup, setup.sh, image pull, and docker compose up into a single step.

Fixed

  • setup.sh never generated DB_PASS — it shipped as the literal placeholder string instead of a random value; now generated like the other secrets.
  • DEFAULT_ADMIN_PASS placeholder mismatch — following the documented install steps left the first-login password as the raw placeholder string instead of the documented mfpro default.

0.42.1-beta — 2026-07-21

Added

  • Editions / licensing system — MFCloud now ships six editions: Free (default, 2 nodes / 10 VMs, no paid features), Homelab ($14.99/mo, 3 nodes / 100 VMs, personal/non-commercial use), Starter ($29/mo, 5 nodes, unlimited VMs, adds S3 backup), Growth ($79/mo, 10 nodes, unlimited VMs, same feature set as Starter — bridges the old Starter→Business price cliff), and Beta/Pro (unlimited nodes and VMs, every feature flag on: HA, S3 backup, multi-tenancy, LDAP, AIOps, Commander, Kubernetes). License keys are Ed25519-signed (MFC2.<payload>.<sig> — the signing private key never ships; only the public key is embedded, so a customer with full source access still cannot forge a key) with the original HMAC-signed MFC1 format still accepted for compatibility with any beta test keys already handed out. A lapsed valid_until downgrades a deployment to Free-tier caps at read time without touching the stored license row or any existing VM/node — enforcement only blocks new node/VM creation and gated-feature access (core/license.py, routers/system/license.py, Settings → License). Node/VM limits and feature gates (assert_can_add_node, assert_can_create_vm, assert_feature) are enforced at every relevant create path across hosts, VMs, HA, S3 backup, multi-tenancy, LDAP, AIOps, Commander, and Kubernetes. An optional, fire-and-forget activation ping (LICENSE_ACTIVATION_URL, off by default) lets the vendor flag a paid key later reused on a second deployment for manual follow-up — it never blocks or remotely disables a console; a valid key is honored fully offline regardless.

0.42.0-beta — 2026-07-17

V40 (Managed Kubernetes Engine) completion pass. The HA and LINSTOR features are code-complete but not yet live-tested — see KNOWN_ISSUES.md → Kubernetes for the pending checklist.

Added

  • YAML manifest deployments — each cluster row has a new YAML button: upload (or paste) a raw Kubernetes YAML file — multi-document supported — and the console runs kubectl apply on the cluster's control-plane. Manifests are listed in their own table with kubectl's own created/configured output as the status detail; re-uploading under the same manifest name updates it in place (declarative apply), and Delete replays the stored YAML through kubectl delete -f. Uploads are pre-parsed server-side so a YAML typo fails immediately, not minutes into the apply task.
  • Per-deployment logs — App Catalog deployments and YAML manifests each have a Log button showing the full deployment trail (Queued → Deploying → Waiting for pods → Success/Failed, including error details), on top of the live Recent Tasks stream that already existed.
  • MCP: deploy apps from any MCP client — the mfcloud-mcp server gains six tools: list_kubernetes_clusters, list_app_catalog, list_app_deployments, get_app_deployment_log, and the confirmation-gated propose_app_deployment / confirm_app_deployment pair (same server-enforced propose→review→confirm flow as VM deployment; single-use 5-minute tokens bound to the engineer's own API key).
  • HA Kubernetes control plane — the deploy wizard now offers a 3-node control plane (embedded etcd quorum) behind a kube-vip virtual IP, chosen at create time. Workers and the downloaded kubeconfig target the VIP, so losing any single control-plane node no longer takes down the API. The VIP is leased from a normal IPAM pool (visible in the leases table, released on cluster delete) and must come from a different pool than MetalLB's. Existing single-CP clusters and the 1-CP path are unchanged. Day-2 operations (node list, worker scaling, storage/networking retries, App Catalog) are now control-plane-failover aware on all clusters.
  • LINSTOR CSI storage — clusters can attach to the platform's existing external LINSTOR controller: every K8s node becomes a diskless satellite, data replicas stay on the compute-node satellites via a dedicated resource group, and a mfcloud-linstor StorageClass (expansion enabled, WaitForFirstConsumer) is created. Joins the existing Ceph CSI option.
  • Per-tenant Kubernetes cluster quota — max_k8s_clusters ceiling (0 = unlimited) enforced at cluster create, settable from the IAM Resource Quotas widget alongside vCPU/RAM.
  • Real TLS certificates (internal PKI) — new TLS_MODE in .env, applied by setup.sh (regenerates the Caddy TLS config in tls.d/): internal (self-signed, unchanged default), byo (operator cert from AD CS or any internal CA, dropped into ./tls/; optional IP-SAN coverage for the by-IP site via TLS_BYO_COVERS_IP), and acme-internal (auto-issue and renew from an internal ACME CA such as step-ca). Trusting the built-in Caddy root CA (export + GPO) is documented as the zero-config path. setup.sh auto-adds https://<TLS_DOMAIN> to ALLOWED_ORIGINS.

Changed

  • OVN-Kubernetes CNI formally dropped from the roadmap — upstream owns its own in-cluster OVN stack and cannot attach to MFCloud's external OVN fabric; Flannel/Canal + MetalLB remain the supported path.

Fixed

  • K8s worker auto-naming no longer collides after a scale-down — the next -wN suffix is now derived from the highest existing suffix instead of the worker-list length, so removing w2 and then adding a worker no longer re-mints w3 and fails with "VM name already taken".
  • Device-mode ZFS clusters get the right scale-up pre-flight — adding a worker to a zfs_pool_source='device' cluster no longer applies the file-mode "disk_gb ≥ pool file + 10 GB" guard (the request-model default of 50 GB was being persisted even for device clusters), and the API now enforces what device mode actually needs: a csi_zfs_pool_name/size for every auto-provisioned worker (previously only the UI checked this).
  • MetalLB pool collision checks now also run at cluster create — the create wizard's MetalLB pool is rejected if it is any cluster's API-VIP pool or already another cluster's MetalLB pool (two MetalLB instances on one range assign duplicate Service IPs); previously only the day-2 Configure Networking path checked, and pool sharing wasn't blocked at all.

0.41.6-beta

Added

  • AI Assistant panel: resize + per-answer stats — the chat panel can now be resized from its top-left corner (size persists, like its position), and each answer shows a stats line: a live "Thinking · Ns" timer while the model works, then total response time and token usage (in/out) reported by the backend (Claude API usage, or Ollama's eval counts).

Fixed

  • AI Assistant input no longer locks up after a backend error — an error mid-turn (e.g. an Ollama timeout) ended the stream without the completion envelope, leaving the message box disabled until the panel was reopened.

0.41.0-beta — current

Upgrading an existing deployment: the compose network is now pinned to a fixed subnet (172.30.238.0/24). Docker Compose cannot change a network in-place — run docker compose down once before up -d with this version.

Added

  • Kubernetes worker scaling — each cluster row now has a Nodes modal showing a live node snapshot (name, IP, role, Ready state), with Scale Workers to join additional workers onto a healthy cluster and drain-and-remove a single worker. Progress is reported through Recent Tasks.
  • App Catalog — deploy curated application stacks (Ollama + Open WebUI, Jupyter + PyTorch, ComfyUI, vLLM) onto an MFCloud-managed Kubernetes cluster from the new App Catalog in the Kubernetes tab. Apps are exposed via a MetalLB LoadBalancer when available, NodePort otherwise. All stacks are CPU-only in this release (GPU support arrives with GPUaaS).
  • Ask AI from search — the Ctrl+K search box now offers an "Ask AI" row on any query, handing it to the console's AI chat panel — useful for questions ("why is this VM slow?") that no inventory name matches.

Changed

  • License gating extended — LDAP, AIOps, and the AI assistant are now edition-gated alongside HA, S3 backup, and multi-tenancy. Existing keys for editions that include a newly gated feature keep it automatically — no key re-issue needed.

Security

  • Hardening pass (2026-07-13) — 17 fixes from a full-codebase review, including: a tenant-isolation bypass on VM power actions and a cross-tenant VM telemetry route; a cloud-init SSH-key injection; Ceph dashboard and LDAP/OIDC bind secrets now encrypted at rest (AES-256-GCM); the worker→master internal callback now pins the master's TLS certificate fingerprint instead of skipping verification; a race in SSH host-key first-pinning; a narrowed trusted-proxy range on the login rate limiter (with the compose-network subnet pin above); RBAC initialization failures now abort startup instead of serving with half-seeded permissions; and database-failover handling that discards connections to a demoted primary.

Fixed

  • Dashboard could hang when a compute node went offline — libvirt connections to unreachable nodes are now bounded, so the VM list stays responsive while a node is down.
  • App Catalog: Jupyter + PyTorch deploys crash-looped — two environment collisions (a template-variable clash and Kubernetes' auto-injected service port variable overriding the image's own port setting) are both fixed.
  • Terraform destroy returning a 500, a route-shadowing bug that hid the pending-node-approval UI, and a virsh regression on localhost-managed hosts.
  • Shipped console images now include the Emergency Host UI overlay (agent_overlay/), which was silently missing from registry builds.

0.40.5-beta

Covers 0.40.1–0.40.5; combined here since they shipped in quick succession.

Added

  • Usage Reports — new admin Reports tab: Compute and Network usage over a chosen time range, plus a point-in-time Storage snapshot, each exportable to CSV.
  • Host anomaly detection — the telemetry pipeline now flags hosts whose CPU/RAM/disk/network behavior breaks from their own recent baseline.

Changed

  • Datacenter-console visual reskin — new masthead color tokens (navy header in both light and dark themes), flatter corner radii, uppercase button labels, underlined tabs, and reduced glow/shadow throughout; fixed a light-mode readability issue in the inventory tree.
  • Telemetry pipeline — the background sampler now runs its per-host collection in parallel (fixes an intermittent stall), and the rollup used by Reports/telemetry charts was widened to cover disk and network in addition to CPU/RAM, making the Reports tab noticeably faster to load.

Fixed

  • Network security groups were not being enforced at all — ACL pushes were silently targeting a logical switch that didn't exist, and the failure was swallowed. Security groups now resolve the VM's real switch and correctly union rules across multiple assigned groups.
  • IPAM: a lease-allocation race, a reserved-IP bypass, a leftover logical switch port on delete, and a pool-naming conflict were all fixed.
  • Storage: an orphaned LINSTOR resource-definition case, a datastore name-conflict split-brain, several leaked SSH connections, and missing guards around iostat output and node-count edge cases.

0.40.0-beta

Added

  • ZFS pool replication + VM restore (DR) — schedule async, snapshot-based replication of a ZFS pool to another host in the same cluster, with a live progress bar (when pv is installed) and a one-click Restore to rebuild and boot the replicated VMs on the target if the source goes down. See the new Knowledge Base guide: ZFS Pool Replication & VM Restore (DR).
  • Windows Secure Boot + vTPM 2.0 across all three guest paths — fresh deploy, V2V import, and the VM Settings firmware toggle. Fresh deploys enroll Microsoft's keys automatically for a Windows guest on UEFI; V2V import and the Settings toggle preserve the guest's existing UEFI variable store (so boot entries and any already-enrolled keys survive).
  • New Knowledge Base guide: Install a Windows VM (Server & 11) — VirtIO driver loading, firmware/Secure Boot settings, and converting to a reusable template.
  • What's new in this release — the console now shows a short release note the first time you log in after an update, with a bell icon in the header to reopen it any time. Sourced from this same changelog.

Changed

  • The Deploy VM dialog and VM Settings → Boot Options now expose a Secure Boot control (Auto / On / Off) alongside Machine Type and Firmware.

V39.17-alpha — public-alpha hardening

The V39.13 → V39.17 band is the public-alpha hardening track. See SECURITY_AUDIT.md and KNOWN_ISSUES.md for the security posture and the short list of accepted limitations.

Added

  • Compact density mode — console-wide density toggle in the top nav (Comfortable ↔ Compact), persisted per-browser and applied flash-free, driven by a single density-token scale so every table/card/widget re-paces at once.
  • Sticky table headers — column headers stay pinned while long VM / host / task lists scroll; sharper row-hover with an accent edge.
  • Console identity — the inventory-tree root now shows the server hostname (or CONSOLE_NAME from .env) so engineers managing several consoles can tell them apart at a glance.

Security (hardening pass — see SECURITY_AUDIT.md "Audit Pass 2")

  • Secrets hygiene: .gitignore + .env.example; setup.sh bootstraps .env and generates DB_PASS / DEFAULT_ADMIN_PASS / REDIS_PASSWORD.
  • Redis broker requirepass + provisioner credential-preserving URL rewrite.
  • Forced first-login password change (must_change_password + blocking overlay).
  • Closed cloud-init YAML injection, config-export secret leak, OIDC token-in-URL, cross-tenant VM access, unauthenticated backup/snapshot routes, webhook SSRF, and hardcoded admin elevation. Weak INTERNAL_SECRET now fails closed.

V39 — Distributed clustering, DR, and backup UX

  • V39 / V39.0.2 — Distributed container clustering: native LXD/Incus dqlite quorum, replicated container storage (LINSTOR/DRBD), free-RAM scheduler, live migration, and HA auto-evacuation of workloads off failed members.
  • V39.6 — Console navigation & UX pass: neutral product naming pass; inventory and recent-tasks polish.
  • V39.7 — PCIe passthrough in VM Settings; UEFI-VM lifecycle fixes.
  • V39.8 — Agentless S3 Disaster Recovery: dirty-bitmap incrementals, ZSTD streaming, AES-256-GCM direct-to-cloud upload, one-click cloud restore.
  • V39.9–V39.12 — Backup UX: cluster-scoped NFS datastores, destination picker, scheduled backups v2, capacity bars, and live task progress.

V38 — Containers, marketplace, and identity federation

  • V38 — LXD / Incus container support: lifecycle, interactive console (xterm.js), live metrics, snapshots/restore, remote image import, and OVN logical-switch binding.
  • V38.2 — VM table right-click context menu.
  • V38.3 — Agent callback restoration & VM inventory discovery.
  • V38.4 — Turnkey marketplace & dedicated Catalog tab.
  • V38.5 — Trust pinning, recent-tasks polish, NFS hardening, light/dark theme.
  • V38.7 — Identity federation (OIDC/SSO): "Sign in with Microsoft" (Entra ID, Google, Okta), auto-provisioning, group-claim → role mapping, RP-initiated logout, shorter OIDC token TTL.

V37 — Database high availability

  • V37 — Patroni + etcd + HAProxy: TimescaleDB in HA; writes routed to the current primary; sync (psycopg2) and async (asyncpg) pools.

V31–V36 — Platform foundation

  • V31.6 — Enterprise identity: LDAP / Active Directory authentication.
  • V31.7 / V31.8 — Performance tab redesign and inventory tree.
  • V31.10 — SDN: OVN logical networks + security groups.
  • V31.11 — LINSTOR REST API rewrite + DRBD cluster wizard.
  • V31.12 / V36 — Hypervisor ISO + imaged-node fast path: zero-touch node provisioning and enrollment.
  • V31.13 — Security audit pass; Caddy TLS termination; password-only SSH path.
  • V32 — Agent Trust API: TOFU host-key pinning, agent credential delivery.
  • V33 — Releases incl. V33.2 iDRAC / BMC hardware health monitoring.
  • V34 — Cloud-Init guest customization.
  • V36.1 — Knowledge base bootstrap (served at /kb).

V23–V30 — Initial platform

  • Core VM lifecycle (create/start/stop/delete/edit), snapshots & backups with retention, clone + template conversion, live/cold migration.
  • VNC console proxy (noVNC) and web SSH terminal over WebSocket.
  • Host auto-bootstrap, maintenance + VM evacuation, multi-host telemetry.
  • OVS VLAN portgroups, KVM NAT networks, NFS datastores, image management.
  • Ceph cluster bootstrap + OSD management; ZFS pool management.
  • JWT + API-key auth, RBAC, per-user vCPU/RAM quotas, webhook alerts, task history, real-time task streaming.