Skip to content

Changelog

The same release notes shown in the console's What's New prompt (bell icon, top nav), in full and in order.

MFCloud Console — Changelog

All notable changes to the MFCloud console, curated by release. Dates are the completion date of each version. This is a distilled highlight list — the full per-item engineering history lives in patch_note/RAOD_MAP_SHIPPED.txt.

Format loosely follows Keep a Changelog. Versions are MFCloud product versions (the APP_VERSION baked into each build).

Versioning scheme (from 0.40.0-beta on): semver, pre-1.0 — 0.<minor>.<patch>-<stage>. The 0.x major signals the product hasn't hit a stable 1.0 yet; <minor> increments per release the way the old V<n> counter did. Older headers below (V39, V38, …) predate this and are left as originally written.

0.42.5.13-beta — 2026-07-29

Added

  • Proxmox VE import (V2V) — a third "From Proxmox (Live Copy)" tab alongside the existing "From Datastore" and "From vCenter" V2V Import options. Browse a Proxmox host's stopped guests over its REST API (API token auth) and pull one straight in; disk conversion runs over SSH + qemu-img on the source Proxmox host (dir/NFS, LVM/LVM-thin, ZFS, and Ceph RBD-backed disks all supported) and then goes through the same virt-v2v conversion, driver injection, and placement pipeline the vCenter path already uses. v1 scope: single-VM only — batch import (like the vCenter tab's multi-VM placement table) is a planned fast-follow. ⚠️ Not yet validated against a real Proxmox host — please test carefully before relying on it, and report anything that doesn't match a real Proxmox deployment's storage layout.

0.42.5.12-beta — 2026-07-29

Fixed

  • noVNC console broken after VM migration: a live migration transfers the currently-running domain's XML as-is — if that VM's <graphics> listen address was ever anything other than 0.0.0.0 (e.g. a VM that reached inventory outside the normal deploy flow), it stayed that way on the destination host too, and the console proxy (which always connects to the compute node's real IP, never localhost) could never reach it. Both the live and cold migration paths now force the destination VM's VNC listen address to 0.0.0.0 before/during the move — self-healing the misconfiguration on every migration instead of only at first deploy.

0.42.5.11-beta — 2026-07-29

Fixed

  • Edit Hardware context-menu item was still masking a real 0 vCPU/RAM as "unset" for stopped containers — the 0.42.5.10-beta fix only covered the running-container branch of the context menu. Caught by testing the 0.42.5.10-beta fix batch locally before this went further.

0.42.5.10-beta — 2026-07-29

Fixed

  • OVN/macvlan container-NIC bypass: creating an LXD container with both an OVN IP pool and a macvlan-backed network selected silently routed the container's traffic through macvlan (bypassing the OVN logical switch's ACLs entirely) while the API still reported it as OVN-bound with a real IPAM-allocated IP/MAC. The combination is now rejected with a clear error.
  • Placement: score_hosts() was probing every label-matching host for live capacity before filtering by OS class, wasting a live qemu+ssh round trip on hosts that could never be selected for the VM being placed.
  • LXD NIC management: remove_nic could silently clear an instance's stateful flag and could wrongly reattach the default profile; both add_nic and remove_nic now use LXD's ETag/If-Match support so a concurrent NIC change on the same instance is rejected and retried instead of silently clobbered.
  • V2V debug mode: a cancelled import with V2V_DEBUG=1 set now cleans up its staging directory like every other exit path, instead of leaving a large (up to tens of GB) source copy behind.
  • Hardened silent-failure handling in the SSH-penalty-exempt/VNC-firewall repair helpers and the LXD-evacuation alarm query — both now log instead of quietly degrading.
  • Assorted V2V batch-import and dashboard-widget correctness/cleanup fixes from a full code-review pass (Chart.js instances leaking on repeated tab opens, a modal-form Enter key submitting unedited fields, a container hardware-edit dialog treating a real 0 as unset, redundant /api/alarms polling).

0.42.5.9-beta — 2026-07-29

Changed

  • Host Health honeycomb (Compute Overview widget) now shows a small IP label under each hex cell, so a bad host can be identified at a glance instead of hovering over every hex to read its tooltip. Shows the last two octets (e.g. 14.11) rather than the full IP — the default text-truncation would otherwise clip from the end and hide exactly the octets that distinguish one host from another.

0.42.5.8-beta — 2026-07-28

Fixed

  • Root cause of the intermittent "0 VMs" / "awaiting host data" dashboard blips: compute nodes' OpenSSH PerSourcePenaltyExemptList only contained the floating control-plane VIP, which is never the actual source IP of an outbound connection — each of the 3 console replicas connects from its own real interface IP. So none of the 3 replicas were actually exempt from OpenSSH 9.8+'s brute-force penalty box, and their redundant polling of the same compute nodes periodically tripped it, causing real, intermittent qemu+ssh failures (not a client-side rendering bug — see 0.42.5.7-beta above for a related but separate fix). routers/hosts/repair.py::_ensure_penalty_exempt() (run on every Repair Worker action) only ever wrote the single VIP; its sibling function, _ensure_vnc_firewall_restricted(), already correctly pulled every real HA host IP from ha_cluster_hosts — the two had drifted. Now both agree. Retroactively fixed on all enrolled compute nodes the same day.

0.42.5.7-beta — 2026-07-28

Fixed

  • The Summary tab's VM/host stats and Overview widgets could occasionally drop to "0 VMs" / empty for one poll cycle (e.g. Active Virtual Machines, Top VMs by CPU/Memory %, CPU Usage Share). Cause: refreshAll() treated ANY non-array response from /api/vms (or non-{hosts:[...]} response from /api/hosts) as an empty list — including a transient error body that still parsed as valid JSON, which a request racing a backend restart could plausibly return. Now only a recognizably well-formed response overwrites the cached VM/host list; anything else keeps the last known-good data and logs a console warning instead of silently blanking the dashboard.

0.42.5.6-beta — 2026-07-28

Fixed

  • Renaming the console root label (sidebar tree root) reverted to the container hostname on the next page load. PUT /api/system/console-name persisted CONSOLE_NAME to .env but never updated the already-running process's environment, so os.getenv("CONSOLE_NAME") kept returning nothing until the container was recreated — falling back to socket.gethostname() (a random container ID like 3ed2785f3a3a). Now also sets os.environ["CONSOLE_NAME"] in the same request so the rename takes effect immediately.

0.42.5.5-beta — 2026-07-28

Fixed

  • The Host Availability rings, Host Health honeycomb, and CPU Usage Share pie visibly reset/regrew every ~3 seconds on the Summary tab. Cause: the underlying Chart.js instances were destroyed and recreated on every background poll tick even when the numbers hadn't changed, replaying the entry animation each time. Now only rebuilds when the data actually changed, and the CPU Usage Share pie updates its existing chart in place instead of tearing it down.

0.42.5.4-beta — 2026-07-28

Fixed

  • The new Host Health honeycomb and Host Availability "Up" ring were an overly bright neon green — switched both to the theme's existing darker --success-solid token (already used for solid-fill buttons) instead of the badge/text-tuned --success green. Same swap for the "Top VMs by Memory %" bars.
  • Switching the sidebar scope (selecting a different host/cluster/datacenter) now immediately repaints the Summary tab's overview widgets instead of waiting up to 3 seconds for the next background poll.

0.42.5.3-beta — 2026-07-28

Added

  • The Summary tab now opens on a real monitoring-style overview instead of just the old stat cards and host tables. New widgets: Host Availability (Up/Down/Unreachable/Maintenance donuts), a per-host Health honeycomb (colored by status and CPU/RAM thresholds), an Alert Summary (Critical/Warning/Info, from the same counts the Alerts panel uses), Top VMs by CPU % and by Memory % ranked lists, and a CPU Usage Share pie for the top 6 VMs. Everything is real data already available client-side or one existing endpoint away — no new telemetry collection was added.
  • Datastores now have an Overview tab alongside the existing per-VM Summary view (Storage tab, or the dashboard Datastores view — click a datastore name). Shows capacity, IOPS, throughput, availability history, and disk utilization. VM/host counts are real (drawn from the same domain-XML disk match as the existing Summary tab); capacity, IOPS, throughput, and history are clearly labeled as sample data until per-datastore telemetry sampling exists.

0.42.5.2-beta — 2026-07-28

Added

  • V2V Import now supports migrating multiple VMs from vCenter in one go. The "From vCenter (Live Copy)" tab's guest picker now lets you check 2 or more VMs at once. With multiple guests selected, pick an MFCloud placement cluster and the console automatically decides which KVM host each VM lands on — scoring free RAM/CPU headroom the same way "Deploy to cluster" does for new VMs, and spreading the batch across hosts rather than stacking them on one. A per-VM preview (target name, OS type, and assigned host) is shown before anything imports, and any row's assigned host can still be overridden by hand. Single-VM imports, and the file-based (OVA/VMX-on-datastore) import path, are unchanged.

0.42.5.0-beta — 2026-07-27

This release rolls up everything since 0.42.3.7-beta into one build — the patch versions in between (0.42.3.7 through 0.42.3.14) were incremental local milestones that never individually shipped. Their detail stays below.

Added

  • LXD containers now have an Edit Hardware action. Previously CPU/memory limits could only be set at create time — now right-click a container (in the Containers tab or the sidebar tree) and change them live, no restart needed.
  • LXD containers now have a NIC manager. Add or remove network adapters on an existing container — previously only the single NIC set at create time was possible. The network dropdown offers the same choices (LXD- managed networks and macvlan port groups) as Create Container.
  • LXD containers can now attach to a macvlan-backed port group directly through the Create Container UI, not just LXD-managed networks like lxdbr0. New networks.macvlan_parent field lets a bridge-backed port group point at a host-side interface (a VLAN sub-interface, or the plain physical NIC for untagged/bridged attachment) that containers attach to via a macvlan NIC — closing a gap where the only way to get this networking shape was to call LXD's API directly.
  • NFS Datastores now show which VMs are actually on them. Clicking a datastore name in the Storage tab or the dashboard Datastores view opens a summary panel listing every VM with a disk on that share.

Changed

  • Replaced several plain browser pop-up boxes with proper on-screen forms — tenant/cluster resource quotas, cluster storage backend, renaming a datacenter, a cluster's HA failover timing, tagging a host's capabilities, and evicting a Ceph node all got a real form instead of chained prompt() boxes. Destroying a Ceph cluster is now one styled confirmation requiring you to type DESTROY, instead of two stacked pop-ups.
  • Combined the Datastores toolbar's three storage buttons into one — "Manage NFS," "Manage ZFS," and "LINSTOR / DRBD" opened the same window on different tabs, so now it's a single "Manage Storage" button.
  • Fixed a button getting cut off in the Datastore Manager's Enterprise Pools list — "Fix Permissions" and "Unmount" now live behind a single "⋯" menu / right-click, matching hosts and ZFS pools elsewhere.
  • Fixed hard-to-read status badges and action buttons across Storage — LINSTOR/DRBD replication status, and the bright-green primary buttons ("Create Policy," "Format & Create Array," "Deploy Replicated Volume," etc.) all used low-contrast colors in dark mode; both now use the same soft-tinted / darker-green style as the rest of the console.
  • Light theme cleanup — telemetry chart colors, capacity-bar tracks, RAM/disk mini-cards, floating-panel shadows, and sidebar/tab/task-drawer hover highlighting all had leftover dark-theme styling that did nothing (or looked broken) in light mode; all now follow the current theme.

Fixed

  • Right-clicking a container in the sidebar tree did nothing (the browser's own context menu appeared instead) — the tree's context-menu dispatcher had no branch for container leaves. Right-click now shows the same menu as the Containers tab table (Console, Stats, Snapshots, Edit Hardware, NICs, Migrate, Power, Delete).
  • K8s Scale Workers could fail with a raw qemu-img error when a requested worker disk_gb was smaller than the chosen template's actual virtual size — now checks the template's real size first and raises a clear error naming the minimum disk_gb required, for every create_vm call, not just K8s workers.
  • K8s cluster node table had no visibility into DB/live drift — nodes the live cluster reports that MFCloud never provisioned, or nodes MFCloud believes exist that the live cluster no longer reports, are now flagged ("⚠ untracked" / a synthetic "Missing" row) with a way to clear stale bookkeeping.
  • LINSTOR CSI cluster delete left every node behind as an OFFLINE satellite — cluster delete now best-effort deregisters each node's LINSTOR satellite before dropping the DB row, instead of requiring manual cleanup afterward.

0.42.3.14-beta — 2026-07-27

Fixed

  • Light theme cleanup — several places still used colors leftover from the dark theme design that were never updated. Telemetry charts (Performance tab, VM/Container/Ceph monitoring) had washed-out axis labels and a nearly invisible "current value" readout on light backgrounds — chart colors now follow the current theme. CPU/RAM/disk capacity bars and modal progress bars had an invisible empty track. The RAM/disk telemetry mini-cards used the old bright green/yellow instead of the readable darker versions. A few floating panels (search results, task log, AI chat) had a much heavier drop shadow than everything else. Hover highlighting on the sidebar, inventory tabs, and task drawer silently did nothing in light mode since it was a "lighten a dark background" effect with nothing to lighten.

0.42.3.13-beta — 2026-07-27

Changed

  • Fixed the same hard-to-read green buttons everywhere in Storage. "Create Policy," "Format & Create Array" (ZFS), "Format & Add to Ceph Cluster," "Deploy Replicated Volume," "Extend Existing Pool," and "Finish Setup" (DRBD) all used the same overly bright, low-contrast green with white text as the LINSTOR/DRBD status badges fixed earlier — now they use a darker green that's easy to read in dark mode while still standing out as the primary action button.

0.42.3.12-beta — 2026-07-27

Changed

  • Combined the Datastores toolbar's three storage buttons into one. "Manage NFS," "Manage ZFS," and "LINSTOR / DRBD" all opened the exact same Storage window, just landing on a different tab — now it's a single "Manage Storage" button, and you pick NFS/ZFS/DRBD from the tabs inside as before. The separate Ceph HCI Cluster button stays on its own, since it opens a different, more specialized screen for managing the whole Ceph cluster.

0.42.3.11-beta — 2026-07-27

Added

  • NFS Datastores now show which VMs are actually on them. Clicking a datastore name in the Storage tab or the dashboard Datastores view opens a summary panel listing every VM with a disk on that share (VM, host, disk path) — previously the only way to check was to hunt through each VM's settings individually.

0.42.3.10-beta — 2026-07-27

Changed

  • Fixed a button getting cut off in the Datastore Manager's Enterprise Pools list. The row for each storage pool had two buttons ("Fix Permissions" and "Unmount") that no longer fit in the window and were getting clipped off the edge. Both actions now live behind a single "⋯" menu button (right-click the row also works) — same style used for hosts and ZFS pools elsewhere in the console.

0.42.3.9-beta — 2026-07-26

Changed

  • Fixed hard-to-read status labels on the LINSTOR / DRBD replication tables (node connection status, storage pool health, and the "Replication State" column on the Datastores dashboard). These used solid bright green/yellow badges with white text, which is very low contrast in dark mode — now they use the same soft-tinted badge style used everywhere else in the console.

0.42.3.8-beta — 2026-07-26

Changed

  • Replaced several plain browser pop-up boxes with proper on-screen forms. Setting a tenant's or cluster's resource quota no longer chains 2-3 separate gray text-entry pop-ups — it's one clean form now. Choosing a cluster's storage backend is now a dropdown of valid choices instead of a box where a typo would just get rejected. Renaming a datacenter, setting a cluster's HA failover timing, tagging a host's capabilities, and evicting a node from a Ceph cluster all got the same treatment.
  • Destroying a Ceph cluster is safer to confirm. This used to be two stacked pop-ups (a yes/no, then "type DESTROY"). It's now a single styled confirmation box, and the Destroy button stays disabled until you actually type DESTROY into it.

0.42.3.7-beta — 2026-07-25

Fixed

  • K8s Scale Workers could fail with a raw qemu-img error when a requested worker disk_gb was smaller than the chosen template's actual virtual size — qemu-img resize refuses to shrink a disk without --shrink (which we never want to pass; truncating a live filesystem corrupts the guest), so the operator saw an opaque subprocess exit-code failure instead of an actionable message. tasks/vm/storage.py now checks the template's real size with qemu-img info before attempting the resize (zfs/nfs/local backends — the ceph backend already handled this via its own grow-only comparison) and raises a clear error naming the minimum disk_gb required. Applies to every create_vm call, not just K8s workers.
  • K8s cluster node table had no visibility into DB/live drift — MFCloud had no way to flag a node the live cluster reports that MFCloud never provisioned (manual kubeadm join, or a scale-up whose DB commit never landed), or the opposite: a worker MFCloud believes it added that the live cluster no longer reports (crashed VM, failed join never rolled back). GET /{cluster_id}/nodes now cross-references worker_ips/ control_plane_ip against the live kubectl get nodes snapshot and annotates each row — the Nodes modal (static/js/kubernetes.js) shows an "⚠ untracked" badge for the former and a synthetic "Missing" row for the latter, with a "Clear" action to drop the stale bookkeeping entry. Additive only — never touches k8s_clusters state itself.
  • LINSTOR CSI cluster delete left every node behind as an OFFLINE satellite — a full cluster delete tore down the k8s engine on each node but never told the LINSTOR controller, so LINSTOR-backed clusters had to be cleaned up by hand (Storage → LINSTOR → node delete) after every teardown. DELETE /{cluster_id} now best-effort deregisters each node's satellite (LinstorClient.delete_node) before dropping the DB row — tolerant of an already-dead cluster or missing LINSTOR config, same as the existing per-node uninstall loop.

0.42.3.6-beta — 2026-07-25

Fixed

  • VM consoles fleet-wide could fail to connect ("Connection error") — every VM's XML correctly asks libvirt to bind VNC to all interfaces, but libvirt only honors that at define-time if the node's own /etc/libvirt/qemu.conf agrees; on affected nodes it silently fell back to loopback-only, so the console proxy could never reach it no matter what the VNC firewall rule allowed through. Node enrollment (services/node_provisioner.py) and the fast Repair Worker path (routers/hosts/repair.py) now both assert qemu.conf's vnc_listen = "0.0.0.0" and restart libvirtd, mirroring the existing VNC-firewall-rule re-assertion pattern — every node, fresh or repaired, gets this from now on. Doesn't retroactively fix already-running VMs on already-affected nodes (a VM's QEMU process keeps whatever address it bound to at its own start time) — those need a restart or a Repair Worker run against the node followed by a VM restart.

0.42.3.5-beta — 2026-07-24

Fixed

  • Sidebar inventory tree (VMs/Hosts/Storage/Networks folders) reset to fully expanded on every browser reload — the tree's collapse/expand state was only ever carried across the 3s polling rebuild in memory, never persisted, so a manually-collapsed folder snapped back open the moment the page was refreshed. Collapse/expand choices are now saved to localStorage (static/js/main.js) keyed by tree node id and re-applied both on the initial page load and on every subsequent tree rebuild — a folder now stays collapsed across reloads until explicitly re-expanded.

0.42.3.4-beta — 2026-07-23

Fixed

  • "Enter Maintenance" could silently leave VMs stuck, and permanently lock a compute node's row for every future admin action against it — the endpoint held one open DB transaction for the entire request, including a live SSH capacity probe of every other host in the cluster (core/placement.score_hosts) that can run for several seconds against a slow or unreachable node. If the client/proxy gave up waiting mid-probe, the cancelled request left that transaction open, and the connection leaked back to the pool "idle in transaction" — blocking any later Maintenance/ Activate/Reactivate call for that same host indefinitely, with no error surfaced anywhere until someone found and killed the stuck DB session. The endpoint now uses short, separately-committed connections instead of one request-long transaction, so nothing is held open across the slow probe — a cancelled request just cancels, it doesn't wedge the row.
  • Postgres now also enforces a 2-minute idle_in_transaction_session_timeout fleet-wide (previously 0/disabled) as a backstop for this class of leak in general, not just this one endpoint.

0.42.3.3-beta — 2026-07-23

Added

  • "Migrate VMs Back" failback action — a node returning from an HA fence no longer leaves its VMs stranded on their emergency host forever. Each failed-over VM now remembers the host it fenced away from (vms.ha_failed_over_from); once that host is healthy again, its right-click menu shows "↩ Migrate VMs Back (N)" so an operator can live-migrate them home in one click — offered, never automatic, since the operator is best placed to judge whether a just-returned node is actually ready for load. The flag clears on any explicit migration, so manually placing a VM elsewhere doesn't keep nagging to send it back to a home you moved it away from on purpose.

Fixed

  • HA fence could fail with a spurious "No module named 'services'" on a node's first-ever fence event — core/fencing.py's Ceph blocklist fence re-imported services.ceph_tasks on every call instead of once; the import is now resolved eagerly at module load and cached, removing that class of failure. A node with no BMC/iDRAC configured (nested/lab hosts) also no longer attempts and logs a power fence that was never going to work — it now checks BMC availability up front and goes straight to the Ceph storage fence.

0.42.3.2-beta — 2026-07-22

Fixed

  • Storage migration to/from Ceph could hang indefinitely — the disk-convert helper behind Move Disk read qemu-img convert's progress from stdout only after its stderr had been fully drained, a deadlock risk any time the child writes enough to stderr to fill the pipe buffer before finishing. Local/ZFS/NFS moves rarely write to stderr so this stayed hidden, but the Ceph (RBD) path routes through librbd/librados, which is chatty enough to trigger it — a migration to or from Ceph HA storage could sit stuck at "Running" forever instead of completing or failing. stderr is now drained concurrently on a background thread so it can never block the progress read.
  • ISO/large file upload showed no progress — the file browser's Upload button (Datastores → Browse) sent the file via fetch(), which gives no upload-progress signal; a multi-GB ISO would sit behind a static "Uploading…" toast for however long the transfer took, indistinguishable from a hang. Now uses XMLHttpRequest with upload.onprogress to drive a live percentage bar and transfer rate, and turns red with the actual error detail (e.g. a Caddy 413) instead of a bare "Upload failed" on failure.
  • Large ISO uploads rejected outright — Caddy's shared request_body cap was 2GB, applied to every route behind it including the upload endpoint, so anything past 2GB never reached the app. Raised to 20GB.

0.42.3-beta — 2026-07-21

Added

  • Resizable inventory sidebar — drag the right edge of the datacenter tree (the Hosts / VMs / Storage / Networks navigator on the left) to widen it. Previously fixed at 280px, so VM and host names with an IP suffix (e.g. MF-VDI-04 (192.168.x.x)) were clipped by the tree's ellipsis with no way to see the full name. Width persists across sessions (200–640px range).

Fixed

  • Cluster/folder row action icons stranded far from the name — the ⓘ / ✎ / 📁+ / 🗑 buttons on cluster and folder rows used margin-left:auto (and the row label used flex:1 1 auto), both of which pin trailing content to the row's far right edge. That was invisible at the old fixed 280px width but left a large empty gap after short names once the sidebar could be widened. Icons now sit immediately after the name.

0.42.2-beta — 2026-07-21

Image hardening pass — no application code changes, console behavior is identical to 0.42.0-beta. Ships from a new public registry (see below).

Security

  • Shipped image CVEs cleared (21 → 0) — the published mfconsole/mfconsole Docker image carried 8 HIGH / 10 MEDIUM / 3 LOW vulnerabilities in OS-level Python packages (cryptography, urllib3, setuptools, jwcrypto, requests, idna) pulled in transitively by the base image's package manager — never imported by the running app, but visible to any image scanner (Docker Scout, Trivy, etc.) now that the repo is public. Dockerfile.ship's runtime stage now patches these in place and clears the stale package metadata so scans come back clean.

Changed

  • Distribution moved to a public Docker Hub org — mfconsole/mfconsole replaces the prior self-hosted, token-gated registry.mfcloud.io. No docker login is required to pull. See mfconsole-ship/README.md for the access-control tradeoff this implies.
  • One-command install — the deploy-files bundle now includes install.sh, which wraps .env setup, setup.sh, image pull, and docker compose up into a single step.

Fixed

  • setup.sh never generated DB_PASS — it shipped as the literal placeholder string instead of a random value; now generated like the other secrets.
  • DEFAULT_ADMIN_PASS placeholder mismatch — following the documented install steps left the first-login password as the raw placeholder string instead of the documented mfpro default.

0.42.1-beta — 2026-07-21

Added

  • Editions / licensing system — MFCloud now ships six editions: Free (default, 2 nodes / 10 VMs, no paid features), Homelab ($14.99/mo, 3 nodes / 100 VMs, personal/non-commercial use), Starter ($29/mo, 5 nodes, unlimited VMs, adds S3 backup), Growth ($79/mo, 10 nodes, unlimited VMs, same feature set as Starter — bridges the old Starter→Business price cliff), and Beta/Pro (unlimited nodes and VMs, every feature flag on: HA, S3 backup, multi-tenancy, LDAP, AIOps, Commander, Kubernetes). License keys are Ed25519-signed (MFC2.<payload>.<sig> — the signing private key never ships; only the public key is embedded, so a customer with full source access still cannot forge a key) with the original HMAC-signed MFC1 format still accepted for compatibility with any beta test keys already handed out. A lapsed valid_until downgrades a deployment to Free-tier caps at read time without touching the stored license row or any existing VM/node — enforcement only blocks new node/VM creation and gated-feature access (core/license.py, routers/system/license.py, Settings → License). Node/VM limits and feature gates (assert_can_add_node, assert_can_create_vm, assert_feature) are enforced at every relevant create path across hosts, VMs, HA, S3 backup, multi-tenancy, LDAP, AIOps, Commander, and Kubernetes. An optional, fire-and-forget activation ping (LICENSE_ACTIVATION_URL, off by default) lets the vendor flag a paid key later reused on a second deployment for manual follow-up — it never blocks or remotely disables a console; a valid key is honored fully offline regardless.

0.42.0-beta — 2026-07-17

V40 (Managed Kubernetes Engine) completion pass. The HA and LINSTOR features are code-complete but not yet live-tested — see KNOWN_ISSUES.md → Kubernetes for the pending checklist.

Added

  • YAML manifest deployments — each cluster row has a new YAML button: upload (or paste) a raw Kubernetes YAML file — multi-document supported — and the console runs kubectl apply on the cluster's control-plane. Manifests are listed in their own table with kubectl's own created/configured output as the status detail; re-uploading under the same manifest name updates it in place (declarative apply), and Delete replays the stored YAML through kubectl delete -f. Uploads are pre-parsed server-side so a YAML typo fails immediately, not minutes into the apply task.
  • Per-deployment logs — App Catalog deployments and YAML manifests each have a Log button showing the full deployment trail (Queued → Deploying → Waiting for pods → Success/Failed, including error details), on top of the live Recent Tasks stream that already existed.
  • MCP: deploy apps from any MCP client — the mfcloud-mcp server gains six tools: list_kubernetes_clusters, list_app_catalog, list_app_deployments, get_app_deployment_log, and the confirmation-gated propose_app_deployment / confirm_app_deployment pair (same server-enforced propose→review→confirm flow as VM deployment; single-use 5-minute tokens bound to the engineer's own API key).
  • HA Kubernetes control plane — the deploy wizard now offers a 3-node control plane (embedded etcd quorum) behind a kube-vip virtual IP, chosen at create time. Workers and the downloaded kubeconfig target the VIP, so losing any single control-plane node no longer takes down the API. The VIP is leased from a normal IPAM pool (visible in the leases table, released on cluster delete) and must come from a different pool than MetalLB's. Existing single-CP clusters and the 1-CP path are unchanged. Day-2 operations (node list, worker scaling, storage/networking retries, App Catalog) are now control-plane-failover aware on all clusters.
  • LINSTOR CSI storage — clusters can attach to the platform's existing external LINSTOR controller: every K8s node becomes a diskless satellite, data replicas stay on the compute-node satellites via a dedicated resource group, and a mfcloud-linstor StorageClass (expansion enabled, WaitForFirstConsumer) is created. Joins the existing Ceph CSI option.
  • Per-tenant Kubernetes cluster quota — max_k8s_clusters ceiling (0 = unlimited) enforced at cluster create, settable from the IAM Resource Quotas widget alongside vCPU/RAM.
  • Real TLS certificates (internal PKI) — new TLS_MODE in .env, applied by setup.sh (regenerates the Caddy TLS config in tls.d/): internal (self-signed, unchanged default), byo (operator cert from AD CS or any internal CA, dropped into ./tls/; optional IP-SAN coverage for the by-IP site via TLS_BYO_COVERS_IP), and acme-internal (auto-issue and renew from an internal ACME CA such as step-ca). Trusting the built-in Caddy root CA (export + GPO) is documented as the zero-config path. setup.sh auto-adds https://<TLS_DOMAIN> to ALLOWED_ORIGINS.

Changed

  • OVN-Kubernetes CNI formally dropped from the roadmap — upstream owns its own in-cluster OVN stack and cannot attach to MFCloud's external OVN fabric; Flannel/Canal + MetalLB remain the supported path.

Fixed

  • K8s worker auto-naming no longer collides after a scale-down — the next -wN suffix is now derived from the highest existing suffix instead of the worker-list length, so removing w2 and then adding a worker no longer re-mints w3 and fails with "VM name already taken".
  • Device-mode ZFS clusters get the right scale-up pre-flight — adding a worker to a zfs_pool_source='device' cluster no longer applies the file-mode "disk_gb ≥ pool file + 10 GB" guard (the request-model default of 50 GB was being persisted even for device clusters), and the API now enforces what device mode actually needs: a csi_zfs_pool_name/size for every auto-provisioned worker (previously only the UI checked this).
  • MetalLB pool collision checks now also run at cluster create — the create wizard's MetalLB pool is rejected if it is any cluster's API-VIP pool or already another cluster's MetalLB pool (two MetalLB instances on one range assign duplicate Service IPs); previously only the day-2 Configure Networking path checked, and pool sharing wasn't blocked at all.

0.41.6-beta

Added

  • AI Assistant panel: resize + per-answer stats — the chat panel can now be resized from its top-left corner (size persists, like its position), and each answer shows a stats line: a live "Thinking · Ns" timer while the model works, then total response time and token usage (in/out) reported by the backend (Claude API usage, or Ollama's eval counts).

Fixed

  • AI Assistant input no longer locks up after a backend error — an error mid-turn (e.g. an Ollama timeout) ended the stream without the completion envelope, leaving the message box disabled until the panel was reopened.

0.41.0-beta — current

Upgrading an existing deployment: the compose network is now pinned to a fixed subnet (172.30.238.0/24). Docker Compose cannot change a network in-place — run docker compose down once before up -d with this version.

Added

  • Kubernetes worker scaling — each cluster row now has a Nodes modal showing a live node snapshot (name, IP, role, Ready state), with Scale Workers to join additional workers onto a healthy cluster and drain-and-remove a single worker. Progress is reported through Recent Tasks.
  • App Catalog — deploy curated application stacks (Ollama + Open WebUI, Jupyter + PyTorch, ComfyUI, vLLM) onto an MFCloud-managed Kubernetes cluster from the new App Catalog in the Kubernetes tab. Apps are exposed via a MetalLB LoadBalancer when available, NodePort otherwise. All stacks are CPU-only in this release (GPU support arrives with GPUaaS).
  • Ask AI from search — the Ctrl+K search box now offers an "Ask AI" row on any query, handing it to the console's AI chat panel — useful for questions ("why is this VM slow?") that no inventory name matches.

Changed

  • License gating extended — LDAP, AIOps, and the AI assistant are now edition-gated alongside HA, S3 backup, and multi-tenancy. Existing keys for editions that include a newly gated feature keep it automatically — no key re-issue needed.

Security

  • Hardening pass (2026-07-13) — 17 fixes from a full-codebase review, including: a tenant-isolation bypass on VM power actions and a cross-tenant VM telemetry route; a cloud-init SSH-key injection; Ceph dashboard and LDAP/OIDC bind secrets now encrypted at rest (AES-256-GCM); the worker→master internal callback now pins the master's TLS certificate fingerprint instead of skipping verification; a race in SSH host-key first-pinning; a narrowed trusted-proxy range on the login rate limiter (with the compose-network subnet pin above); RBAC initialization failures now abort startup instead of serving with half-seeded permissions; and database-failover handling that discards connections to a demoted primary.

Fixed

  • Dashboard could hang when a compute node went offline — libvirt connections to unreachable nodes are now bounded, so the VM list stays responsive while a node is down.
  • App Catalog: Jupyter + PyTorch deploys crash-looped — two environment collisions (a template-variable clash and Kubernetes' auto-injected service port variable overriding the image's own port setting) are both fixed.
  • Terraform destroy returning a 500, a route-shadowing bug that hid the pending-node-approval UI, and a virsh regression on localhost-managed hosts.
  • Shipped console images now include the Emergency Host UI overlay (agent_overlay/), which was silently missing from registry builds.

0.40.5-beta

Covers 0.40.1–0.40.5; combined here since they shipped in quick succession.

Added

  • Usage Reports — new admin Reports tab: Compute and Network usage over a chosen time range, plus a point-in-time Storage snapshot, each exportable to CSV.
  • Host anomaly detection — the telemetry pipeline now flags hosts whose CPU/RAM/disk/network behavior breaks from their own recent baseline.

Changed

  • Datacenter-console visual reskin — new masthead color tokens (navy header in both light and dark themes), flatter corner radii, uppercase button labels, underlined tabs, and reduced glow/shadow throughout; fixed a light-mode readability issue in the inventory tree.
  • Telemetry pipeline — the background sampler now runs its per-host collection in parallel (fixes an intermittent stall), and the rollup used by Reports/telemetry charts was widened to cover disk and network in addition to CPU/RAM, making the Reports tab noticeably faster to load.

Fixed

  • Network security groups were not being enforced at all — ACL pushes were silently targeting a logical switch that didn't exist, and the failure was swallowed. Security groups now resolve the VM's real switch and correctly union rules across multiple assigned groups.
  • IPAM: a lease-allocation race, a reserved-IP bypass, a leftover logical switch port on delete, and a pool-naming conflict were all fixed.
  • Storage: an orphaned LINSTOR resource-definition case, a datastore name-conflict split-brain, several leaked SSH connections, and missing guards around iostat output and node-count edge cases.

0.40.0-beta

Added

  • ZFS pool replication + VM restore (DR) — schedule async, snapshot-based replication of a ZFS pool to another host in the same cluster, with a live progress bar (when pv is installed) and a one-click Restore to rebuild and boot the replicated VMs on the target if the source goes down. See the new Knowledge Base guide: ZFS Pool Replication & VM Restore (DR).
  • Windows Secure Boot + vTPM 2.0 across all three guest paths — fresh deploy, V2V import, and the VM Settings firmware toggle. Fresh deploys enroll Microsoft's keys automatically for a Windows guest on UEFI; V2V import and the Settings toggle preserve the guest's existing UEFI variable store (so boot entries and any already-enrolled keys survive).
  • New Knowledge Base guide: Install a Windows VM (Server & 11) — VirtIO driver loading, firmware/Secure Boot settings, and converting to a reusable template.
  • What's new in this release — the console now shows a short release note the first time you log in after an update, with a bell icon in the header to reopen it any time. Sourced from this same changelog.

Changed

  • The Deploy VM dialog and VM Settings → Boot Options now expose a Secure Boot control (Auto / On / Off) alongside Machine Type and Firmware.

V39.17-alpha — public-alpha hardening

The V39.13 → V39.17 band is the public-alpha hardening track. See SECURITY_AUDIT.md and KNOWN_ISSUES.md for the security posture and the short list of accepted limitations.

Added

  • Compact density mode — console-wide density toggle in the top nav (Comfortable ↔ Compact), persisted per-browser and applied flash-free, driven by a single density-token scale so every table/card/widget re-paces at once.
  • Sticky table headers — column headers stay pinned while long VM / host / task lists scroll; sharper row-hover with an accent edge.
  • Console identity — the inventory-tree root now shows the server hostname (or CONSOLE_NAME from .env) so engineers managing several consoles can tell them apart at a glance.

Security (hardening pass — see SECURITY_AUDIT.md "Audit Pass 2")

  • Secrets hygiene: .gitignore + .env.example; setup.sh bootstraps .env and generates DB_PASS / DEFAULT_ADMIN_PASS / REDIS_PASSWORD.
  • Redis broker requirepass + provisioner credential-preserving URL rewrite.
  • Forced first-login password change (must_change_password + blocking overlay).
  • Closed cloud-init YAML injection, config-export secret leak, OIDC token-in-URL, cross-tenant VM access, unauthenticated backup/snapshot routes, webhook SSRF, and hardcoded admin elevation. Weak INTERNAL_SECRET now fails closed.

V39 — Distributed clustering, DR, and backup UX

  • V39 / V39.0.2 — Distributed container clustering: native LXD/Incus dqlite quorum, replicated container storage (LINSTOR/DRBD), free-RAM scheduler, live migration, and HA auto-evacuation of workloads off failed members.
  • V39.6 — Console navigation & UX pass: neutral product naming pass; inventory and recent-tasks polish.
  • V39.7 — PCIe passthrough in VM Settings; UEFI-VM lifecycle fixes.
  • V39.8 — Agentless S3 Disaster Recovery: dirty-bitmap incrementals, ZSTD streaming, AES-256-GCM direct-to-cloud upload, one-click cloud restore.
  • V39.9–V39.12 — Backup UX: cluster-scoped NFS datastores, destination picker, scheduled backups v2, capacity bars, and live task progress.

V38 — Containers, marketplace, and identity federation

  • V38 — LXD / Incus container support: lifecycle, interactive console (xterm.js), live metrics, snapshots/restore, remote image import, and OVN logical-switch binding.
  • V38.2 — VM table right-click context menu.
  • V38.3 — Agent callback restoration & VM inventory discovery.
  • V38.4 — Turnkey marketplace & dedicated Catalog tab.
  • V38.5 — Trust pinning, recent-tasks polish, NFS hardening, light/dark theme.
  • V38.7 — Identity federation (OIDC/SSO): "Sign in with Microsoft" (Entra ID, Google, Okta), auto-provisioning, group-claim → role mapping, RP-initiated logout, shorter OIDC token TTL.

V37 — Database high availability

  • V37 — Patroni + etcd + HAProxy: TimescaleDB in HA; writes routed to the current primary; sync (psycopg2) and async (asyncpg) pools.

V31–V36 — Platform foundation

  • V31.6 — Enterprise identity: LDAP / Active Directory authentication.
  • V31.7 / V31.8 — Performance tab redesign and inventory tree.
  • V31.10 — SDN: OVN logical networks + security groups.
  • V31.11 — LINSTOR REST API rewrite + DRBD cluster wizard.
  • V31.12 / V36 — Hypervisor ISO + imaged-node fast path: zero-touch node provisioning and enrollment.
  • V31.13 — Security audit pass; Caddy TLS termination; password-only SSH path.
  • V32 — Agent Trust API: TOFU host-key pinning, agent credential delivery.
  • V33 — Releases incl. V33.2 iDRAC / BMC hardware health monitoring.
  • V34 — Cloud-Init guest customization.
  • V36.1 — Knowledge base bootstrap (served at /kb).

V23–V30 — Initial platform

  • Core VM lifecycle (create/start/stop/delete/edit), snapshots & backups with retention, clone + template conversion, live/cold migration.
  • VNC console proxy (noVNC) and web SSH terminal over WebSocket.
  • Host auto-bootstrap, maintenance + VM evacuation, multi-host telemetry.
  • OVS VLAN portgroups, KVM NAT networks, NFS datastores, image management.
  • Ceph cluster bootstrap + OSD management; ZFS pool management.
  • JWT + API-key auth, RBAC, per-user vCPU/RAM quotas, webhook alerts, task history, real-time task streaming.