Changelog & Roadmap

Found while investigating a fleet-wide false alarm on the Hosts table.

Bug Fixes

• Compute nodes updated by the console's own automatic sync could stay flagged “needs repair” forever, even once fully up to date. Only the manual Repair Worker button was recording the version it had just pushed; the automatic path now does the same.

Found by a post-deploy fleet health check, right after 0.45.5.1 went out.

Bug Fixes

• A healthy Ceph storage cluster could show as unreachable on the dashboard. Newer Ceph releases report cluster health in a different shape than older ones, and the monitor only recognized the older one. The cluster itself was healthy the entire time — this only affected what the dashboard displayed. Now recognizes both.

Caught in review of 0.45.5.0's new AI Assistant features, before they saw wider use.

Bug Fixes

• A long AI Assistant conversation could permanently break mid-session. Automatic conversation summarization (new in 0.45.5.0, below) could occasionally cut the history at the wrong point. Fixed so summarization can never split an in-progress action.
• A message sent at the exact moment the AI Assistant was summarizing a conversation could silently disappear. Fixed.
• The “Explain this task” shortcut could bypass an organization's configured AI usage ceiling. It now counts against the same monthly limit as the main chat panel.
• The GPU-reservation fix below could, in a narrow timing window, clear a different VM's GPU assignment instead of only the failed one's. Fixed to only ever release its own reservation.

Introduces usage controls for the AI Assistant chat panel, and closes a GPU-reservation leak.

New Features

• Per-tenant AI Assistant usage tracking, with an optional monthly spend ceiling. Every conversation now logs its token usage; admins can optionally cap monthly spend per tenant from Settings, with a running total visible to the tenant.
• Long AI Assistant conversations now compact automatically instead of growing without bound, keeping sessions responsive.
• IPv6 groundwork in the software-defined networking layer — foundation for dual-stack tenant networks, not yet exposed in any provisioning workflow.

Bug Fixes

• A failed VM creation could permanently strand a reserved GPU, recoverable only by retrying under the exact same VM name. Fixed, with a new admin option to manually release a stuck GPU reservation.

Caught immediately after deploying 0.45.4.0, before it saw wider use.

Bug Fixes

• The new node-version-mismatch check (0.45.4.0, below) never actually detected anything on a real multi-node deployment. Fixed the comparison logic and verified against the live fleet.

Compute nodes now report their own software version, closing a blind spot for fleet upgrades.

New Features

• Compute nodes now report their own software version, and the Hosts table flags any node that's out of sync with the console. Click the badge to run Repair Worker directly, instead of having to infer it from release notes. Every already-enrolled node shows this badge once, harmlessly, until it's repaired under this release.

Caught in review before 0.45.3.6's new HA Manager restore panel saw wide use.

Bug Fixes

• The new control-plane restore panel could fail to stop/start services on some installs. It referenced an internal file that isn't part of every deployment layout — registry-pull installs in particular. Now resolved automatically per host, the same way an existing internal mechanism already does it.

Control-plane backup and restore is now manageable from the out-of-band HA Manager, plus a safety fix for the restore download's integrity check.

New Features

• Control-plane backup and restore is now reachable from the out-of-band HA Manager (:9444 on every control-plane host). A new panel lists available backups and can trigger a full restore — out-of-band like the rest of that UI, so it works even with the main console down.

Bug Fixes

• A corrupted control-plane backup download wasn't reliably caught before a restore. The download's integrity check only logged a warning on failure instead of stopping the restore — it now hard-fails as it should.

Found while exercising a real control-plane backup end to end: the restore half had a bug that would have mattered in an actual disaster recovery.

Bug Fixes

• Every automated control-plane backup verification was failing, and a real disaster-recovery restore following the documented steps would have failed too. The console's database needs a couple of extra steps beyond a plain restore; those are now applied consistently everywhere a restore happens — the automated nightly verification, the disaster-recovery runbook, and the manual restore tool. Restore failures also now show the actual underlying database error instead of a generic one, making any future issue far easier to diagnose.

Found while testing control-plane backups against a real Backblaze B2 bucket.

Bug Fixes

• VM backups and control-plane backups sent to a Backblaze B2 bucket failed every time with a connection error. A default setting in the upload library conflicts with B2's S3-compatible API; it's now disabled for all S3-compatible targets. AWS S3 and MinIO were unaffected either way and see no change in behavior.

Packaging gap closed in the public Docker Hub image: two pieces of the 0.45.3.0 control-plane backup feature were missing from the shipped image, even though the feature itself worked correctly.

Bug Fixes

• The public image shipped with no database backup/restore tools, so nightly control-plane backups silently failed. 0.45.3.0 added this dependency to the internal build, but the separate build used for the public image was never updated to match — every registry-pull deployment on 0.45.3.0–0.45.3.2 had a worker container missing it entirely. Now installed.
• The manual disaster-recovery restore helper was never included in the public image at all. The runbook told operators to run it, but the packaging step that decides what ships never picked it up. Now included.

Critical fix for a fleet-wide regression introduced in 0.45.3.0: every enrolled compute node's worker service was crash-looping, silently blocking all VM/container create, power, and migration operations cluster-wide.

Bug Fixes

• Every compute node's worker service crash-looped after updating to 0.45.3.0. A new module imported a dependency that's never shipped to compute nodes by design, and since it loaded unconditionally at package-import time, every node's worker process failed to even start — regardless of whether that specific feature was ever used. VM/container create, power, and migration requests were accepted by the API and written to history as "Queued," then silently never executed. The Queue Registry showed every compute node Offline from the moment it updated. The dependency is now loaded only where it's actually used, which never happens on a node.

Found live during a post-deploy cluster health check: the LXC Marketplace catalog was failing to refresh on all 3 control-plane hosts.

Bug Fixes

• The LXC Marketplace catalog stopped refreshing. GET /marketplace/catalog was throwing TurnKey mirror unreachable: HTTP Error 403: Forbidden on every refresh — Cloudflare (fronting the TurnKey mirror) silently 403s the bare Python User-Agent urllib sends by default, while a real browser gets 200 from the same host at the same time. All mirror requests now send a browser-like User-Agent — this also fixes TurnKey appliance installs, which were silently broken by the same root cause.

Opens the 1.0 GA cut line: full audit-trail coverage for identity/tenancy mutations, and the first real control-plane disaster-recovery mechanism.

Security Fixes

• Every identity/tenancy mutation is now audited. User creation, role grants, and tenant changes were previously invisible to the audit trail — "who granted this person Super Admin, and when" was unanswerable. All 13 mutating routes across user and IAM management now write an audit-log entry after the change commits, enforced going forward by an automated CI check.

New Features

• Automated, verified control-plane backups. The console's own database and non-DB state (not just your VMs) now back up nightly to your configured S3 target, encrypted the same way VM backups already are. Every run also test-restores the dump into a throwaway database and queries it — real restore verification, not just a check that the backup job exited cleanly. Manage retention, schedule, and history from Settings, with an on-demand "run now" trigger and a manual restore helper for disaster recovery.

Security/code-review hardening pass on 0.45.2.2's WireGuard relay feature, found before any relay was ever provisioned against it in production.

Security Fixes

• WireGuard private keys no longer transit the console's backend. Relay and chassis provisioning previously read the private key back over SSH into a Python variable before writing it back out — it now stays entirely on the box that generated it.
• Removing a chassis now revokes its WireGuard peer on the relay. Unpairing previously only touched the chassis side, leaving the relay's live tunnel config trusting a decommissioned box's key indefinitely.
• Relay/chassis inventory now requires network-management permission, instead of being visible to any authenticated user on any tenant.
• The container health-check endpoint added in 0.45.2.2 is now loopback-only and no longer echoes raw database error detail to unauthenticated callers.

Bug Fixes

• Chassis tunnel-IP allocation no longer collides after a chassis is removed and re-added. The allocator now derives the next address from the highest already assigned, backed by a database-level uniqueness constraint.
• Tunnel health-check failures now update the stored status instead of leaving the Networks tab showing a stale "Active" state indefinitely.

Turnkey WireGuard provisioning for the Home VPS Public IP Relay — MFConsole now SSHes into a relay VPS and a home hypervisor and pairs their WireGuard keys automatically.

New Features

• Home VPS Public IP Relay. Give a home-lab hypervisor behind CGNAT (no public IP) a stable public presence by pairing it over WireGuard with a small relay VPS, set up and paired automatically from the console — no manual WireGuard configuration required.

Fresh installs (empty database, no prior tables) crashed on every startup — a schema migration-ordering bug. Existing deployments were unaffected.

Bug Fixes

• A brand-new database could crash-loop on first boot. lxd_containers.crs_last_migrated_at was added roughly 194 lines before CREATE TABLE IF NOT EXISTS lxd_containers itself, so a fresh database hit UndefinedTable: relation "lxd_containers" does not exist and never came up. Moved the column-add to after the table creation.
• The Home VPS Relay feature's relays/relay_chassis tables were missing their CREATE TABLE IF NOT EXISTS entirely — same class of fresh-install gap. Added alongside the schema's other tables.

GPU utilization and VRAM on the Host Monitor tab (NVIDIA and Intel), a new llama.cpp AI Assistant backend for GPU-accelerated local inference, and a fix for Windows VM deploys that could fail outright on some hypervisor builds.

New Features

• GPU utilization and VRAM on the Host Monitor tab, alongside CPU/RAM/Disk/Network — NVIDIA and Intel GPUs both supported. Host-level only: a GPU passed through to a VM is invisible to the host by design, so per-VM GPU monitoring isn’t covered here.
• New AI Assistant backend: llama.cpp. SYCL-accelerated for Intel GPUs, alongside the existing Claude and Ollama backends — measured roughly 8× faster token generation than Ollama on the same card and model (3.9 → 32.3 tok/s). Both backends can run at once; pick per chat session.
• CPU/RAM now get CUR/AVG/MAX and Pause/Resume on the VM and Host Monitor tabs, matching what Network already showed. Pause freezes the visible chart without stopping collection, so Resume never replays a gap.
• The AI Assistant panel can dock to the right edge as a fixed sidebar instead of only floating over the page.

Bug Fixes

• Windows VM deploys could fail outright on some hypervisor builds. The boot disk now attaches to a SATA/AHCI controller — Windows has shipped an inbox driver for it since Vista — instead of a SCSI chip some QEMU builds don’t compile in. No virtio-win.iso driver-load step needed either way.
• A brief database hiccup could make a network or security-group change silently fail to apply. The console kept showing it as applied even when the write never reached the OVN fabric. Fixed, and the last known-good state is now cached so a blip can’t redirect where writes go at all.
• The IaC/Terraform API rejected VM-create requests that included a GPU, and silently ignored a requested tenant assignment. Both fields are now accepted correctly.

UI Improvements

• The inventory tree is keyboard-navigable for the first time — arrow keys to move and expand/collapse, Enter/Space to activate.
• The ~67 legacy modal dialogs now close with Escape and trap keyboard focus, matching the newer modal system.
• The sidebar now auto-collapses and re-expands live as the window is resized, not just on page load.
• Accessibility pass: 49 icon-only buttons gained screen-reader labels, and a low-contrast status badge color was fixed in light theme.

A packaging bug in 0.45.0.0 could leave compute nodes unable to come back online after updating.

Bug Fixes

• Compute nodes could get stuck unreachable after updating to 0.45.0.0. A file the new HA-Ready fix depends on was missing from the package pushed out to nodes, so the node-side worker service failed to start and the console lost contact with the node until this fix. If you updated to 0.45.0.0 and a node still shows unreachable after updating to this version, use Repair Worker on that host to resync it.

A VM HA badge that could read “safe to fail over” when a real incident would have refused to move the VM — plus a license expiry warning, so a key doesn’t lapse silently.

New Features

• License expiry countdown. A top banner appears from 14 days out, with a shortcut straight to Manage License. Dismissal is per warning tier — 14, 7, 3 and 1 day, and expired — so clearing the two-week notice doesn’t silence the final-day one. Perpetual keys never warn; expiry still only reverts to Free-tier caps, existing VMs and nodes keep running.

Bug Fixes

• The HA-Ready badge could tell you a VM was safe to fail over when it wasn’t. The badge judged “HA-Ready” from storage type and survivor count alone — it never checked whether the host actually had an iDRAC/BMC configured. The failover engine itself always got this right: a non-Ceph VM on a BMC-less host is skipped during a real failover, since there’s no way to confirm the dead host is actually powered off first. Only the badge was lying, and an operator would have found out mid-incident. It now checks BMC presence per host and shows HA-Degraded instead of green wherever a real failover would refuse to act. A live audit of the production fleet found no VM currently in that state.
• ZFS cross-host replication could be pointed at a target pool that wasn’t empty. Replication force-rolls-back the destination to match the incoming stream, so an existing dataset on that pool — including a live VM’s disk — would have been silently destroyed on the first run. Creating a schedule now checks the target first and refuses if it’s already in use. Leftover replication snapshots are also pruned automatically now instead of accumulating on both ends forever.

GPU-as-a-Service reaches SR-IOV virtual functions and Kubernetes. Whole-card GPU passthrough has been proven in production for a while — this release is about the two paths around it.

New Features

• SR-IOV virtual functions are now assignable GPU rows, picked from the same GPU widget as a whole card, instead of showing up as an unlabeled generic PCI device.
• LXD containers can hold a GPU for the first time, through a new GPU panel on the container hardware menu — previously VM-only.
• A Kubernetes node can request a GPU at provision time. Once a GPU-carrying worker joins the cluster, the matching device-plugin DaemonSet is applied automatically — no extra step.
• The vLLM Marketplace deploy uses a GPU automatically when one is there — no toggle to find, it detects the hardware and requests it.

Freshly shipped — verified against this codebase’s existing GPU helpers, but SR-IOV-into-Kubernetes hasn’t run against real hardware yet. Whole-card passthrough (VM and now K8s) remains the proven path today.

VMs on OVN tenant networks get an L4 load balancer — Kubernetes clusters have had one since V40, VMs had no VIP primitive at all.

New Features

• OVN-native load balancer. Define a VIP and its backend pool, then attach it to a logical switch or router. OVN’s load balancer is distributed — every chassis translates in its own flow tables, so there’s no appliance to place, scale or fail over, and traffic never detours through a central node. Attaching to a router is what makes a VIP reachable from outside the tenant network through a gateway.

Bug Fixes

• Editing a backend pool could silently detach the whole load balancer. Removing the last backend from a pool removed the VIP’s underlying object entirely, taking every switch and router attachment down with it. Updates now modify the existing object in place, so the attachments survive.
• Attaching a load balancer right after creating it could fail on a natural create-then-attach sequence, from a timing race between the two background syncs. Fixed by having the attach step push the load balancer itself first if it isn’t there yet.

OVN learns north-south routing. Tenant networks could only ever route east–west between themselves — the only way off one was bridging straight onto a physical VLAN.

New Features

• External gateway on a logical router, giving a tenant network an actual way out to the outside world instead of only to other tenant networks.
• SNAT and floating IPs. Masquerade a whole subnet behind one external address, or pin a single external address to a single VM in both directions.
• Gateway chassis HA. Nominate a second and third hypervisor to carry a router’s north-south traffic at lower priority, and OVN elects the highest-priority live one on its own — automatic failover with nothing else to configure.

Bug Fixes

• A gateway could look fully configured and forward nothing. Gateway chassis were scheduled by hostname, which OVN’s own chassis table never matches — so no chassis ever actually claimed the gateway, silently. Found and fixed during live validation of the feature above, before it shipped broadly.
• Removing a gateway chassis always failed on a command-flag mismatch. Fixed.

The LXD container platform closes its remaining gaps against VMs — quotas, off-site backup, cross-host migration, continuous load balancing, and an edition ceiling.

New Features

• Containers now count against tenant and cluster quotas, closing a gap where a tenant at their vCPU/RAM ceiling could keep allocating compute indefinitely by creating containers instead of VMs.
• Off-site container backup to S3, with the same encryption and per-container schedules as VM backup — including restore onto a different host. Every container backup is a full copy (LXD has no incremental export), and is crash-consistent unless you opt to stop the container for the run.
• Migration between unclustered hosts. Moving a container previously required a formed LXD cluster. The original is only deleted after the target confirms it fully arrived — a failure at any point leaves the container running exactly where it was.
• Container CRS continuously rebalances CPU, RAM and container-count spread across a cluster, the same way the VM balancer already does. Opt-in per cluster, and defaults to recommend-only rather than acting on its own.

Bug Fixes

• A brief database blip could stop the container HA monitor from watching the entire fleet for that cycle, instead of just the one member it couldn’t reach — found while verifying this release. Health checks are now isolated per member and retried, and a member whose status can’t be confirmed is skipped rather than treated as failed, since evacuating a healthy node on a bad guess is worse than doing nothing.

OVN gets the two things it needed to be trustworthy on its own: guests that actually receive the address IPAM allocated them, and something watching the fabric for silent drift.

New Features

• The IPAM lease now actually reaches the guest. Allocating an address and binding the port already worked — the VM still booted with no address configured, because nothing ever handed the lease to the guest. Two independent deliveries cover it now: OVN’s own DHCP responder answers on the VM’s own chassis, and the same lease is written into cloud-init for images that come up before the DHCP offer lands.
• OVN topology drift detection. A background sweep every 15 minutes diffs the console’s database against what OVN actually has, in both directions, and surfaces it on the Networks Topology tab — it flags drift, it doesn’t silently auto-repair it, since the two sides can diverge for opposite reasons.
• Rebuild OVN from the database re-pushes every switch, router, connection and security-group rule the console’s tables describe — the recovery action for whatever the drift sweep flags as missing.

Bug Fixes

• Deleting an IPAM pool could leave its DHCP configuration behind when another pool still shared the same subnet. Fixed.

Mostly about the install working where the product is actually meant to run, and about connections that could hang forever no longer being able to. Plus the first view that shows the whole network fabric in one place.

New Features

• Networks has a landing dashboard instead of only leaves. Selecting the root of the Networks tree now opens a view over the whole fabric: headline counts for L2 segments, logical routers, chassis, host bridges, security groups and address pools; a logical topology diagram drawn from the real router-to-switch attachments; per-chassis health; and address-pool utilization. It reads the data the tree already fetched, so it costs no extra API calls and cannot disagree with the tree beside it.
• A single Segments list across all three networking models. Portgroups, cluster VLANs and overlay segments are configured in three different screens and nothing previously showed them together. A new Segments tab lists them side by side, alongside Address Space (every pool with allocation meters and free counts).
• Two fabric problems that previously meant clicking every leaf are now called out by name — a logical switch with no router attached (east–west only, no gateway), and any chassis that is not active, marked on the topology with both a status color and a dashed border so it survives greyscale and color-vision deficiency.
• Live stack dumps for the control plane. The image now carries a sampling profiler that prints what every thread is actually executing, without stopping or restarting the process — so a hang can be explained rather than inferred, even when a restart is the only remedy and would destroy the evidence.

Bug Fixes

• Installing from the published image never worked on Podman — on any host without Docker. The bundle relied on a Compose feature Podman does not implement, and Podman is the default engine on the enterprise Linux hosts this product targets, so that was the entire pull-and-install path. The compose layering has been inverted so both engines now work from the same bundle.
• The auto-update helper failed to start on an unattended first boot — it was referenced by a short image name, which Podman refuses to resolve without a terminal to prompt at. That took out the container the console’s “Update Now” button depends on. Now fully qualified.
• Every message-broker connection in the console could hang forever on a host that vanished. A machine that is powered off or unplugged sends no shutdown packet, so its sockets stay established from the kernel’s point of view and the default keepalive will not question them for over two hours. All clients now share one hardened option set — measured against a blackholed peer, the request path is bounded at 2 seconds and background connections at 6, where the old ones never returned at all.
• The rate limiter could take the console down. With the shared store unreachable, every rate-limited route returned a server error. It can no longer turn a storage failure into one, and limits stay enforced from per-process counters during an outage rather than leaving the login endpoint unmetered.
• A typo in the deployment-size setting was indistinguishable from leaving it blank — a misspelled tier silently ran the untuned legacy profile while the operator believed their sizing was in force. It is now a startup error naming the valid tiers.

Performance

• Deployment sizing was documentation, not configuration. The setting had exactly one consumer — a line in the startup log — so choosing a tier changed no connection pool, no database setting and no worker concurrency, while muting the one genuine warning about the untuned profile’s connection peak. Sizing now drives the real values.
• The background worker’s connection pool ignored your configuration. A documented pool limit was silently overridden back to the default on that container, so a node could open four times more database connections than its operator had asked for. Settings now win where they should.

The control plane now holds up its own promises. Three components sat behind the HA VIP without actually being highly available, and the health check meant to catch that had never once succeeded.

New Features

• HA Manager — live telemetry and a one-click diagnostics bundle. Real CPU, network and disk rates with about an hour of history, charted per host alongside load, PSI, disk fill and temperatures — a host that stops answering keeps its charts, because the run-up is exactly what you need to see. One zip now collects every host’s container logs, journals, database state and network state, with secrets masked by default.
• Maintenance mode for control-plane hosts. Drain a host from the HA Manager and it hands over the database leader first, while the host is still healthy, then sheds the VIP through an ordinary health transition. The drain flag survives a reboot, so a host comes back still drained rather than seizing the VIP before anyone has confirmed it is fit.
• CRS can explain itself. A Preview next pass dry run shows the next move before it happens, and a Last pass line records each sweep’s verdict — so “balanced, CPU spread 4% against a 20% threshold” is no longer indistinguishable from “a trigger fired but nothing was eligible.”
• Networks tab modernized — the same shared dialogs as the rest of the console, typed-confirm prompts on deletes with real blast radius (which now show the affected pool and lease counts first), a filter box across all six tables, and client-side CIDR/MAC validation.

Security Fixes

• The Kubernetes pod-CIDR firewall grant could be removed with nothing narrower in its place — the narrowing step trusted an exit code instead of checking the resulting state, so on a host where the interface was already bound elsewhere it removed the broad grant, dropped cross-node pod traffic, and reported success. Every step is now verified by querying the firewall back.
• Third-party frontend libraries are no longer fetched from a public CDN into the page context of an admin console — all four are served from the install itself and version-pinned.

Bug Fixes

• Health-based VIP failover had never worked, on any host, silently. The failover daemon’s scripts run in a confined security domain that denied them the container runtime — and a denied execute check makes the shell report “command not found”, so every failure looked like an ordinary non-zero exit. All hosts sat permanently at reduced priority, and a sick host could only shed the VIP by dying outright. Fixed without weakening the security policy: the privileged work moved out of that domain entirely, and a transition now applies in under a second.
• The message broker behind the VIP was three separate, non-replicated instances. A failover repointed every console replica, worker and compute node at a different, empty broker, so queued background work vanished. The VIP holder is now the master and the other hosts replicate from it, promoted and demoted on every transition.
• The scheduler now follows the VIP instead of being pinned to one host. If that host died the console kept serving but nothing periodic fired at all — no telemetry, no backups, and no VM HA monitor, so VM failover stopped working exactly when it was needed.
• One host’s database proxy was a single point of failure for the whole control plane. All three replicas reached PostgreSQL through it, so powering that host off took the database away from the other two and surfaced at the login page as “invalid credentials”. Each host now uses its own.
• A VIP failover could lock every browser out of the console. With internal TLS each host ran its own certificate authority, and HSTS removes the “proceed anyway” escape hatch — a hard block with the cluster perfectly healthy. All hosts now share one authority.
• Telemetry charts, every terminal, and the Settings widget grid silently died on any install without outbound internet — the libraries behind them were fetched from a CDN at page load, which is unreachable on the egress-filtered management network that is the normal placement for a control plane. All are now vendored into the install.
• Deleting an IP pool could hand its addresses to a second VM while the original VM’s logical switch port still held them. Deleting a VM also left its port-forward firewall rules running forever, and a firewall reload or host reboot silently dropped every port-forward rule. All three fixed.
• CRS picked the roomiest destination host, not the coolest one — and could migrate a VM onto a host with no room for it, count powered-off VMs when deciding which host was “hot”, or re-queue a recommendation you had just dismissed.

Performance

• CRS predictive models went from measurably harmful to useful. The models were fine; the evaluation was broken — the validation split was an accidental per-host holdout masquerading as a time split. Corrected, CPU forecasting moved from far worse than assuming no change to +23% skill, and a promotion gate now refuses to serve any model that cannot beat that baseline.

New Features

• Topology board (Atlas). The Topology tab is now an interactive infrastructure planning canvas — drag nodes to reposition, drag from a node's connect handle to draw a new connector, and edit Title/Subtitle/Status/Tags/Parent/Fields in place. Seed from Inventory auto-populates the board with your real Clusters, Hosts, and VMs (safe to re-run — only adds what changed), and boards export to JSON or Mermaid.
• Two-level resource alerts — CPU and RAM thresholds are now a Warning and a Critical value instead of one, firing independently — plus a new Stale Snapshot alert that flags VM snapshots older than a configurable age.
• App Catalog grew from 15 to 23 curated stacks — Nextcloud, DokuWiki, Paperless-ngx, Prometheus, MariaDB, Metabase, Pi-hole, and Nginx Proxy Manager — redesigned as a 2-column card grid.

Security Fixes

• Kubernetes clusters no longer blanket-trust the entire pod CIDR in firewalld — a gap that let any pod reach any port on any node. Now only the confirmed CNI interfaces are trusted, with a reconcile endpoint for clusters provisioned before this fix.
• Multi-host HA clusters could mis-attribute every proxied request across replicas, collapsing the login brute-force limiter into one shared budget and mis-recording audit-log source IPs. Cluster hosts are now recognized automatically.
• API-driven user creation and admin-issued password resets now enforce the same password rules as the change-password flow — previously neither did.
• beat's primary-election guard now fails closed instead of open, so a .env that ever loses its primary flag idles instead of silently becoming a second scheduler and double/triple-firing every periodic task.

Bug Fixes

• Cross-node pod/Service traffic was silently dropped on every HA or multi-node Kubernetes cluster — firewalld wasn't trusting the decapsulated VXLAN packet once it landed on the receiving node. Fixed for both k3s and rke2.
• Fixed a Day-2 control-plane add that could leave a cluster permanently stuck out of "healthy," and a 1→HA promotion that could permanently fail every retry after one transient race. Live-verified end-to-end against a real 3-node HA lab, including LINSTOR CSI (real PVC read/write) and kube-vip VIP failover (~4s recovery).
• Boot disk could never be resized from the VM Settings modal — it now gets the same Resize control every other disk has.
• A round of Topology board editing fixes: clicking a node no longer instantly deselects it, low-zoom clicks no longer register as drags, names containing a quote or backslash no longer corrupt on edit, and an active search filter no longer drops on a board rebuild.

Security Fixes

• Direct-connect clients bypassing Caddy could spoof their source IP and defeat the login brute-force limiter (and forge the IP behind internal node-enrollment checks). The app server now only trusts forwarded-IP headers from the same trusted-proxy list the rate limiter itself uses, instead of any direct peer.
• A hardened install (admin account deleted or renamed for security) would silently reintroduce the default admin password on the next restart. Fixed with a real existence check instead of a username-only conflict check.
• An admin-issued password reset now forces a real password to be chosen at next login, same as first-time bootstrap.

Bug Fixes

• A new password longer than bcrypt's 72-byte limit could leave an old password still valid after a "successful" change, or crash the request — now capped correctly.
• An empty (but present) admin default-password variable in .env seeded a blank admin password instead of falling back to the documented default.
• Fixed the rate limiter's fallback Redis connection using the wrong password when run outside the compose stack.
• Fixed a database failover landing mid-startup that could leave a stuck internal lock and crash-loop that replica indefinitely.
• Fixed the HA app replicas colliding with the primary on host port 8000 when brought up without the HA profile.

New Features

• Kubernetes Engine gets a real fleet console. A health-first overview — total clusters, healthy/in-progress/failed counts, control-plane and worker node totals — replaces three stacked tables, with a per-cluster drill-down (Overview, Nodes, Add-ons, Workloads) and search/filter/list-view options.
• Multi-cluster Ceph support. A second Ceph cluster is now a fully first-class citizen — HA fencing, storage migration, VM disk placement, V2V import, and Kubernetes/LXD Ceph-CSI bootstrap all target the cluster that actually backs the node or VM in question, instead of an arbitrary row.
• VM Catalog: network and storage chosen at deploy time. Deploying a blueprint now opens a full dialog with a real network picker and a storage backend selector (local/ZFS/Ceph/NFS), instead of a 2-field prompt.

UI Improvements

• Every remaining native browser dialog is gone. 201 confirm()/alert()/prompt() pop-ups — unstyleable, theme-blind, and blocking — now use the console's own keyboard- and screen-reader-accessible dialogs. Destructive actions name the object and require typing it to confirm; focus defaults to Cancel.

Bug Fixes

• Ceph storage fencing could silently refuse to fence a dead node, leaving its VMs stuck instead of restarted elsewhere — the normal state for any cluster created since the last restart. Now falls back correctly instead of declining.
• Destroying a Ceph cluster could leave every compute cluster pointing at the deleted row, with no in-app way to recover — fencing, Ceph-backed deploys, migration, V2V, and capacity reporting all broke until fixed by hand. Destroy now clears the link atomically, and a startup migration repairs deployments already affected.
• Destroy Ceph Cluster could wipe the wrong cluster's registration entirely once a second cluster existed — a live data-loss trap, now scoped correctly.

Security Fixes

• Host detail panel's Containers tab was leaking cross-tenant data — it listed every tenant's container names/owners for a host to any authenticated user, unlike every other container-listing endpoint. Now scoped consistently with the rest of the app.

Bug Fixes

• Container console resize (drag-to-resize, fullscreen) now reaches the backing terminal process, not just the on-screen grid — full-screen terminal apps render at the correct size again.
• Dragging a container folder onto a different host is now rejected with an error instead of silently snapping back to its original host.
• Fullscreen container console is no longer clamped short of the full viewport height.

New Features

• Traffic Control — bandwidth/QoS management — three layers: Settings → Traffic Control rate caps for local/NFS/cloud backups and storage migrations, per-VM NIC Inbound/Outbound Mbps limits enforced via libvirt, and a Ceph mClock QoS profile selector (balanced / high client ops / high recovery ops).
• CRS Predictive Trigger (ML) — an XGBoost-based load forecaster layered on top of the existing reactive cluster rebalancer, retraining every 6 hours with a baseline-MAE comparison so you can see it's actually beating a naive forecast, plus a live predictions view.
• Hardware Lifecycle Manager: out-of-band Redfish/iDRAC telemetry — Network link health, Storage/RAID + physical-disk health, and DIMM slot population, read straight from the BMC so it still answers even when the host OS itself is down.
• Networks tab rebuilt as a browsable tree — L2 Bridges, Portgroups, Cluster VLAN Fabric, OVN Logical Switches/Routers/Chassis, Security Groups, and IP Pools now open in the same tabbed detail pane as VMs and hosts, instead of sitting in unclickable flat tables.
• Virtual SCSI controller management (VirtIO SCSI, LSI Logic Parallel, LSI Logic SAS, BusLogic — multiple per VM) and a VM Settings redesign into collapsible per-device rows.
• Configuration Parameters — raw QEMU launch-arg passthrough for advanced tuning, gated behind a dedicated permission.
• Selectable vNIC link speed (1/10/25 Gbps), so a guest reports a real speed instead of virtio-net's default "Unknown!".
• LXD container parity — Monitor/Logs tabs, a resizable/fullscreen console, and host-scoped container folders with the same drag-and-drop UX the VM folder tree already has.

UI Improvements

• VM and Host Monitor tabs redesigned into multi-panel dashboards (stat tiles plus CPU/RAM/Network/Disk charts); the standalone Performance page was retired in favor of the same range picker built directly into both.
• Host detail panel gained VMs and Monitor tabs inline; HA Cluster status moved off a floating overlay onto the Global Summary tab; alarms moved into the Recent Tasks drawer, replacing a separate floating bell.
• Two-tab Tasks / Cluster log view on the Recent Tasks drawer, backed by a new audit-log table — the console's first daemon-level audit trail, not just per-VM task history.

Security Fixes

• Fixed OVN security-group ACL rules matching the wrong port field and silently enforcing nothing — Security Groups now block traffic as configured.
• Added HA failover split-brain protection so two overlapping monitor cycles can't both fence the same node and restart its VMs twice.
• iDRAC/BMC credentials are now encrypted at rest, and Redfish/iDRAC connections use TLS certificate pinning, matching the console's existing SSH host-key pinning.
• Closed a login-audit-log IP-spoofing gap on the direct (non-proxied) port.

Bug Fixes

• Celery's scheduler was running duplicated across every HA node instead of the primary only, firing every scheduled task 2-3x concurrently — fixed with a durable primary-only guard.
• The Ceph dashboard now survives its seed monitor node going down, self-healing to whichever peer is actually active.
• VM live-migration now warns immediately instead of silently doing nothing when there's no eligible target host.
• Disk/NIC/SCSI-controller hot-attach no longer requires powering off a running VM first; CD/DVD eject/attach now reaches a running guest immediately instead of waiting for its next reboot.
• The in-console AI chat assistant no longer hangs forever on a backend error.

New Features

• GPU-as-a-Service, passthrough v1 — live per-host GPU inventory scan plus one-click VM GPU assignment, with a 1-GPU-1-VM availability check so the same card can't be double-assigned. Whole-GPU passthrough only for v1; SR-IOV slicing is planned as a fast-follow.
• Hard/soft VM placement affinity rules — any affinity group can now be flipped from the existing soft anti-affinity spread into hard "always share a host" / "never share a host" enforcement, managed from a new table on the IAM dashboard.
• A fifth per-tenant quota ceiling for node-count footprint, alongside the existing vCPU / RAM / Kubernetes-cluster limits.
• ZFS: child datasets, a dedicated "ZFS Local Storage" panel (Pools / Import / Replication), and ARC memory usage telemetry on the datastore summary.
• ~17 new App Catalog appliances, including Django, Drupal 7/9, BookStack, Ansible, and Canvas LMS.

UI Improvements

• Deploy Virtual Machine rebuilt as a step wizard — VM Identity → Source & Placement → Storage → Hardware → Provisioning → Network → Ready to Complete — replacing one long scrolling form, and no longer requires a password or SSH key to submit.
• Container Summary panel brought to parity with the VM Summary panel — Related Objects, a live NIC hardware list, snapshot count, a new tag editor, notes, and live IPs read straight from LXD instance state.
• Related Objects / Snapshots / Tags / Notes cards added to the VM Summary panel, now the primary way to inspect a VM.
• Removed the flat "Virtual Machines" and "LXD/Incus Containers" tables — the sidebar tree already lists everything, and clicking a leaf opens an inline Summary panel instead. Import VM (V2V) and Scan VM Storage moved into the header's Actions dropdown.
• Ceph modal rebuilt as a 5-tab panel instead of 9 stacked sections; Networks page toolbar reduced from 9 buttons to 7; DRBD storage tab moved to the same right-click context-menu pattern already used by ZFS pools and NFS datastores.
• VM list "vCPU" column renamed to "CPU"; Memory column now shows allocated GB instead of just usage percentage.

Performance

• All static assets (~60 JS/CSS files) now serve with a one-year immutable cache header instead of being re-validated on every page load.
• ZFS / Ceph live log streaming no longer runs blocking work directly on the server's async event loop — moved to a background thread so one log stream connecting can't stall every other request.

Security Fixes

• Closed a cross-tenant VM deletion path — an unscoped internal API-key fallback let any authenticated user delete another tenant's VM inventory record.
• Closed a missing ownership check on VM-to-template conversion that let a VM-manage user convert another tenant's VM into a template.

Bug Fixes

• VM-to-OVN network wiring was silently incomplete even with a correctly paired control plane — tap interfaces are now bound with a stable logical switch port identity, and an ACL-reapply race condition was fixed with proper locking. Verified live end-to-end (VM-to-VM traffic, Security Group enforcement).
• Ceph RBD-backed VMs could not live-migrate ("Migration without shared storage is unsafe").
• VM list state badges and HA tags rendered as unstyled plain text; the Memory column always showed 0%.
• Closing the detail panel by clicking its backdrop skipped cleanup, leaving WebSocket connections (ZFS Live Log, Ceph cluster stream) open.
• Proxmox V2V connection form didn't trim the API token secret.

Fixed

• Patched CVE-2026-69247 (CVSS 8.2, High) in the bundled Python cryptography package, found by Docker Scout shortly after the previous release published — updated within hours of disclosure.

Fixed

• OVN networking was non-functional for live traffic fleet-wide — root-caused to a version mismatch between the OVN control plane and its per-host chassis agents, which could silently misroute a legitimate ARP-responder reply back into the packet pipeline forever, dropping all traffic between VMs on the same logical network. Verified fixed against real VM-to-VM traffic on the same host and across separate hosts. Security Group ACL enforcement — previously never validated against live traffic — is now confirmed working end-to-end (rule applied → traffic blocked; rule removed → traffic restored).
• A Security Group ACL removal could silently leave a stale rule in place when two VMs' group-membership changes landed close together, with no error shown anywhere. ACL updates are now serialized per network.

Added

• OVN chassis bootstrap can now pin an exact release instead of always installing whatever's newest available, preventing the version-mismatch issue above from silently recurring on a newly added host.
• A new background sweep automatically self-heals any Security Group ACL drift, closing a gap where a missed update (e.g. an app restart at the wrong moment) could leave a VM's rules out of sync indefinitely.

Fixed

• Node-side worker crash-looped with ModuleNotFoundError: core.zfs_node_setup on every EL9 (Rocky 9) compute node — the node-image build script had its own hardcoded module list that had drifted from the one actually used to push code to nodes, so the module never shipped in the EL9 bundle. Fixed, with cross-reference comments added so the two lists don't silently drift apart again.

Fixed

• A single-server console could refuse to start after rotating any secret (e.g. re-running the installer), requiring a manual database fix to unblock. The cluster-consistency check now recognizes when no HA peers are registered and self-heals instead of refusing to start; the strict check is unchanged for real multi-node clusters.

Fixed

• Single-server installs could fail to recreate the console container with a "no such host" error after a reboot or manual update — the image location wasn't being persisted, so it fell back to an old, now-dead registry.
• The "Update Now" sidecar introduced in 0.42.8.8 could crash-loop on current Docker Engine versions.

Added

• "Update Now" button on the Docker Hub update banner — pull and install a new release directly from the console instead of just linking out. Super Admin only, and scoped to just the app/worker/beat containers.

Fixed

• Single-server (all-in-one) installs could end up with an unrestricted VNC console exposed on the public interface, or a broken one, depending on the host's firewall state — three compounding provisioning bugs, now fixed.
• LXD install could fail on EL9 nodes with an Unable to find a match: snapd error.
• A manually-entered control-plane IP during setup is now validated against the host's actual interfaces instead of accepted unchecked.

Fixed

• Security: several list views (Ceph, LINSTOR/DRBD, NFS, ZFS, network, and LVM pool names) were vulnerable to XSS via attribute-breakout in names read from external systems. Escaping is now correct everywhere these names render.
• Ceph pool creation could silently revoke a live VM's RBD access, causing it to fail its next boot with a silent permission error.
• Alarms: a dead Celery worker could vanish from the dashboard after 24h, even on a still-active node.
• Host repair: a Redis password containing &, |, or \ could corrupt the worker env file during auto-repair.

Fixed

• The Emergency Host UI (:9443) was unreachable on every enrolled compute node — node firewall setup opened the wrong port (LXD's cluster API port instead of the Emergency UI's). The service itself was always running fine; the firewall just silently dropped every inbound connection. Existing nodes need a Repair Worker run to pick up the corrected rule.

Fixed

• The "update available" banner never fired — the background check was still querying the console's old Docker Hub repo name, so it silently 404'd every cycle and Super Admins were never notified when a newer image was out.

Fixed

• VM Settings: Q35↔i440FX machine switch could fail and roll back the whole settings update.
• VM Settings: UEFI↔BIOS conversion could fail after toggling Secure Boot off.
• VM Settings: BIOS→UEFI conversion could fail on VMs deployed without ACPI.
• VM Settings: a contradictory Secure Boot + firmware combination could be submitted from the dialog, producing a confusing error.
• VM Settings: UEFI VMs could be misreported as BIOS in the dialog.

Added

• LVM-Thin local storage backend — a thin-provisioned, snapshot-capable option alongside ZFS/Ceph/DRBD/NFS, with a matching Storage tab and VM deploy option.

Fixed

• Rate limiter lost its Redis connection after Redis auth was enabled, returning server errors on login and other rate-limited requests.

Fixed

• Console (noVNC) could stop working for a VM after a live or cold migration — the destination host now self-heals the display configuration on every move.

Fixed

• A container network misconfiguration could silently bypass its security-group ACLs while still reporting as protected — now rejected outright with a clear error.
• Placement and LXD NIC-management hardening.

Added

• The Summary tab now opens on a real monitoring-style overview — host availability donuts, a per-host health honeycomb, alert summary, and top-VMs-by-usage rankings.
• Datastores got a matching Overview tab (capacity, IOPS, throughput, utilization).