Update Compute Nodes from a Release

A release bundle can carry the packages for the compute nodes as well as the console: a package repository for each Rocky/RHEL major version (el9, el10). The console serves that repository to its own nodes and installs it one host at a time, so the nodes never need internet access either.

Import the bundle first (Update a Console Without Internet Access), and update the console to that release before its nodes: the nodes take their worker code from the console.

What happens to each host

Settings → Node Updates, as a Super Admin. Pick the release, a cluster and the hosts, in the order you want them done.

1. Preflight. The host must be reachable, the release must have packages for its OS version, the other hosts must have room for its VMs, and nothing may run on it that maintenance mode can't move: VMs on local-only storage, VMs the console doesn't manage, or LXD containers. On a host that also runs Ceph OSDs, the Ceph cluster must be healthy (all OSDs up, all placement groups active+clean). The console records what runs on the host: its services, its storage modules, its kernel.
2. Maintenance. The host enters maintenance mode and its VMs move to the other hosts.
3. Install. dnf upgrade runs from the console's repository only. Packages keep their distribution signatures and dnf checks them. If dkms is installed, it rebuilds its modules for a new kernel. The worker code is synced.
4. Reboot, if the kernel or core libraries changed, or every time if you chose Every host. Before rebooting a Ceph OSD host the console sets Ceph's noout flag.
5. Health check. libvirt must answer, every service and module recorded in step 1 must be back, a test VM definition must work, the worker must check in with current code, and the host must run the newest installed kernel. On a Ceph host the console waits for all OSDs to be up and every placement group to be active+clean, then clears noout. The check is repeated for up to 15 minutes.
6. Back in service. The host leaves maintenance and takes VMs back, and the next host starts.

If a step fails, the host stays in maintenance and the run stops with the reason. Each host has a Log. Retry resumes at the failed step; Skip moves on without that host (take it out of maintenance yourself once it's fixed). Cancel run stops after the step in progress. A run carries on across a console restart.

Preview asks each selected host what it would install, without installing anything.

How nodes reach the repository

Nodes read the repository at https://mfcloud/node-repo/<release>/<el9|el10>/, the address they already use to report to the console. The console writes two files on each node it updates or previews:

• /etc/yum.repos.d/mfconsole-release.repo: root-only, disabled, used only by the update with dnf --repo mfconsole-release. It holds the login the repository asks for.
• /etc/pki/mfconsole/console-ca.pem: the console's certificate, so dnf can verify it.

Building a release with node packages (vendor)

On a representative node of each OS version, with internet access and the same repositories the fleet uses:

./make_node_repo.sh /root/node-repo                    # every installed package
./make_node_repo.sh /root/node-repo kernel-core qemu-kvm  # or a chosen set

Then pack it into the release with make_release_bundle.py <version> --node-repo <folder>. A node-only release (no console image) takes --no-console-image.

API

Call	Notes
GET /api/system/node-updates	releases with node packages, clusters and hosts, the last 10 runs
POST /api/system/node-updates/preview	{"release", "hosts": [...]}; audited
POST /api/system/node-updates/runs	{"release", "cluster", "hosts": [...], "reboot": "if-needed"\|"always"}; audited
POST /api/system/node-updates/runs/{id}/cancel	audited
POST /api/system/node-updates/runs/{id}/hosts/{ip}/retry	audited
POST /api/system/node-updates/runs/{id}/hosts/{ip}/skip	audited

All are Super Admin only. A halted or finished run sends a node.update notification.