AI Workers

AI Workers split a change into tasks and run one AI agent per task, each on its own VM. Every worker clones the repository, works on its task, can test in a sandbox, and sends back a patch. You review the patches and apply the ones you want. Nothing is pushed for you.

Use it for work that splits cleanly, for example "move these twelve routers to the new quota check, one per worker". Each worker has its own context and its own VM, so the tasks run side by side.

Each worker:

• runs Claude Code on a fresh clone, as an unprivileged user;
• never holds the Claude API key: every model call goes through the console, which adds the key and counts the cost against the job;
• gets an API key that can use sandboxes and its own job's gateway, nothing else, and that is deleted when the worker finishes;
• is deleted as soon as it has sent back its result.

AI Workers are off until an admin turns them on. They use the AI Assistant's backends, so the AI Assistant has to be set up first (Settings → AI Assistant).

1. Build the worker image

On any compute node, as root:

dnf install -y guestfs-tools        # once, for virt-customize
./deployment/ai_worker/build-ai-worker-image.sh

This downloads the Rocky 9 cloud image and writes /var/lib/libvirt/images/ai-worker-rocky9_tmplt.qcow2. It has the QEMU guest agent with command execution allowed (like the sandbox template), Claude Code, git, Python 3, Podman and build tools, the mf-worker runner and the mf-sandbox command. No password, key or token is baked in.

Put the image on every host in the worker cluster: publish it from Content Library, or copy the file to the same path on each host. A worker placed on a host without it fails with "Creating the worker VM failed: Template missing!".

2. Pick a network

Workers need to reach:

• the console, at the URL you give in step 3, on its HTTPS port (for the model gateway and to send results back);
• your Git server, to clone;
• the internet on HTTPS, if builds download packages.

Unlike the sandbox network, the worker network can't block the console. Block everything else: the hosts' management addresses, the database and other private ranges. Code the agent writes runs on the worker, so treat it as untrusted.

3. Turn AI Workers on

Settings → AI Workers (Super Admin):

Setting	What it does
Allow AI Workers jobs	Off by default. Can't be turned on until network, placement, worker image and console URL are set.
Network	The port group from step 2.
Placement	The cluster whose hosts run workers.
Worker image	ai-worker-rocky9_tmplt.qcow2 from step 1.
Console URL	How a worker VM reaches this console, e.g. https://192.168.7.16.
CA certificate	The CA that signed that URL's certificate, if it isn't publicly trusted. Workers use it for the gateway, results and mf-sandbox.
Git username / token	A read-only token for cloning. It reaches a worker only for its clone and is never written to its disk.
At once	Worker VMs alive at the same time, across all jobs (default 2). Others wait.
Tasks per job, max runtime	Caps on what a job may ask for.
Default / max budget	Dollar budget for a Claude job, and the most one job may set.
vCPU, RAM, disk	Size of each worker VM. The job owner's quota and tenant limits apply.
Boot timeout	How long a worker VM may take to answer through the guest agent before it counts as failed.

Turning AI Workers off stops new workers being created. Running ones finish.

4. Start a job

AI Workers → New job:

• Repository clone URL and branch, tag or commit.
• Backend: Claude (costs money per token, needs a budget) or a local model (no per-token cost, but it runs as many requests at once as it has slots, so workers may queue).
• Goal: what the whole job is for. Every worker sees it.
• Tasks: one per line. Each line becomes one worker. Split the work so workers don't edit the same files.
• Test command (optional): runs in each worker after its agent finishes, for example python3 -m pytest -q.

The job page shows each worker's state, cost and what it's doing. Click a finished worker to see its summary, patch (colored), test output and agent log. Download patch gives every finished worker's patch as one file:

git apply --3way ai-workers-job-12.patch

A job ends when every worker has finished, when it reaches its runtime, or when it has spent its budget or token limit. Cancel job stops it early. In every case the worker VMs are deleted and the patches already sent back are kept. Delete removes a finished job's patches and logs. What it spent stays in the usage records.

Budgets and cost

The console meters every model call a worker makes: input, output, cache reads and cache writes, priced per model. The gateway checks the job's spend before each call, so a job stops at its budget within one call. Claude spend also counts toward the AI Assistant's monthly limit, if one is set.

Agent sessions resend their conversation on every turn, so input tokens dominate. Start with a small budget (the default is $5 per job) and raise it once you've seen what real jobs cost.

Local models

A local backend works if its server serves the Anthropic Messages endpoint (/v1/messages), which recent llama.cpp and Ollama builds do. Whatever model name the agent asks for, the gateway sends the request to the model set for that backend under AI Assistant. If the server has no such endpoint, workers fail with a message saying so.

From an assistant (MCP)

mfcloud-mcp can start and follow jobs, so one assistant session can split a large change across workers:

Tool	What it does
propose_worker_job	Checks a job against the limits and returns a confirmation token. Starts nothing.
confirm_worker_job	Starts the proposed job. Creates VMs and, on Claude, spends money.
list_worker_jobs, get_worker_job	Follow jobs and their workers.
get_worker_patch	Each finished worker's summary, test result and patch.
cancel_worker_job	Stops a job at once.

The key the assistant uses needs the Full scope: a sandbox or read-only key can't start jobs.

What a worker can't do

• Push, merge or deploy. Its output is a patch for a person.
• Use the Claude API key or call the model except through the gateway.
• Reach any console route except sandboxes, its own gateway and its own result and progress routes.
• Outlive its job. Workers are deleted when the job ends, and their keys expire at the job's deadline even if something goes wrong.