Skip to content

Notifications & Alarms

The console sends alerts to as many destinations as you need: chat webhooks (Discord, Slack, Mattermost, Rocket.Chat), Microsoft Teams, email, and a syslog server or SIEM. Each destination is a channel with its own event list and minimum severity. Alarms you've already seen can be acknowledged or muted, so they stop counting toward the badge and stop re-notifying.

Managing channels needs a Super Admin.

Add a channel

Settings → Notifications → Add channel.

  1. Name it (for example Ops chat).
  2. Pick the Type:
    • Webhook (Discord, Slack, generic): paste the webhook URL and pick the Message format: Discord, Slack, Mattermost, Rocket.Chat, or Full event as JSON for anything that wants structured data.
    • Microsoft Teams: paste the Teams incoming-webhook URL.
    • Email: see Email below.
    • Syslog / SIEM: see Syslog / SIEM below.
  3. Set the Minimum severity: Info and above (everything), Warning and above, or Critical only.
  4. Tick the Events the channel should get. With none ticked it gets every alert. VM lifecycle events (created, deleted, powered on, shut down, moved, backup finished) are only sent to channels that tick them, so a channel left on "everything" doesn't get a message for every power-on.
  5. Save channel, then Send test on its row. The Last result column shows Delivered, or Failed with the reason.

A channel's type can't be changed after it's saved. Untick Enabled to pause a channel without deleting it.

Existing webhook settings

The Discord/Slack and Teams webhook URLs from Platform Configuration appear here as two built-in channels and keep working unchanged. You can edit their events and severity, but not delete them.

Email

Field What to enter
Mail server / Port Your SMTP server, for example smtp.office365.com and 587.
Security STARTTLS (usually port 587), TLS (usually port 465), or None (plain relay) for an internal relay that accepts mail without either.
Username / Password Optional. Leave blank for a relay that doesn't need a login.
From address The sender the mail server will accept, for example [email protected].
Send to One or more addresses, separated by commas.

Email is the channel most people want only a few events on. Just the essentials ticks the set worth an inbox: failed VM and container backups, failed restore checks, the console's own nightly backup failing, hosts down, HA fencing and HA-degraded VMs, storage filling up, and workers that stopped responding.

If Send test fails, the delivery log shows the mail server's own reply (for example an authentication or relay-denied error).

Syslog / SIEM

Field What to enter
Syslog server Host name or IP of the collector.
Transport TLS (recommended, port 6514), TCP (port 514), or UDP (port 514, no delivery check).
Port Follows the transport unless you type your own.
Facility local0–local7, user, daemon, auth or authpriv.
CA certificate Only if a private CA signed the collector's certificate. Paste the PEM.

Messages are RFC 5424. Also send the audit trail (on by default) forwards every sign-in, admin change and task outcome from the cluster log, not just alerts. That gives you a copy of the audit trail the console's own admins can't edit.

Events you can subscribe to

Event Sent when
Host down, offline or unreachable A host's state changes, and again when it recovers.
VMs that would not fail over (HA-degraded) HA can't place a VM; resolved when it can.
HA fencing and failover A host was fenced and its VMs restarted elsewhere.
Container host needs manual recovery An LXD host failed in a way HA can't fix on its own.
Worker stopped responding A node's worker went silent.
Storage pool filling up (and resolved) A pool is forecast to fill soon.
Ceph health changed Ceph moved between OK, WARN and ERR.
Host CPU or RAM over its alert threshold A host crossed its configured threshold.
Host / VM usage anomaly Usage far outside its own normal range.
Benchmark regression A scheduled benchmark came in well below its baseline.
A VM or container backup failed Any local, cloud or container backup failed.
The console's own nightly backup failed The control-plane backup failed.
A restore check failed See Backup Protection.
Node update halted or finished See Update Compute Nodes.
A newer console release is available See Update a Console Without Internet Access.
VM deployment started A deploy began.
VM created / deleted / powered on / shut down or powered off / moved / backup finished Lifecycle events: only to channels that tick them.

Conditions such as a host going down or a pool filling up have to be seen twice before they notify, and send a resolved message when they end. A host that keeps flapping sends one notification, not one per flap.

Delivery and retries

Alerts are queued and sent in the background, so a slow or unreachable endpoint never holds anything else up. A failed send is retried with growing gaps (30 s, 1 min, 2 min, …) for about 15 minutes before it's marked failed. Recent deliveries, under the channel list, shows each message, its channel and event, the status, how many tries it took, and the error from the last try.

Sovereign mode

With sovereign mode on, a channel whose host isn't private or on the allowlist fails straight away with an error naming the setting.

Acknowledge and mute alarms

Open the Alarms tab in the Recent Tasks drawer. On each alarm:

  • Acknowledge records your name and the time on it (with an optional note). Everyone sees who acknowledged it.
  • Mute stops notifications for that alarm for 1 hour, 4 hours, 24 hours or 7 days. It stays in the list.
  • Acknowledge all acknowledges every open alarm at once.

Acknowledged and muted alarms stay listed, dimmed at the bottom, and stop counting toward the badge. Unacknowledge and Unmute undo them. Every acknowledgement and mute is written to the cluster log. Tenant Viewers can see alarms but can't acknowledge or mute them.

History (admins) lists past alarms for the last 24 hours, 7, 30 or 90 days, filterable by host or VM, with who acknowledged each one.

See also