Alerting

Alert rules — shown as health checks in the app — watch your telemetry and notify you over email, Slack, webhook, and PagerDuty when something slips. Nearly everything alerting lives on the Alerts page, in four tabs (the exception is flow-completion rules, which are configured on the integration they watch):

• Health checks — every rule, firing ones on top, with ack/resolve, on/off toggle, and search.
• Notification channels — the delivery targets and notification profiles.
• Sent notifications — delivery history, filterable by service, integration, system, or check.
• Maintenance — active and scheduled maintenance windows.

Rule types

Every rule has a name, a severity (info, warning, or critical), and a condition over a trailing time window. There are five kinds:

Rule type	Fires when
Metric check	An aggregated metric value crosses a threshold
Log check	The count of matching log lines is too high — or too low
Failed traces	Too many traces fail over the window
Response time	p95 (or max) latency crosses a threshold
Traffic volume	Fewer traces than expected arrive — a dead-man’s switch

Metric checks watch any metric you ingest. You pick the metric, an aggregation, a comparison (>, ≥, <, ≤, =, ≠), a threshold, and a window (1 minute to 24 hours). Optional attribute filters scope the check (say region=eu-north), and split by attribute turns one rule into an independent check per attribute value — the alert fires when any value breaches, and lists which. The aggregations:

Aggregation	Meaning
last	Most recent value — current queue depth
min / max / avg / sum / p95	The usual reductions over the window
increase	Counter delta over the window — “exceptions per 5m”
rate	increase per second
age	Now minus the value, for timestamp metrics — “file untouched for over an hour”

A metric check normally aggregates ingested telemetry, but can instead watch a pushed value that an external scraper POSTs in — useful for numbers that never pass through OpenTelemetry.

Log checks count log lines matching a severity floor, an optional case-insensitive body substring, and optional attribute filters, over a window of 1 minute to 30 days. The comparison runs both ways: at least N matches (an error flood) or fewer than N (an expected heartbeat line that stopped appearing).

Failed traces, response time, and traffic volume checks bind to a service or an integration. Failed-trace checks fire on the count of traces with error status (optionally narrowed by span attributes); response-time checks compare p95 — or max, for worst-case — latency against a millisecond threshold; traffic-volume checks fire when fewer traces than the threshold arrive, including none at all — the dead-man’s switch for a flow that should always have traffic.

You create rules in context: the alert icon in the Metrics explorer, the Logs page, or an integration’s Errors tab — and manage them all from Alerts → Health checks.

Alert lifecycle

Rules are evaluated continuously (every 30 seconds by default). An alert is either firing or resolved, and notifications follow the state:

• Fires — the first breach opens the alert and sends the firing notification.
• Re-notifies — while it stays firing and unacknowledged, the owning notification profile’s re-notify interval paces reminders; an interval of zero means notify once.
• Acknowledge — clicking Ack on the Alerts page stops notifications for that alert. It stays firing until the condition actually clears — acking silences the paging, not the problem.
• Resolves — how depends on the rule’s resolve mode:
  • Auto (the default for metric checks): the alert resolves itself as soon as the condition clears, and a resolved notification goes out.
  • Manual (the default for log and trace checks): the alert stays firing until someone acknowledges it, even if the condition clears — these signals are spiky, and a count that dips back under the threshold doesn’t mean anyone has looked at it. Once acked, it resolves when the condition is clear.

A resolved notification is only sent for alerts that notified when they fired — an alert suppressed by a maintenance window, or already acknowledged, stays quiet on resolve too.

Flow-completion rules (stuck messages)

One more rule kind lives outside the Alerts page: flow-completion rules, configured on an integration under its settings’ Trace completion section. Where health checks watch aggregates, a flow-completion rule watches each message individually against the pipeline it is supposed to walk:

order-received ─(within 60s)─▶ order-validated ─(within 300s)─▶ order-shipped

A rule names a start span that gates it — only traces containing that span are evaluated — and an ordered list of stages: each stage is one or more span names that satisfy the hop, with a timeout (1 second to 24 hours) for the trace to reach it. The rule carries its own severity and its own channel picks.

When a trace stalls — the next stage’s span hasn’t arrived within its timeout — it becomes a stuck message. The rule fires through the same notification pipeline as every other alert (channels, routing, delivery), once per trace and stage, and the trace is marked delayed wherever it appears. Stuck messages are sticky:

• If the missing span eventually arrives, the firing auto-resolves as delivered with delay.
• Otherwise it stays firing until someone marks it handled — it then stops counting against the integration and won’t re-fire for the same delay.

Stuck messages surface on the integration’s Errors tab (delayed traces), in its Messages tab behind the delayed filter, and as a delayed count on the integration page itself.

The rule is the alert

The stuck count isn’t exposed as a metric, so a metric health check can’t watch it. To get paged about stuck messages, put channels on the flow-completion rule itself.

Notification channels

Channels are the delivery targets. They are org-wide, managed on Alerts → Notification channels (with a test-send button per channel):

Channel	You provide
Email	Recipient list; optionally per-channel SMTP settings, otherwise the cell’s SMTP (Settings → System → Email) is used
Slack	An incoming webhook URL
Webhook	An HTTP(S) URL; optionally a signing secret and the CloudEvents format
PagerDuty	An Events API v2 routing key; optionally the EU events endpoint

Notification profiles and routing

A notification profile bundles a channel selection with delivery behaviour — the re-notify interval, and a grouping mode (with per-integration grouping, one representative alert carries an integration’s notifications instead of every check paging separately). Profiles exist per team and at the org level, managed on the same Notification channels tab.

When a rule fires, Sluicio resolves where to deliver most-specific-first:

1. The rule’s explicitly picked channels, if it has any.
2. The integration’s assigned profile, if the rule is bound to an integration that has one.
3. The owning team’s default profile.
4. The org default profile.

The message format resolves the same way but independently — see Notification templates for the per-field template ladder. A rule’s Notification content section also has toggles for which blocks a notification includes: service and integration info, their metadata, and the failing check’s details.

Webhook and PagerDuty payloads

Webhook and PagerDuty deliveries are a fixed contract — consumers parse them, so they are deliberately not templatable. A webhook POST carries JSON like:

{
  "state": "firing",
  "severity": "critical",
  "summary": "Orders backlog: queue.depth 12500 > 10000 over 5m",
  "rule": { "name": "Orders backlog", "signal": "metric", "kind": "" },
  "link": "https://sluicio.example.com/alerts?instance=8f2c…",
  "sent_at": "2026-07-24T12:34:56Z",
  "source": "sluicio",
  "check": { "name": "Orders backlog", "metric": "queue.depth", "value": "12500", "threshold": "10000", "window": "5m" },
  "service": { "name": "orders", "status": "unhealthy", "metadata": { "runbook_url": "https://…" } }
}

The check, service, and integration blocks appear according to the rule’s Notification content toggles.

Since v0.11.27, webhook channels have two opt-ins:

• CloudEvents 1.0 format — the payload above is wrapped in a CloudEvents envelope (type is com.sluicio.alert.fired or com.sluicio.alert.resolved) and posted as application/cloudevents+json.
• Request signing — with a secret set, every POST carries X-Sluicio-Timestamp (Unix seconds) and X-Sluicio-Signature (sha256=<hex>, the HMAC-SHA256 of <timestamp>.<body> under the secret). Verify both to reject tampered or replayed deliveries.

Webhook channels do double duty: the same channels — same format and signing settings — are also the destinations for event subscriptions, which push Sluicio’s domain events (alerts, service discoveries, configuration changes) to your own platform.

PagerDuty deliveries use the Events API v2: a firing alert sends trigger, resolution sends resolve, with a stable per-rule dedup key so flapping updates one PagerDuty incident instead of opening a new one each time. The event’s severity mirrors the rule’s.

Maintenance windows

Since v0.11.4, planned work doesn’t have to page anyone. A maintenance window (Alerts → Maintenance) is scheduled with a name, a reason, a time range, and a scope: the whole organization, a team’s rules, or a set of integrations, systems, and services.

While a window is active, matching alerts still evaluate and record — they show on the Alerts page marked as suppressed — but notifications are not sent. Suppressed alerts stay quiet on resolve too (no “resolved” for a firing you never saw), and if the window ends while the condition still breaches, the alert pages you then. A window can also publish an announcement for its duration, and can be ended early.

Windows are the mute button

There is no per-channel mute or per-alert snooze. To silence a rule indefinitely, toggle it off on the Health checks tab; to silence a planned period, schedule a maintenance window.

Windows cover flow-completion rules too: a message that goes stuck during a window is still recorded on the integration’s delayed surfaces, but doesn’t page — and if it’s still stuck when the window ends, it pages then, like any other alert.

Before v0.11.39

On older versions, flow-completion firings paged straight through a maintenance window. If you’re not yet on v0.11.39, disable a pipeline’s flow-completion rule for the duration of planned work that will stall it.

Permissions

• Notification channels and profiles are org-global configuration — managing them takes an org editor.
• Alert rules take an org editor — or, on Enterprise with advanced RBAC, a team-scoped editor for rules bound to services within their scope.
• Acknowledging an alert only requires being able to see the service it fired for.
• Org-wide maintenance windows take an org admin; narrower scopes an editor.