Alarm definitions

An alarm definition is a rule that evaluates a signal. Each evaluation that fires becomes an alert and records which version of the definition produced it.

How it works#

Alarm definitions evaluate signals continuously. A definition does not open incidents or raise alarms in other systems directly: a firing evaluation produces an alert, and the alert follows the normal path into incident grouping (see Incidents).

Every definition has five parts.

Condition kinds#

A predicted breach is shown with the Predicted badge and the severity it would have. It is never counted as a current failure and never grouped with current failures.

Lagging signals, such as provider event log files that arrive after the event, never raise alerts. They are attached to an incident as evidence.

Versions#

A definition is versioned on every change. Each evaluation is stored with the definition identifier and version that produced it, so an alert can always be traced to the exact rule text that fired. Past evaluations keep the version that fired them; editing a definition changes later evaluations only.

Deleting a definition is a soft delete. History, evaluations and delivered alarms keep referencing the version they used.

Configuration#

Administrators create and edit alarm definitions in the console.

1. Open Alarm definitions and select a definition, or choose to create one.
2. Set the entity selector, for example the service order-api.
3. Choose the condition kind and its parameters, such as the direction and the number of consecutive evaluations.
4. Set the severity mapping.
5. Set the alarm identity.
6. Save. The save creates a new version.

Disabling a definition stops it raising new alerts. It still clears what it already raised, so a disabled definition never leaves an alert or a delivered alarm stuck open.

What you see in the console#

The evaluated chart for an alert shows what the detector saw, not a recalculation. It distinguishes three things:

• the expected band, learned from the signal's history;
• the firing threshold;
• the active episode, from the first firing evaluation to recovery.

The chart reads the stored evaluation values and bands. When late data arrives and an evaluation is recomputed, the chart shows a revision and keeps the original decision. Normal and pending evaluations are stored as well as firing ones.

The excerpt shows the order-api error rate leaving its expected band at 14:02:11 in incident INC-20931.

Limits and defaults#

• Rate anomalies are evaluated against a seven-day baseline by default; see Baselines.
• Recovery needs two consecutive valid evaluations within the configured recovery threshold. Probe and health checks recover after two successful checks. Source silence clears when the source next acquires data or sends a heartbeat. Intervals with missing data do not count toward recovery.
• A transition is published only when state changes, once per episode, so a signal that stays in breach does not produce a new alert on every evaluation.

Failure states#

Related#

• Baselines
• Severity
• Incidents
• Destinations