DetectionWhat MonitOps watches and when it alertsWorked example · INC-20931

Detect what users feel, not just what servers report.

MonitOps learns the normal shape of every service and journey and flags departures from it: error rates, latency, failed synthetic checks, provider status changes. It also watches for silence. When a feed stops sending, that is raised as its own problem, so missing data never looks like a healthy service.

Signals

Every signal has a kind, a unit and a cadence.

A signal is a measured series for an entity, such as the error rate of order-api. The signal catalogue is the list of signals MonitOps knows, with each one’s kind, unit, expected cadence and recovery rule.

Signal	Kind	Unit	Data from
error_rate	rate	Per interval	Log records
log_rate	rate	Per interval	Log records
latency_ms	rate	Milliseconds	The check duration
success	state	1 or 0	The check result
provider_status	state	Named states	The provider’s status feed
limit_remaining_pct	state	Percent	The provider’s limit report
freshness_s	state	Seconds	Since the last acquisition or heartbeat
Provider event log files	lagging	Arrive after the event	Evidence on the incident, never an alert

Fig. 1 — Standard signals. Lagging signals never raise an alert; they are attached to an incident as evidence.

Alarm definitions

An alarm definition says what to watch and when it counts.

An alarm definition is a rule that evaluates a signal. Each evaluation that fires becomes an alert and records which version of the definition produced it.

Threshold: A signal crosses a fixed limit, or takes a given state value, for a configured number of consecutive evaluations.
Baseline: A signal departs from the statistical band learned from its history.
Probe failure: Synthetic checks against a customer journey or management endpoint fail.
Provider status: A provider reports degraded or down; the alert severity follows the provider’s own level.
Source silence: A source misses its feed deadline. Missing data never looks like a healthy service.
Operational event: A provider event on a configured list of actionable event definitions occurs.
Predicted breach: A forecast crosses a limit within its horizon.

Fig. 2 — The seven condition kinds. A definition is versioned on every change, and each evaluation records the version that produced it.

Baselines

The expected range comes from the signal’s own last seven days.

For rate signals such as error rate, MonitOps compares each new value with the values the same signal took at the same time of day over the previous seven days, by default. The comparison uses the median of that history and its spread, so a few earlier spikes do not widen the expected range.

checkout-web error rate

Last 30 min · vs 7-day baseline

checkout-web error rateBaseline (p5–p95)alertAnomaly

Sustained: The value stays outside the expected range across consecutive evaluations.
Transient: A single evaluation departs by a larger margin than a sustained departure requires.
Recurring: The same entity and signal depart repeatedly within a day.

Direction

Only a value above the expected range is anomalous. A latency signal set to upper does not alert when latency falls below its range.

Missing data: Not zero. An interval with no data is not counted as a low value, so a silent source does not look like a drop in errors.
Recovery: After two consecutive valid evaluations that meet the recovery threshold. Intervals with missing data do not count.
Late data: The affected interval is re-evaluated and stored as a revision. The original decision is kept.

Fig. 3 — checkout-web error rate against its baseline, 14:02:11 ICT. The kind of departure is recorded on the alert.

Silence

Silence is raised as its own problem.

A source that stops sending is raised as its own problem, so a quiet feed is never mistaken for a healthy one. Silence is detected from freshness against a deadline, not from the absence of an alert.

Source silent

Unknown

crm-sync logs · last record 14 min ago

Source: crm-sync logs
Last record: 13:48:02
Expected: Every 30 s
Silent when: Freshness reaches five times the expected interval, by default.
Cleared: On the next successful acquisition or heartbeat.
Idle stream: A stream with no activity guarantee is unknown, not failed.

Fig. 4 — Source silent. crm-sync logs, last record 13:48:02, expected every 30 s. Shown as unknown, never as healthy.

Synthetic checks

A check fails after two consecutive runs and recovers after two.

Synthetic checks are scheduled probes against customer journeys and management endpoints. Each check emits success and latency on every run, and a check that stops reporting is raised as silence, never counted as a pass.

Run	success	latency	Result	Check state
14:00	1	412 ms	Passed	—
14:01	1	398 ms	Passed	—
14:02	0	1860 ms	Failed	—
14:03	0	1904 ms	Failed	Failing
14:04	1	405 ms	Passed	—
14:05	1	391 ms	Passed	Recovered

Fig. 5 — Checkout journey check, one run a minute. Simulated data. Defaults: failing after two consecutive failed runs (major), recovered after two consecutive passes.

Operational events

Provider and platform events are matched against an allowlist.

MonitOps alerts on the events that are on a configured list of actionable definitions and keeps every other event as evidence. An allowlisted event is evaluated directly; it does not wait for the event rate to become anomalous.

Definition	Severity	Recovery	Evidence
provider.service-degraded	Minor	Recovery rule	Reopens the record it came from
provider.service-down	Major	Explicit	Reopens the record it came from
platform.maintenance-started	Warning	Operator resolves	Reopens the record it came from
provider.unlisted-event	No alert	None	Kept as evidence, raises nothing

Fig. 6 — Operational event allowlist. Illustrative definitions. Events are deduplicated by their source event identity, so a redelivered event is not counted twice.

Severity and priority

Severity and priority are shown without relying on colour.

Severity describes how serious a single alert or an incident is. Priority is the P1 to P4 ranking an incident carries for the people who respond to it. Each has its own label and shape, so neither is conveyed by colour alone.

Critical
Major
Minor
Warning
Indeterminate
Clear

P1
P2
P3
P4
MajorCleared

Impacted systems: The highest criticality among the systems the incident affects.
Journeys: Whether journeys flagged as customer-facing pass through the affected services.
Magnitude: How far the signals involved depart from their expected range.

Fig. 7 — Every severity has its own icon and label; every priority has its label, fill and bars. A cleared alert keeps the severity it had.

Predicted breaches

A predicted breach is an alert about the future.

A prediction is a forecast that a capacity or quota signal will cross a limit within a horizon, shown with an uncertainty band. It gives the team time to act before a limit is reached.

API usage

Forecast with uncertainty band

API usageForecastUncertainty bandAPI limit

PredictedBreach in 3 h 40 m · Would be major

Badge: The Predicted badge, with the severity it would have as plain text. It carries no severity badge.
Counting: Never counted as firing and never grouped with current failures.
Revisions: Each revision of the forecast is kept, so the history shows what MonitOps expected and when.
Outcomes: Recorded against the forecast, including the moment a limit is actually breached.

Fig. 8 — Provider limit forecast with its uncertainty band. The alert is raised before the limit is hit, and the severity it would have is plain text.

From alert to incident

Detection ends where correlation begins.

A firing evaluation produces an alert, and the alert follows the normal path into incident grouping. Alerts that share a time window and a path on the service map are grouped into one incident, with the facts that caused the grouping.

INC-20931P1MajorGrouped by MonitOps · 7

Same 90 s window · shared path checkout-web → order-api → payments-gw

Fig. 9 — INC-20931: seven alerts grouped by MonitOps. The raw alerts are always kept.

See MonitOps on your own incidents.

We run a working session with your operations team on your services and recent incidents.

Contact sales Log in