The Alarm Has a Run Length

A dashboard alarm is usually written as a binary event:

drift detected

That sentence hides the real object. A drift detector is not a verdict. It is a stopping rule:

watch x_1, x_2, ...
stop the first time accumulated evidence crosses a boundary

The boundary is not a technicality. It is the price of sleep.

If the boundary is low, the detector reacts quickly when the world changes. It also raises more false alarms when the world has not changed. If the boundary is high, the detector becomes calmer, but real changes spend more time undetected. Sequential detection lives in that trade: average run length to false alarm versus detection delay.

This is why I distrust drift dashboards that report only the last alarm time. The missing question is:

How long would this detector run under no change before it cried wolf?

Page’s Accumulator

E. S. Page’s 1954 continuous inspection scheme introduced the cumulative-sum idea for detecting a process shift while observations arrive one at a time.¹ For a known pre-change density \(p_0\) and post-change density \(p_1\), the natural increment is the log-likelihood ratio

\[\ell_t = \log \frac{p_1(x_t)}{p_0(x_t)}.\]

The one-sided CUSUM statistic is

\[G_t = \max(0, G_{t-1} + \ell_t), \qquad G_0 = 0.\]

The alarm time is

\[\tau_h = \inf \{t : G_t \ge h\}.\]

That reset to zero is the important emotional move. If recent evidence has been against the change, the detector forgives the past and starts a fresh run. It is looking for a suffix of the stream whose cumulative evidence is unusually large.

For a Gaussian mean increase with known variance,

\[p_0 = N(0,\sigma^2), \qquad p_1 = N(\delta\sigma,\sigma^2),\]

the log-likelihood increment is, up to the same units used in the lab,

\[\ell_t = \delta \left(\frac{x_t}{\sigma} - \frac{\delta}{2}\right).\]

When the mean is still zero, those increments have negative drift. When the mean really shifts by about \(\delta\sigma\), they acquire positive drift. The detector is a random walk with a reflecting floor and an alarm ceiling.

Lorden later formulated quickest detection as an optimal-stopping problem and used average run length to false alarm and worst-case detection delay as central objects.² Moustakides proved Page’s stopping time optimal in a precise minimax sense for detecting distribution changes.³ The practical lesson is smaller but sharp: a CUSUM chart is not “sensitive” or “insensitive” in isolation. It is tuned against a false-alarm clock.

Lab: Spend the False-Alarm Budget

The lab simulates a 240-tick stream. The default stream has a one-time upward mean shift at tick 96. Two online detectors watch the same observations:

CUSUM. Uses a Gaussian mean-shift log-likelihood ratio with a target shift you choose.

Page-Hinkley. Tracks cumulative deviations from the running mean with a small slack term. It is often used as a lightweight mean-drift detector in data streams, but its threshold still needs calibration. Hinkley’s 1970 change-point work is a useful companion because it treats inference about where a distribution changed, not only whether an alarm should be raised online.⁴

The frontier plot sweeps thresholds over matched Monte Carlo streams. The x-axis is the no-change false-alarm rate within the 240-tick horizon. The y-axis is average delay after the change. Good detectors live down and left, but no point gets there for free.

Scenario

abrupt mean shift

abrupt mean shift slow drift variance burst no change Shift size

1.10

Noise scale

1

CUSUM target

1

PH slack

0.04

Threshold

8.0

Seed

29

observations true change/mean CUSUM Page-Hinkley

The Null Clock Matters

Average run length is easy to ignore because it sounds like a factory-floor quality-control relic. It is not. It is the monitoring version of a false positive rate.

Suppose a detector alarms 1% of the time in a 240-sample no-change simulation. That sounds small. Now run 5,000 independent metrics, model slices, customer segments, or symbols every hour. A “small” false-alarm probability becomes an operations policy.

This is the same reason ordinary p-values become dangerous under peeking. A single fixed-time test asks one question. A sequential detector asks a question after every new observation. The threshold has to pay for that repeated chance to stop.

The right unit is not only “did it alarm on this historical incident?” It is:

• average run length to false alarm under plausible no-change streams,
• delay distribution after known or simulated changes,
• pre-change alarm rate when the change time is known,
• miss rate within the response horizon,
• and the cost of the action triggered by the alarm.

Page-Hinkley is a good example. It is attractive because it is light: update a running mean, accumulate deviations, compare to a threshold. It appears in concept-drift tooling because many production problems do not give you a clean pre-change and post-change likelihood ratio.⁵ But the lab shows the same law: the threshold and slack parameter determine how often stationarity looks like a story.

The Target Shift Is A Model

CUSUM is beautifully sharp when the post-change alternative is well chosen. That is also its limitation.

If the detector is tuned for a one-sigma mean increase and the real shift is half a sigma, the log-likelihood increments have weaker drift than promised. Detection slows down. If the real event is a variance burst rather than a mean shift, the mean-shift CUSUM is not testing the right alternative at all. It may still alarm because a high-variance stream produces large positive excursions, but that alarm is not evidence for the modeled mean shift.

This is where ML monitoring often gets confused. “Distribution drift” is not one phenomenon. Gama and coauthors’ concept-drift survey distinguishes changes in the relation between inputs and targets from other changes in the data stream and stresses that adaptation/evaluation have to be matched to the learning problem.⁵ A Page-Hinkley alarm on loss, a KS test on a feature, an ADWIN window shrink, and a CUSUM on residuals are different instruments.

If you want to monitor a model, decide what has permission to move:

input distribution?
label distribution?
conditional label distribution?
calibration?
loss?
latency?
market regime?
sensor noise?

Then give the detector a null clock for that object.

Why This Old Tool Still Feels Modern

Sequential change detection is old enough to have beautiful theorems and annoying vocabulary. It also keeps reappearing under new names because streaming systems keep asking the same question:

Should I keep trusting the old state?

A recommender watches click-through residuals. A fraud model watches chargeback loss. A pricing system watches fill quality. A risk process watches spread residuals. An LLM product watches refusal rate, latency, or judge score. None of these systems wants to wait for a monthly batch report. None can afford to page humans every time random variation looks dramatic for five minutes.

That is the small discipline CUSUM teaches:

1. Write down what evidence would accumulate under the change you care about.
2. Reset evidence when recent data argues against that change.
3. Calibrate the boundary under no change.
4. Report detection delay and false-alarm run length together.

The alarm is not the result. The stopping rule is the result.

Implementation Notes

The browser lab is deterministic for a given seed. It exports SequentialChangeLab for Node and browser checks. Its audit verifies:

CUSUM recurrence equals cumulative log-likelihood minus its running minimum
raising a threshold cannot move an alarm earlier on a fixed evidence path
Page-Hinkley statistics stay nonnegative by construction
a high threshold does not alarm on a fixed stationary stream
Monte Carlo false-alarm and detection rates stay in probability bounds

The simulation is intentionally small, not a production calibration engine. A real detector needs longer null runs, dependence-aware simulations, multiple testing control across monitored slices, reset policy, and an action policy for what happens after the alarm.

1. E. S. Page, “Continuous Inspection Schemes”, Biometrika 41(1-2):100-115, 1954. ↩

2. Gary Lorden, “Procedures for reacting to a change in distribution”, Annals of Mathematical Statistics 42(6):1897-1908, 1971. ↩

3. George V. Moustakides, “Optimal Stopping Times for Detecting Changes in Distributions”, Annals of Statistics 14(4):1379-1387, 1986. ↩

4. David V. Hinkley, “Inference about the change-point in a sequence of random variables”, Biometrika 57(1):1-17, 1970. ↩

5. Joao Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia, “A Survey on Concept Drift Adaptation”, ACM Computing Surveys 46(4), 2014. Bifet and Gavalda’s ADWIN work is another useful streaming reference: “Learning from Time-Changing Data with Adaptive Windowing”, SDM 2007. ↩ ↩²