Change-Point Detector Comparison on SPX Realized Volatility

A pre-registered bake-off of three named change-point detectors — HMM, CUSUM, and PELT — against a fixed-threshold baseline on the same SPX log realized-volatility series, scored under one event-matching rule fixed in advance of computing any metric.

Detector comparisons are usually written by the people who built one of the detectors, and the matching rule is most often chosen after looking at the output. The result here goes the other way. The same standardised SPX log realized-volatility series was fed to HMM, CUSUM, PELT, and a fixed-threshold baseline, the event-matching rule was committed in advance with a SHA-256 fingerprint, and the headline metric pair — median latency together with false alarms per year — was specified before metrics were computed.

Under that protocol the fixed-threshold baseline dominates. It posts F1 = 0.438 against the next-best 0.297, hits one third of events (16 / 48), and produces 0.55 false alarms per year against PELT's 2.02, CUSUM's 11.07, and HMM's 56.81. CUSUM and HMM purchase high recall with hundreds of false alarms — HMM emits 968 alarms across 4,119 trading days at precision 0.04 and is essentially noise under this rule. PELT, the only named detector chosen by hyperparameter search, still produces nearly four times the baseline's false-alarm rate for one-quarter of its hits. The contribution of this work is therefore not a new detector but the bake-off harness itself, a 4,119-row labelled regime-break series consumed downstream, and a list of conditions under which any of the three named detectors might be expected to beat the baseline out-of-sample.

The pipeline builds a single daily log realized-volatility series, feeds it identically to all four detectors, and scores their alarms under one matching rule.

Target. Daily log realized volatility on SPX, standardised against full-sample mean and standard deviation before being passed to any detector. Detectors. Two-state Gaussian HMM (Viterbi-decoded, EM, seed 1729); two-sided CUSUM with bootstrap-calibrated threshold; PELT with L2 cost and penalty selected by elbow plus held-out validation; fixed-threshold baseline at z > 1.75 with cross-up trigger. Events. 47 VIX-stress episodes (VIXCLS > 30 days clustered into a single episode dated by the first day) plus 1 NBER recession start (the 2020-03-02 COVID start) — 48 events total. Matching rule. An alarm matches an event when |τ̂ − τ| ≤ 10 trading days; nearest wins, each alarm matches one event, each event matches one alarm. Reproducibility. Configuration hash, detector-metrics checksum, data-vintage checksum, and the matching-rule fingerprint (SHA-256 prefix 22b80a4a) are pinned in the run manifest, so any edit to the protocol after the fact would be detectable.

The intraday realized-variance estimator is RV_t = Σ_j r²_t,j over 5-minute bars, with a fallback to squared daily log return r²_t on any day with fewer than 60 intraday bars. The modelled target is z_t = log(max(RV_t, ε)) with ε = 10⁻¹² to prevent −∞ on flat days.

Caveat: the fallback dominates. The free intraday feed returns at most roughly 60 calendar days of 5-minute bars at any one time, so over the 2010–2026 window only 40 trading days — the last two months of the sample — are built from intraday RV; the other 4,079 days (99.0%) are squared daily returns. Every comparison on this page should be read as a comparison on log squared daily return, not on intraday RV. Replacing the intraday feed with one that has long historical coverage is the single largest follow-up. VIX daily closes (FRED VIXCLS) supply the stress labels; the NBER recession indicator (FRED USREC) supplies the recession-start labels.

All four detectors consume the same standardised z_t. Their configured hyperparameters and per-run diagnostics are persisted alongside the metrics, so any number on this page is reproducible from the run manifest.

HMM. Two-state Gaussian HMM with diagonal covariance, fit by EM (n_iter = 250, seed 1729) and decoded by Viterbi. States are relabelled post-hoc by mean to remove the well-known permutation ambiguity. An alarm fires whenever the decoded state changes between t−1 and t.

CUSUM. Two-sided cumulative sum with drift, bootstrap-calibrated to a 1.0 / yr false-alarm target using a circular block bootstrap (block size 10, 64 samples) on the z_t series. No grid point met the 1.0 / yr target on the bootstrap; the calibrator fell back to the candidate whose bootstrap median rate was closest to target, yielding h = 5.0, k = 0.1. The actual sample produced 13.5 / yr at that configuration, against the bootstrap estimate of roughly 11 / yr.

PELT. Offline change-point detection with L2 within-segment loss and a complexity penalty selected in two steps: an elbow heuristic on the within-segment-loss versus penalty curve (chose β = 4.0), then held-out validation on the final 25% of the sample, which moved the choice to β = 5.0. PELT is non-causal — a change point at t uses data after t — and is included as an offline benchmark, not a live alerting system.

Fixed-threshold baseline. An alarm fires whenever z_t > 1.75 on a crossing-up, so a run of consecutive high days produces one alarm rather than many. One parameter, fixed across the sample, no calibration step.

The matching rule was committed in advance of computing any primary metric; its content carries a SHA-256 fingerprint beginning 22b80a4a, recorded in the run manifest. Reported metrics follow the standard definitions, with the headline pair being median latency together with false alarms per year. No detector is described as "earlier" than another unless the computed median latency supports it.

The primary-parameter comparison is summarised in the table and the two bar charts below. The fixed-threshold baseline dominates on F1 and on false-alarm rate. CUSUM has the highest recall (0.833) in the non-baseline group but purchases it at twenty times the baseline's false-alarm rate. HMM's 81% hit rate is artifactual: with 968 alarms in 4,119 days there is an alarm somewhere within ±10 days of nearly every event by chance, and precision is 0.04. PELT, the only named detector chosen by hyperparameter selection, still produces nearly four times as many false alarms as the baseline for one-quarter of the hits.

The median is a coarse view of latency. The empirical CDF of signed trading-day distance for each detector's matched alarms makes the difference much clearer. The baseline's distribution is the most concentrated around zero: roughly 60% of its matched alarms fall in the [−1, +1] trading-day window. PELT's matches are dispersed across the full ±10-day envelope, with a fat negative tail that says when PELT does catch an event it tends to catch it days early — a hallmark of an offline detector dating the change at the pre-event drift rather than at the event itself. CUSUM is biased late; HMM is roughly symmetric and dispersed, consistent with mostly chance alignment.

Restated in calendar time, the baseline produces alarms only in years that were actually stressful — 2010, 2011, 2018, 2020, 2022, 2025 — and is silent through every calm year. The 2020 column does most of the work: 4 hits against the COVID-stress cluster, with 6 additional alarms in the same year that were not within ±10 days of a labelled event. PELT, CUSUM, and HMM produce nearly uniform alarms across years regardless of which years were actually stressful, which is the signature of a too-loose detector that interprets the long-memory of squared returns as a constant stream of regime breaks.

The trade-offs above are not knife-edge in any of the named detectors. The HMM panel is flat because EM converges in roughly 88 iterations regardless of the iteration cap — the result is genuinely a property of the two-state Gaussian specification, not the optimiser. CUSUM sweeps across the grid without any cell reaching the 1.0 / yr target; extending the grid to higher h would be needed to find one, and at that point the detector is silent for most of the sample. PELT at β = 8 does drop below 1 false alarm per year, but its median latency goes to −3, which is suspicious in an offline detector and consistent with PELT fitting the post-event decompression as a level shift while dating the change at the pre-event drift.

A fixed z > 1.75 rule has two structural advantages on this dataset. It is level-based rather than change-based, and the true-event labels are themselves level-based — a high-VIX day and a high-z_t day are nearly the same object by construction. And it has a single parameter, which is hard to overfit on a 48-event problem. The named detectors are change-based: they spend their alarm budget on the entry into a stress regime and on the exit, neither of which necessarily lines up with the level-based labels.

The HMM result in particular is essentially noise. The fitted transition matrix has a 67% per-day probability of switching from the high-vol state back to low-vol, which after relabelling produces 968 state-change alarms in 4,119 days. The standard fix — requiring multi-day persistence of the high-vol state before declaring a change — would belong to a future revision. For CUSUM, the bootstrap "null" is not white because the resampler draws from z_t itself, which inherits the long-memory of squared returns; a model-based null such as a fitted GARCH(1,1) would lower the achievable threshold.

Define a simple operational score Score = F₁ − λ_lag · max(0, ℓ̄) − λ_fp · FA/yr with λ_lag = λ_fp = 0.02 (penalise one trading-day of mean lag the same as one extra false alarm per year, each costing two F1 points). The ranking is unchanged: the baseline scores 0.427, PELT 0.193, CUSUM 0.034, HMM −1.059, and any λ_fp ≥ 0.01 keeps the baseline on top.

The single largest limitation is the fallback share: 99.0% of the modelled series is squared daily returns rather than intraday realized variance, so the conclusions strictly apply to log squared daily return as a regime input. Replacing the intraday path with a feed that has long historical coverage and re-running the entire bake-off — to see whether the headline result changes when the input is genuine intraday RV across the full window — is the highest-leverage next step.

Two further limitations concern the event set. The labels are level-based by construction, which favours the level-based baseline structurally; a version of the bake-off using macroeconomic change points such as FOMC pivots and rates surprises would be a stronger test of the change-based detectors. The event set also contains only one NBER recession within the window — pre-2010 starts sit before the sample begins — so the experiment is effectively dominated by VIX events.

A handful of methodological caveats apply to the named detectors. PELT is non-causal: it uses future observations to date past change points and is included only as an offline benchmark. CUSUM's bootstrap calibration null is not white because the resampler inherits the long-memory of squared returns. HMM is sensitive to initialisation: although the seed is pinned at 1729 and states are post-hoc relabelled by mean, a different seed could produce a different transition matrix and therefore a different alarm count. All four detectors are also evaluated on the same 2010–2026 sample on which their hyperparameters were chosen; PELT's elbow selection and CUSUM's bootstrap calibration are particularly exposed to that double-use. Natural follow-ups on the detector side are a persistence rule on the HMM (alarm only when the high-vol state has been decoded for ≥ k consecutive days) and a Bayesian online change-point detector as a like-for-like online competitor to the fixed threshold; on the evaluation side, holding out a final two years strictly out-of-sample for hyperparameter selection.

The full state of the run is captured by four 16-hex fingerprints: a configuration hash, the SHA-256 of the metrics CSV, the data-vintage checksum on the downloaded raw inputs at run time, and the evaluation-protocol fingerprint that pins the matching rule. Any subsequent edit to the protocol, configuration, or downloaded data would be detectable. The Python environment is pinned: NumPy 2.4.6, pandas 3.0.3, SciPy 1.17.1, hmmlearn 0.3.3, ruptures 1.1.10, Matplotlib 3.10.9, Jinja2 3.1.6, yfinance 1.3.0.