Real-Time Machine Learning in the Cross-Section: A Public-Data Reproduction

A leak-audited real-time ML pipeline — expanding-recursive annual windows, the paper’s exact 15-point grid, decile portfolios, six-factor alphas, and a cost overlay — built to reproduce the Chen–Zimmermann Table 7 sub-analysis of a prominent equity-ML paper. The paper-faithful track is blocked at a single identifier seam; a second, commercial-clean public track runs end to end and is the source of every live number here.

A public replication audit is the sharpest available test of whether a published result survives contact with public data and an independently built pipeline. This audit targets Real-Time Machine Learning in the Cross-Section of Stock Returns (Li, Rossi, Yan & Zheng, Journal of Financial Economics, 2025). It deliberately does not chase the paper’s marquee 18,113-signal claim — that universe is built from Compustat in the Yan–Zheng style and is not reproducible from public data — but the paper’s Table 7 Chen–Zimmermann sub-analysis, a robustness check on 207 published anomalies for which the paper reports a boosted-tree equal-weighted 10−1 return of 5.18%/month and an annualized Sharpe of 3.68.

The honest finding has three parts. Methodologically, a faithful, leak-audited real-time pipeline can be built and verified end to end, and the paper-faithful OSAP track is ready to run the moment an identifier-to-ticker bridge is available. On scope, that OSAP track is blocked at a single, nameable seam — the Open Source Asset Pricing library ships its signals keyed by a CRSP security identifier with no public route to a tradeable ticker — so no OSAP Table 7 number is claimed; the seam is located rather than papered over. Empirically, the one configuration that runs fully on public, commercial-clean data delivers an equal-weighted boosted-tree Sharpe of 1.25 (t = 3.30) that survives six-factor adjustment and short-term-reversal removal, yet collapses under value-weighting (0.43) and microcap exclusion (0.54) and is bracketed by a linear model above it and a two-layer neural net below — real, significant, well understood, and correctly not comparable to the paper’s 3.68, for reasons enumerated rather than hidden.

Li, Rossi, Yan & Zheng (LRYZ) ask whether ML return-prediction strategies are truly implementable in real time, or whether their reported performance benefits from hindsight about anomalies that were only published later. Their headline finding is sobering: on a universe of 18,113 fundamental signals, a real-time boosted regression tree earns an equal-weighted 10−1 spread of only 0.95%/month (t = 6.63, Sharpe 1.02) and a value-weighted spread of 0.40%/month — substantially weaker than the prior anomaly-ML literature had led readers to expect.

The paper anticipates the obvious objection — “maybe the ML is just bad” — with a battery of robustness samples. Only one cell of that structure can be faithfully targeted from public inputs, and even it runs into a hard data seam. The annual cadence is essential and easy to get wrong: LRYZ predict annual July(t)-to-June(t+1) excess returns, refit annually, hold for twelve months, and rebalance once a year. Every design choice in this audit assumes that annual cadence.

Replicating Table 7 demonstrates that an OSAP-plus-boosting-plus-public-price pipeline can reproduce prior-literature-magnitude alphas on published anomalies. It does not test the paper’s central claim, because the 18k universe cannot be built from public data. Every empirical number in this report carries Table 7 / CZ-sample scope and must never be read as a full-universe result.

In the course of the work the project split into two tracks, both engineered behind a single interface and schema layer. Track A — the OSAP headline (paper-faithful) targets Table 7 directly with the OSAP March 2022 207-signal library, a CRSP-substitute price panel, and the paper’s exact protocol. It is fully engineered and tested, but its production headline run is blocked at the identifier seam. Track B — the commercial-clean public track (ticker-native) removes all WRDS/CRSP/Compustat dependencies by using the SEC company-ticker universe with the central index key (CIK) as the entity id, SEC EDGAR XBRL fundamentals (point-in-time by filing date), and a licensed price feed. It recomputes a focused 17-signal anomaly subset, runs end to end on real data, and is the source of every live result below.

The split is itself a governance finding: the identifier and licensing constraints on the canonical academic data are as binding as the statistical questions, and a deployable pipeline has to be designed around them from the start. The seam that blocks Track A — an identifier with no public route to a ticker — is the very thing Track B dissolves by choosing a public, ticker-mappable universe.

The pipeline is a sequence of validated stages, each recording its row counts, source vintages, runtime, and warnings into a run manifest. Four stages are empirically load-bearing.

Prediction target. Signals for formation year t are paired with the compounded July-to-June forward annual excess return (Equation 1), following the Fama–French publication-availability convention, so signals dated t never see returns that postdate their formation.

Rank transform. Each raw signal is mapped, within each cross-section, to a rank scaled to [−1, +1] (Equation 2), with missing values imputed to 0 — the cross-sectional median by construction, the GKX and paper convention. Because only the ordering survives, a level-vs-log choice for any single predictor is immaterial — a fact the audit exploits when proxying the identifier-gated signals and verifies directly with a missing-value ablation.

The real-time expanding-recursive protocol is the centerpiece (Equation 3). Training expands recursively while a fixed-length validation span rolls forward, so no window ever touches data on or after its test year. The paper uses L = V = 12; Track B uses a shortened L = 6, V = 4 forced by the post-2009 XBRL panel. The implementation enforces y₀ ≤ train_end < val_start ≤ val_end < τ as a hard invariant and runs two leak guards on every fit — one forbids any feature column from equalling the target to numerical tolerance, the other forbids access to any year at or beyond the test year. A dedicated leak test is part of the suite.

Models and evaluation. The headline model is a boosted regression tree (LightGBM) selected over the paper’s exact 15-point grid (Equation 4); two comparison models share the same harness — a linear elastic-net-style baseline and a single two-layer [32, 16] NN2 shape check. Within each test year, stocks are sorted into deciles on the prediction; the strategy is long decile 10, short decile 1, held twelve months, in equal- and value-weighted form. Reported statistics use Newey–West HAC standard errors (6 lags), and risk-adjusted alphas come from time-series regressions under six factor models: CAPM, FF3, Carhart, FF5, FF5+MOM, and the Hou–Xue–Zhang Q-factor model.

Track A implements every methodology stage against the paper-pinned inputs and passes its unit, integration, and leak tests. The OSAP March 2022 release downloads in full, and the boosting library fits cleanly. The blocker is neither of those — it is a single identifier seam.

OSAP distributes every signal as a panel keyed only by a CRSP security identifier and a year-month, with no shipped ticker or CUSIP. Forming portfolios requires a price and market-capitalization series for each identifier, but public price feeds are keyed by ticker, and no public, commercial-clean table bridges the CRSP identifier to a tradeable ticker. The canonical crosswalk is the CRSP security-name file behind WRDS, which is both access-gated and non-commercial; gray community crosswalks are licensing-ambiguous and survivorship-biased on exactly the delisted names that matter. The 207-signal panel therefore cannot be priced without CRSP/WRDS, and no OSAP Table 7 alpha is claimed in this report. Reporting a number from a partial or mis-linked panel would be a governance failure; locating and naming the seam is the finding.

Even blocked, Track A yields a production compute profile, measured on dimension-matched synthetic data at full production scale. Boosting cost scales with rows × features × trees, all of which match production, so the timing is representative even though the feature values are synthetic — the moment an identifier bridge lands, the headline run is a roughly half-hour job, not a re-engineering effort.

SEC XBRL begins at the ~2009 mandate, so the fundamentals-bearing panel spans 2009–2024: 394 names, 4,623 firm-years (~289 names/year), 17 ranked signals, with overall signal coverage of 84.6%. Three consequences of this short, large-cap panel drive every downstream caveat. The protocol must be shortened to 6+4, making 2019 the first forecast year and giving six windows (2019–2024) and 72 holding months (July 2019–June 2025). No in-sample-versus-post-publication decay comparison is possible, because every Track B holding month is already post-publication — a per-forecast-year view stands in for it. And the universe is a free-tier large-cap set of the ~470 largest SEC filers, so value-weighting should, and does, dampen the spread sharply.

The boosted-tree 10−1 portfolio earns an equal-weighted 16.6%/year (1.387%/month) at an annualized Sharpe of 1.25 with a Newey–West t of 3.30; the value-weighted spread is 11.5%/year at Sharpe 0.43 (t = 1.20) — statistically indistinguishable from zero. The equal-over-value gap is exactly the smaller-cap concentration of anomaly returns operating on a large-cap universe, and the VW drawdown (−33% versus EW’s −14%) reflects the handful of mega-caps that dominate the value-weighted legs. The spread is real but top-heavy: it is driven mainly by decile 10 against a relatively flat middle, rather than by a clean monotone staircase — useful texture for anyone tempted to read a single Sharpe number as a uniform signal.

The equal-weighted spread is not a factor exposure in disguise. Across all six factor models the EW 10−1 alpha is 1.2–1.5%/month with Newey–West t-statistics between 2.9 and 4.6; the Carhart and FF5+MOM alphas confirm that momentum does not absorb it, and the Q-factor alpha (1.48%/mo, t = 3.5) survives the investment and profitability factors. The value-weighted alphas, by contrast, are economically smaller and statistically insignificant (t ≤ 1.9 in every model) — the same equal-over-value story the raw spread tells, now visible through a risk-model lens.

The headline survives — or fails in interpretable ways — across the ablations the panel permits, and four findings stand out. The spread is not a short-term-reversal artifact: dropping the one-month reversal signal leaves the EW Sharpe essentially unchanged (1.265 versus 1.253) and actually raises the VW Sharpe, echoing the paper’s similarly modest CZ reversal-excluded result. Size concentration is the dominant lever: removing the smallest within-panel size decile cuts the EW Sharpe by more than half to 0.54, and value-weighting cuts it to 0.43 — both point at the premium living in the smaller names. Complexity is not rewarded: a two-layer NN2 underperforms the boosted tree badly (0.54 versus 1.25) while a plain linear model slightly beats it (1.30), because on a 17-signal, ~289-name annual panel there is little non-linear structure to exploit. And imputation is load-bearing: letting the booster handle missing values natively reproduces the headline (1.225), but dropping every row with any missing signal is degenerate — only 7.6% of firm-years are fully observed, so the panel thins and the Sharpe falls to 0.84, which is itself the finding that the accounting signals are sparse early and zero-imputation is doing real work. The long-window ablations remain skipped — reported as skipped rather than silently omitted — because the post-2009 panel cannot form windows needing 22–30 years of history.

A governance audit should be able to say which signals a model leans on, not just how it scores. BRT gain importance averaged over the six forecast windows is coherent with the ablations: size and price characteristics dominate. The size proxy alone is 15.5% of total gain, and together the two size measures plus the price level account for roughly 28%; volatility, reversal, value, and momentum signals form a broad middle band; and the sparsely-populated accounting fundamentals sit at the bottom. This is exactly why removing the smallest size decile guts the spread — the model’s single most important axis is size, and the premium it captures lives in the smaller names. It also flags a redundancy: the panel carries both a market-equity size proxy and an accounting size signal, which a production build would collapse to one.

The 17-signal set is itself worth naming. The price- and volume-based signals are six- and twelve-month momentum, twelve-month cumulative return, one-month reversal, long-term (months 13–60) reversal, the twelve-month maximum daily return, and twelve- and thirty-six-month volatility. The accounting signals from SEC XBRL are gross and operating profitability, accruals, asset growth, net issuance, book-to-market, and size. Two market-linked proxies — the price level and a market-equity size proxy — complete the set.

With no in-sample leg to decay from, the honest substitute is to ask whether the composite has weakened across the six post-publication forecast years. It has not: the equal-weighted 10−1 spread is positive in every year from 2019 to 2024 (+25%, +15%, +3%, +21%, +9%, +27%), with no monotone fade. The value-weighted leg is far choppier — two negative years (2021 and 2023) — which is the year-by-year shadow of its weak full-sample Sharpe. The equal-weighted composite is durable over this window; it is simply small-cap-tilted.

Because the holding window (2019–2025) lies entirely past the Chen–Velikov 2017 cutoff, the paper-faithful low-frequency-spread overlay cannot be applied to Track B, and a fixed-bp sensitivity is reported instead. Measured annual one-way turnover averages ~67% per leg (~134% round-trip), so even at an aggressive 50 bp/side the equal-weighted net Sharpe is 1.216 (gross 1.253) and the net return is 16.1%/yr (gross 16.6%) — a trivial drag, because the strategy rebalances once a year. Costs are not what separates this composite from the paper’s; the universe, breadth, and weighting are.

The paper’s CZ-sample BRT posts an EW Sharpe of 3.68; Track B posts 1.25. This gap is expected and attributable to design differences, not a contradiction of the paper. The runnable levers are large and all point the same way: value-weighting (−0.82 Sharpe) and removing the smallest names (−0.71) each roughly halve the spread, confirming that the bulk of the magnitude difference is a size-and-weighting story, not an ML story. The residual to 3.68 is carried by the levers that cannot be reproduced from public data — the OSAP 207-signal breadth versus the 17 here, the full CRSP cross-section versus ~470 large caps, the 12+12 window versus 6+4, and the 1987–2019 sample versus a six-year post-publication window.

The disciplined reading is that a fully public, commercial-clean anomaly-ML pipeline produces an economically meaningful, statistically significant equal-weighted spread out of sample, while value-weighting it on a large-cap universe removes most of the premium — directionally consistent with both the anomaly literature and the target paper’s caution about real-time implementability.

The audit is engineered so that any number traces to a configuration, a code path, and a data vintage. Thirteen frozen project decisions — scope tier, window, protocol, OSAP pin, grid, factor roster, cost model, imputation, filters, seeds, and acceptance bands — together with two labelled deviations are recorded in a decision log and mirrored into versioned configuration, so changing one requires an explicit log update. Every live number is rebuilt from the committed processed panel and a single date-stamped seed by the project’s own modules, and the reconstructed equal- and value-weighted Sharpes (1.253 / 0.433) match the committed metrics to four decimals.

A fail-closed production-validation gate refuses to pass on smoke or fixture paths, missing artifacts, placeholder report text, bad snapshot hashes, or a forbidden full-universe scope tag. Its required-artifact list is wired to the blocked Track A OSAP headline, so it gates the paper-faithful deliverable rather than this Track B note, whose governance evidence is the manifest, the deterministic reconstruction, and the test suite. The two deviations are recorded with rationale and auto-superseding behaviour, never as silent substitutions: D1 proxies the three identifier-gated OSAP signals from public prices (adequate because the rank-only transform makes level proxies sufficient), and D2 adopts the commercial-clean ticker-native SEC CIK track because CRSP/Compustat are non-commercial and academic.

Several limitations bound these claims, and naming them is part of the deliverable.

The framework is a reusable template, and a second replication every six months compounds reputational credit far faster than one-off pieces. The gap between what the pipeline can prove and what the public data will allow is precisely the kind of finding a governance function exists to surface.