Software companion
Writeup Series Charter and Persistent Metric Protocol
Overview
A ten-report programme accumulates a comparability problem: numbers computed under different runs, different splits and different selection rules drift out of joint, and by the end nobody can say which comparisons are legitimate. This document is the contract that prevents that, and it is enforced in code rather than by convention.
Three things carry it. Metric identifiers are persistent, so a metric named in report two means the same computation in report nine. The house statistic is one paired moving-block bootstrap with a fixed policy, reported as a delta with a 90% interval, and a difference is either resolvable at that bar or recorded as unresolved — there is no third option and no per-comparison choice of test. And the validation ledger is append-only, so a result cannot quietly improve after the fact.
The audits are the part that makes the contract self-checking. audit_skill_metrics.py recomputes metrics from stored artifacts, audit_decision_resolvability.py rebuilds the retrospective decision record and asks which past decisions were actually resolvable on the evidence available at the time, and check_text_integrity.py checks the written record against the artifacts it cites.
Implementation
- Statistic
- A paired moving-block bootstrap under a fixed block policy, reported as a delta with a 90% interval
- Standard
- A difference is resolvable or it is unresolved; the bar is stated in advance and does not vary per comparison
- Identifiers
- Persistent metric names, so the same name denotes the same computation across all eleven documents
- Ledger
- An append-only validation ledger recording every evaluation as it happened
- Gates
- Protocol freezing with hash checks, enforcing single-use reads of sealed periods
- Audits
- Metric recomputation, retrospective decision resolvability, and text-versus-artifact integrity
Components
skill.py- The skill definitions and persistent metric identifiers
training/metrics.py- Field, index and amplitude metrics, computed once and reused everywhere
stats.py- The paired moving-block bootstrap, interval construction and the resolvability rule
decision_reporting.py- Decision records binding each adoption or rejection to its evidence
experiment_reporting.py- Experiment-level reporting shared by every phase
pretrained_reporting.py- Reporting for the pretraining and transfer comparisons
final_protocol.py- Protocol freezing, hash gating and single-use test enforcement
scripts/audit_skill_metrics.py- Recomputation of reported metrics from stored artifacts
scripts/compare_evaluations.py- The standard paired comparison between two existing evaluations
scripts/audit_decision_resolvability.py- Retrospective audit of whether past decisions were resolvable at the time
scripts/check_text_integrity.py- Consistency checking between the written record and the artifacts it cites
Measurement contract
declareName the metric from the persistent identifier set; new names are additions, not renames.evaluateProduce an evaluation directory with its own artifacts and provenance.compareRun the paired bootstrap between two evaluations under the fixed block policy.resolveRecord the result as resolvable or unresolved at the stated bar.ledgerAppend to the validation ledger; entries are never rewritten.auditRecompute metrics, re-derive resolvability, and check the prose against the artifacts.
Reproducibility and validation
- The bootstrap policy is fixed in advance, so no comparison gets to choose the test that suits it.
- The validation ledger is append-only, which makes the sequence of what was known when recoverable rather than reconstructed.
- Protocol gates are hash-checked in code, so a sealed-period read cannot happen without a frozen protocol document behind it.
- The resolvability audit is retrospective and adversarial by design: it asks which past decisions the evidence actually supported, and its answers are part of the record.
- A text-integrity check ties the written reports back to the artifacts they cite, so a number cannot survive in prose after the artifact behind it changes.
Availability
Not publicly released. This is the most reusable part of the repository — the statistics, metric registry and gating logic have no dependency on the wavelet representation — but it is currently entangled with the series’ artifact layout rather than packaged as a standalone library.