Study 14 · prospective evaluation
Evaluation design
Comparison of correction reliability under matched evidence, including state construction and total costs. The study remains in development.
Comparison conditions
| Arm | Available context | Distinguishing treatment |
|---|---|---|
| A | Full raw context | No status labels or dependency-governed scheduling. |
| B | Identical full raw context | Status labels, without dependency-governed scheduling. |
| C | Competitive semantic retrieval from raw evidence | Authority and dependency labels excluded from the retrieval index. |
| D | Premise-governed scheduling | Operational state constructed independently of hidden grading references. |
No comparative model results are established. Fixtures and readiness checks are development evidence. Comparison changes require a versioned amendment.
Development record
The successor A/B parity checker compares complete source text, identifiers and order, rejecting text changes the earlier check accepted. This demonstrates one checker correction, without establishing model-performance gains. The original 208 assertions have not been fully recovered and freshly reconstructed.
Controls remain necessary for initial-stage reliance, false completion, independent support, required reopening and valid no-change behavior. Separating construction from grading does not itself establish independent semantic labels.
Assessment requirements
Systems must construct and maintain correction distinctions without privileged answers, with full cost accounting.
- Unit: independently constructed episode; correlated stages and assertions are not independent samples.
- Outcomes: missed corrections, unnecessary reopening, unauthorized reliance, unsupported completion and task quality.
- Costs: state construction and maintenance, context tokens, latency, model and tool calls, human effort.
- Falsification: no material reliability gain under matched information and resources, or gains explained by privileged labels or uncounted construction.
Scoring, retrieval, uncertainty analysis and stopping rules must be fixed before held-out assessment. Pilot execution, new model collection and confirmatory runs remain separately governed.