Programme Algebra Retrieval Benchmark
Generated: 26 October 2026
Replacement verdict: OPEN — benchmark not qualified
Outcome
The algebraic projection materially widens retrieval: on the original holdout it raises Recall@10 from 0.333 to 1.000 and MRR from 0.222 to 0.337, principally by recovering JSON ledgers, transformation sources and empirical-return records absent from Markdown search. It is not a drop-in replacement: algebra-only ranking worsens the prose lineage cases, naive score fusion fails, and the keyword intent router does not pass its separately frozen validation stratum.
The later, separately frozen typed-facet candidate scores Recall@10 1.000 and MRR 0.492. This is evidence about an explicit source-kind interface, not permission to erase the failed router result or replace canonical retrieval.
Metrics
Each cell is Recall@10 / mean reciprocal rank.
| Split | Cases | Canonical | Algebra | Naive fusion | Routed |
|---|---|---|---|---|---|
| calibration | 6 | 0.333 / 0.250 | 1.000 / 0.487 | 0.833 / 0.380 | 0.833 / 0.546 |
| holdout | 6 | 0.333 / 0.222 | 1.000 / 0.337 | 0.667 / 0.432 | 1.000 / 0.480 |
| validation | 6 | 0.167 / 0.083 | 0.500 / 0.124 | 0.500 / 0.141 | 0.500 / 0.124 |
| faceted-validation | 6 | 0.333 / 0.333 | 1.000 / 0.242 | 1.000 / 0.492 | 1.000 / 0.492 |
Case ledger
| Split | Case | Canonical rank | Algebra rank | Fusion rank | Routed rank |
|---|---|---|---|---|---|
| calibration | calibration-hydrodynamic-lineage | 1 | 7 | 1 | 1 |
| calibration | calibration-e8-edge-lineage | 2 | 9 | 3 | 9 |
| calibration | calibration-complete-experiment-programmes | — | 1 | 2 | 1 |
| calibration | calibration-incomplete-experiment-atlas | — | 2 | — | — |
| calibration | calibration-instrumentation-return | — | 6 | 9 | 6 |
| calibration | calibration-defeat-graph | — | 1 | 3 | 1 |
| holdout | holdout-detector-route-refusal | 3 | 7 | 1 | 3 |
| holdout | holdout-casimir-platform-route | 1 | 3 | 1 | 1 |
| holdout | holdout-th229-incomplete-return | — | 3 | 9 | 3 |
| holdout | holdout-evidence-spine-records | — | 1 | 3 | 1 |
| holdout | holdout-claimipedia-publication-batch | — | 10 | 13 | 10 |
| holdout | holdout-d02-boundary-source | — | 9 | 14 | 9 |
| validation | validation-complexity-cards | — | — | — | — |
| validation | validation-detector-register | — | 17 | — | 17 |
| validation | validation-mystery-distance | — | 5 | 6 | 5 |
| validation | validation-open-topics | — | 5 | 9 | 5 |
| validation | validation-e8-tomography-prose | 2 | 5 | 2 | 5 |
| validation | validation-wave-gate-audit | — | 12 | 15 | 12 |
| faceted-validation | faceted-continuation-map | — | 6 | 6 | 6 |
| faceted-validation | faceted-admissibility-zone-register | — | 3 | 3 | 3 |
| faceted-validation | faceted-quarter-turn-propagation-audit | — | 4 | 4 | 4 |
| faceted-validation | faceted-documentation-gate | — | 5 | 5 | 5 |
| faceted-validation | faceted-transverse-detector-prose | 1 | 3 | 1 | 1 |
| faceted-validation | faceted-fixed-index-defect-prose | 1 | 6 | 1 | 1 |
Failed replacement gates
routedValidationMarkdownNonRegressingroutedValidationMachineRecoveredAt10
Implications
- Keep both indices. Canonical Markdown search remains better for ordinary prose and exact conceptual destinations.
- Use algebra for source classes the current index cannot see. JSON records, empirical packets, explicit refusals and transformation-family sources are the demonstrated gain.
- Do not average scores blindly. The retained fusion control buries machine sources without reliably improving prose.
- Do not infer output lane from broad keywords alone. “Protocol” can name a prose research document as easily as a machine schema.
- Treat source kind as an explicit selector. The typed facet is a user- or query-planner declaration, not a semantic fact inferred from loose vocabulary.
- Next gate: test typed facets on a larger untouched query set, then compare reasoning answers and provenance—not retrieval alone—before considering replacement.
Authority boundary
Expected sources test navigation and structural recovery only. They do not establish the truth, completeness or scientific authority of any source.
- Corpus digest:
d09c3db276dab1800bf0c8555418fdd25151f4ab9214af40e9bb4aafb80c940c - Algebra archive:
df63624275ec89aa7548340c5743f69ac28e9b9304b14ee52f7b086169dfaa7c - Suite:
2fc979749f8788ff56b3278fa4c510af05127ae56c34fdac57393268a834cb64