robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tooling: add pre-anchored update-only mode to canonical harness

Browse Source 
Add FIXTURE_MODE=updateOnly support that bypasses Start and sends the
committed fixture (tests/fixtures/pre-anchored-update-savings-realism.json)
directly as an Update request body through production HTTP route.

scripts/reproduce-multi-turn-investigation.mjs:
  - Added ESM imports for deterministic fixture loading (fs, fileURLToPath, path)
  - Added FIXTURE_PATH constant pointing to committed fixture
  - Added fixtureMode env-var selector and runUpdateOnlyMode() function
  - Validates ANSWER_2 before any live call (zero calls if missing)
  - Verifies single savings-realism anchor invariant on load
  - Preserves all hardened capture fields in pre-anchored mode
  - Normal-mode Start→Update chain preserved under guard clause

tests/reproduce-multi-turn-investigation.harness.test.js:
  - Added 7 new harness tests for pre-anchored scenarios (46 total, all pass)
  - Updated runPreAnchoredSimulation to persist rejectedProposalSnapshot on rejection
  - Added runPreAnchoredSimulationWithBlock() helper

docs/:
  - New docs/experiment-57j78.md with full apparatus description
  - Updated docs/current-handoff.md with 57J.78 section
This commit is contained in:
robbond
2026-08-12 10:19:09 +01:00
parent 9b7721c610
commit ce01e70010
4 changed files with 486 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+36
View File
@@ -1472,3 +1472,39 @@ The 57J.75 experiment document ("experiment: validate controlled structural no-o
**What remains unproven:** How the 57J.75 live call was actually sent (uncommitted script edit vs. direct API invocation); reproducibility without reverting those uncommitted changes.
**Smallest next tooling boundary: A.** Add committed pre-anchored mode to the canonical harness by adding a single config flag that bypasses Start and reads the fixture into the Update request body, mirroring what `runPreAnchoredSimulation()` documents as its intended behaviour.
### Experiment 57J.78 — Pre-Anchored Update-Only Mode (Committed)
**Objective:** Add committed pre-anchored update-only mode to the canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`), eliminating the dependency on temporary uncommitted script modifications identified in audit 57J.77. **Classification: IMPLEMENTED.**
**Changes made:**
1. `scripts/reproduce-multi-turn-investigation.mjs` — Added ESM imports (`fs`, `fileURLToPath`, `path`) for deterministic fixture loading. Introduced `FIXTURE_PATH` constant pointing to `tests/fixtures/pre-anchored-update-savings-realism.json`. Added `fixtureMode` env-var selector and `runUpdateOnlyMode()` function that:
- Loads the committed fixture file (exits with error on read failure)
- Verifies the single savings-realism anchor invariant
- Deep-copies the fixture graph (no mutation of original fixture)
- Skips Start entirely; sends exactly one Update through production HTTP route via `postJson()`
- Preserves all hardened capture fields: answerMeaning, updatedProposal, structuralActionRequired, selectedQuestion, persistent graph snapshot
- Blocks on missing `ANSWER_2` env-var (zero live calls)
- Reports rejection diagnostics identically to normal mode
- Normal-mode path is preserved unmodified under a guard (`fixtureMode !== undefined`)
2. `tests/reproduce-multi-turn-investigation.harness.test.js` — Added 7 new harness tests:
- Blocked ANSWER_2: zero calls, correct error message
- Accepted structuralActionRequired=true in capture
- Rejected snapshot preservation with structural linkage errors
- Exact ANSWER_2 body forwarding verification
- Pre-anchored rejected answerMeaning preservation
- Blocked mode verification (zero fixture load errors)
- Normal-mode isolation proof (accepted/rejected capture unchanged)
Updated `runPreAnchoredSimulation` mock to persist `rejectedProposalSnapshot` on rejection return values. Added `runPreAnchoredSimulationWithBlock()` helper.
**Evidence:** 46 harness tests pass (39 pre-existing + 7 new). No production code changed. No Ollama calls. No live API calls. Normal-mode Start→Update chain unmodified under guard.
**Execution command:**
```bash
FIXTURE_MODE=updateOnly ANSWER_2="I am unsure whether the projected office savings from the relocation are realistic." node scripts/reproduce-multi-turn-investigation.mjs
```
This satisfies 57J.77's boundary A recommendation: a committed update-only path that loads `tests/fixtures/pre-anchored-update-savings-realism.json` and sends it as an Update request body without first running Start.