tooling: add pre-anchored update-only mode to canonical harness
Add FIXTURE_MODE=updateOnly support that bypasses Start and sends the committed fixture (tests/fixtures/pre-anchored-update-savings-realism.json) directly as an Update request body through production HTTP route. scripts/reproduce-multi-turn-investigation.mjs: - Added ESM imports for deterministic fixture loading (fs, fileURLToPath, path) - Added FIXTURE_PATH constant pointing to committed fixture - Added fixtureMode env-var selector and runUpdateOnlyMode() function - Validates ANSWER_2 before any live call (zero calls if missing) - Verifies single savings-realism anchor invariant on load - Preserves all hardened capture fields in pre-anchored mode - Normal-mode Start→Update chain preserved under guard clause tests/reproduce-multi-turn-investigation.harness.test.js: - Added 7 new harness tests for pre-anchored scenarios (46 total, all pass) - Updated runPreAnchoredSimulation to persist rejectedProposalSnapshot on rejection - Added runPreAnchoredSimulationWithBlock() helper docs/: - New docs/experiment-57j78.md with full apparatus description - Updated docs/current-handoff.md with 57J.78 section
This commit is contained in:
@@ -1472,3 +1472,39 @@ The 57J.75 experiment document ("experiment: validate controlled structural no-o
|
||||
**What remains unproven:** How the 57J.75 live call was actually sent (uncommitted script edit vs. direct API invocation); reproducibility without reverting those uncommitted changes.
|
||||
|
||||
**Smallest next tooling boundary: A.** Add committed pre-anchored mode to the canonical harness by adding a single config flag that bypasses Start and reads the fixture into the Update request body, mirroring what `runPreAnchoredSimulation()` documents as its intended behaviour.
|
||||
|
||||
### Experiment 57J.78 — Pre-Anchored Update-Only Mode (Committed)
|
||||
|
||||
**Objective:** Add committed pre-anchored update-only mode to the canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`), eliminating the dependency on temporary uncommitted script modifications identified in audit 57J.77. **Classification: IMPLEMENTED.**
|
||||
|
||||
**Changes made:**
|
||||
|
||||
1. `scripts/reproduce-multi-turn-investigation.mjs` — Added ESM imports (`fs`, `fileURLToPath`, `path`) for deterministic fixture loading. Introduced `FIXTURE_PATH` constant pointing to `tests/fixtures/pre-anchored-update-savings-realism.json`. Added `fixtureMode` env-var selector and `runUpdateOnlyMode()` function that:
|
||||
- Loads the committed fixture file (exits with error on read failure)
|
||||
- Verifies the single savings-realism anchor invariant
|
||||
- Deep-copies the fixture graph (no mutation of original fixture)
|
||||
- Skips Start entirely; sends exactly one Update through production HTTP route via `postJson()`
|
||||
- Preserves all hardened capture fields: answerMeaning, updatedProposal, structuralActionRequired, selectedQuestion, persistent graph snapshot
|
||||
- Blocks on missing `ANSWER_2` env-var (zero live calls)
|
||||
- Reports rejection diagnostics identically to normal mode
|
||||
- Normal-mode path is preserved unmodified under a guard (`fixtureMode !== undefined`)
|
||||
|
||||
2. `tests/reproduce-multi-turn-investigation.harness.test.js` — Added 7 new harness tests:
|
||||
- Blocked ANSWER_2: zero calls, correct error message
|
||||
- Accepted structuralActionRequired=true in capture
|
||||
- Rejected snapshot preservation with structural linkage errors
|
||||
- Exact ANSWER_2 body forwarding verification
|
||||
- Pre-anchored rejected answerMeaning preservation
|
||||
- Blocked mode verification (zero fixture load errors)
|
||||
- Normal-mode isolation proof (accepted/rejected capture unchanged)
|
||||
|
||||
Updated `runPreAnchoredSimulation` mock to persist `rejectedProposalSnapshot` on rejection return values. Added `runPreAnchoredSimulationWithBlock()` helper.
|
||||
|
||||
**Evidence:** 46 harness tests pass (39 pre-existing + 7 new). No production code changed. No Ollama calls. No live API calls. Normal-mode Start→Update chain unmodified under guard.
|
||||
|
||||
**Execution command:**
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly ANSWER_2="I am unsure whether the projected office savings from the relocation are realistic." node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
This satisfies 57J.77's boundary A recommendation: a committed update-only path that loads `tests/fixtures/pre-anchored-update-savings-realism.json` and sends it as an Update request body without first running Start.
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# Experiment 57J.78 — Pre-Anchored Update-Only Mode (Committed)
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `9b7721c` (experiment: audit pre-anchored live apparatus)
|
||||
**Commit message:** `tooling: add pre-anchored update-only mode to canonical harness`
|
||||
|
||||
## Objective
|
||||
|
||||
Eliminate the dependency on temporary uncommitted script modifications identified in audit 57J.77, by adding a committed pre-anchored update-only mode to the canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`). This allows any agent session at current HEAD to inject an arbitrary graph into the Update request body without first running Start.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. scripts/reproduce-multi-turn-investigation.mjs (+177 lines)
|
||||
|
||||
Added:
|
||||
- ESM imports (`fs`, `fileURLToPath`, `path`) for deterministic fixture loading
|
||||
- `FIXTURE_PATH` constant pointing to `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- `fixtureMode` env-var selector (default: undefined → normal mode)
|
||||
- `runUpdateOnlyMode()` async function:
|
||||
- Validates ANSWER_2 env-var exists before any live call
|
||||
- Loads committed fixture from deterministic path
|
||||
- Verifies single savings-realism anchor invariant
|
||||
- Deep-copies fixture graph (no mutation of original)
|
||||
- Skips Start entirely; sends exactly one Update via `postJson()` through production HTTP route
|
||||
- Preserves all hardened capture fields (answerMeaning, updatedProposal, structuralActionRequired, selectedQuestion, persistent graph snapshot)
|
||||
- Blocks on missing ANSWER_2 with zero live calls
|
||||
- Reports rejection diagnostics identically to normal mode
|
||||
|
||||
### 2. tests/reproduce-multi-turn-investigation.harness.test.js (+183 lines)
|
||||
|
||||
Added 7 new harness tests:
|
||||
- Blocked ANSWER_2 → zero calls, correct error message
|
||||
- Accepted structuralActionRequired=true in capture
|
||||
- Rejected snapshot preservation with structural linkage errors
|
||||
- Exact ANSWER_2 body forwarding verification
|
||||
- Pre-anchored rejected answerMeaning preservation
|
||||
- Blocked mode verification (zero fixture load errors)
|
||||
- Normal-mode isolation proof (accepted/rejected capture unchanged)
|
||||
|
||||
Updated `runPreAnchoredSimulation` mock to persist `rejectedProposalSnapshot` on rejection return values. Added `runPreAnchoredSimulationWithBlock()` helper.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Test Suite | Pre-existing | New | Total | Result |
|
||||
|------------|-------------|-----|-------|--------|
|
||||
| Harness harness tests | 39 | 7 | 46 | ALL PASS (19ms) |
|
||||
|
||||
- No production code changed
|
||||
- No Ollama calls made
|
||||
- No live API calls made
|
||||
- Normal-mode Start→Update chain preserved under guard
|
||||
- Syntax validated via `node --check`
|
||||
|
||||
## Execution Commands
|
||||
|
||||
### Pre-anchored update-only mode:
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
ANSWER_2="I am unsure whether the projected office savings from the relocation are realistic." \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
### Normal start→update mode (unchanged):
|
||||
```bash
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
## Design Decisions
|
||||
|
||||
1. **Environment variable over CLI flag:** `FIXTURE_MODE` env-var is simplest, requires no arg parsing, and matches existing pattern (`CONFIDENCE_ENGINE_BASE_URL`).
|
||||
|
||||
2. **ANSWER_2 required guard:** Prevents accidental live calls without a clear answer payload. Zero calls made if missing.
|
||||
|
||||
3. **ESM imports for path resolution:** `fileURLToPath(import.meta.url)` resolves the fixture path relative to the script location, matching Node.js ESM best practices.
|
||||
|
||||
4. **No schema/schema validator changes:** The committed fixture file was already validated per existing schema enums in test 57J.74 (tests on lines 582-624 of the test file).
|
||||
|
||||
5. **Normal mode guard:** `fixtureMode !== undefined` check prevents the pre-anchored path from being activated when no env-var is set, preserving all existing start→update behavior.
|
||||
|
||||
## Verification
|
||||
|
||||
1. All 46 harness tests pass in under 20ms
|
||||
2. No production code was modified
|
||||
3. Syntax validated via `node --check`
|
||||
4. Normal-mode Start→Update chain preserved at its original location (line 63 of the mjs file)
|
||||
5. Pre-anchored mode explicitly documented with inline JSDoc comments
|
||||
|
||||
## Satisfies 57J.77 Recommendation
|
||||
|
||||
> "Add committed pre-anchored mode to the canonical harness by adding a single config flag that bypasses Start and reads the fixture into the Update request body, mirroring what runPreAnchoredSimulation() documents as its intended behaviour."
|
||||
|
||||
This commit implements exactly that recommendation — `runUpdateOnlyMode()` is the committed implementation of what `runPreAnchoredSimulation()` previously documented only as a test-only mock.
|
||||
Reference in New Issue
Block a user