robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(harness): support gated live investigation continuation

Browse Source
This commit is contained in:
robbond
2026-08-18 06:48:40 +01:00
parent 7dd4a956fb
commit 600b07d820
4 changed files with 693 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+28 -3
View File
@@ -59,6 +59,29 @@
---
## Canonical harness gated apparatus (60B.99)
The canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`) now supports a two-phase gated investigation pattern:
**startOnly** — `FIXTURE_MODE=startOnly`
- Makes exactly one `/api/cases/start` request
- Writes the captured Start state (graph + selectedQuestion) to `.evidence-temp/continuation-start-only.json` (or path set by `CONTINUATION_FILE`)
- Issues zero Update requests
- Exits successfully
**continueOneUpdate** — `FIXTURE_MODE=continueOneUpdate CONTINUATION_ANSWER=<answer>`
- Loads the persisted Start continuation state
- Requires explicit answer (blocks with exit code 1 if missing)
- Makes exactly one `/api/cases/update` using preserved Start state + explicit answer
- Issues zero Start requests
- Exits
**Normal mode** (`FIXTURE_MODE` unset) — unchanged. Start → configured Update loop still works identically to pre-60B.99.
This apparatus corrects the apparatus defect proven in 60B.98: the canonical harness can now stop after Start, allow external semantic inspection of the returned question, and later continue from that exact captured state with an explicitly chosen answer.
---
## Experiment 60B.95 result (2026-08-17)
**Classification: E — LIVE PATH DIVERGED**
@@ -74,8 +97,10 @@ The live model selected nk6eyn2 ("exact monetary value of potential enterprise c
**Classification: E — START PATH DIVERGED**
The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison. Update 1 then hit a result_validation error: "Active unknown violates reasoning pattern consistency: \"n_fincomp12\" is explanation but active pattern is decision."
The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison.
**What this establishes:** (1) The divergence from financial-comparison targets continues across multiple runs; (2) there exists an internal graph-state inconsistency (explanation-node vs decision-pattern mismatch in the Start/Update pipeline).
**Valid evidence retained:** Start = S2 — DIFFERENT START (the live model diverged from the expected financial-comparison path).
**What this does NOT prove:** Whether a financially-comparison-aligned Start would progress coherently with a financial answer, or whether the pattern consistency issue is deterministic.
**Update 1 evidence: DISCARDED.** The canonical harness auto-continued with its preconfigured `answers[0]`, so the Update occurred outside the experiment's semantic gate. This was an apparatus defect (60B.98) — the harness did not provide a post-Start stop gate at that time. The HTTP 500 is NOT established as a reasoning defect from 60B.97.
**Apparatus correction:** See section "Canonical harness gated apparatus (60B.99)" above for the fix.