test(harness): support gated live investigation continuation
This commit is contained in:
+28
-3
@@ -59,6 +59,29 @@
|
||||
|
||||
---
|
||||
|
||||
## Canonical harness gated apparatus (60B.99)
|
||||
|
||||
The canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`) now supports a two-phase gated investigation pattern:
|
||||
|
||||
**startOnly** — `FIXTURE_MODE=startOnly`
|
||||
- Makes exactly one `/api/cases/start` request
|
||||
- Writes the captured Start state (graph + selectedQuestion) to `.evidence-temp/continuation-start-only.json` (or path set by `CONTINUATION_FILE`)
|
||||
- Issues zero Update requests
|
||||
- Exits successfully
|
||||
|
||||
**continueOneUpdate** — `FIXTURE_MODE=continueOneUpdate CONTINUATION_ANSWER=<answer>`
|
||||
- Loads the persisted Start continuation state
|
||||
- Requires explicit answer (blocks with exit code 1 if missing)
|
||||
- Makes exactly one `/api/cases/update` using preserved Start state + explicit answer
|
||||
- Issues zero Start requests
|
||||
- Exits
|
||||
|
||||
**Normal mode** (`FIXTURE_MODE` unset) — unchanged. Start → configured Update loop still works identically to pre-60B.99.
|
||||
|
||||
This apparatus corrects the apparatus defect proven in 60B.98: the canonical harness can now stop after Start, allow external semantic inspection of the returned question, and later continue from that exact captured state with an explicitly chosen answer.
|
||||
|
||||
---
|
||||
|
||||
## Experiment 60B.95 result (2026-08-17)
|
||||
|
||||
**Classification: E — LIVE PATH DIVERGED**
|
||||
@@ -74,8 +97,10 @@ The live model selected nk6eyn2 ("exact monetary value of potential enterprise c
|
||||
|
||||
**Classification: E — START PATH DIVERGED**
|
||||
|
||||
The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison. Update 1 then hit a result_validation error: "Active unknown violates reasoning pattern consistency: \"n_fincomp12\" is explanation but active pattern is decision."
|
||||
The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison.
|
||||
|
||||
**What this establishes:** (1) The divergence from financial-comparison targets continues across multiple runs; (2) there exists an internal graph-state inconsistency (explanation-node vs decision-pattern mismatch in the Start/Update pipeline).
|
||||
**Valid evidence retained:** Start = S2 — DIFFERENT START (the live model diverged from the expected financial-comparison path).
|
||||
|
||||
**What this does NOT prove:** Whether a financially-comparison-aligned Start would progress coherently with a financial answer, or whether the pattern consistency issue is deterministic.
|
||||
**Update 1 evidence: DISCARDED.** The canonical harness auto-continued with its preconfigured `answers[0]`, so the Update occurred outside the experiment's semantic gate. This was an apparatus defect (60B.98) — the harness did not provide a post-Start stop gate at that time. The HTTP 500 is NOT established as a reasoning defect from 60B.97.
|
||||
|
||||
**Apparatus correction:** See section "Canonical harness gated apparatus (60B.99)" above for the fix.
|
||||
|
||||
+18
-11
@@ -47,26 +47,33 @@ Per critical gate rules, Fixed Answer 1 was not submitted because the Start sele
|
||||
|
||||
## Update 1 Result
|
||||
|
||||
**Reached:** YES
|
||||
**DISCARDED** — The canonical harness auto-continued with its preconfigured `answers[0]`, so the Update occurred outside the experiment's semantic gate. This evidence is invalid for 60B.97 conclusions.
|
||||
|
||||
**HTTP:** 500
|
||||
**Stage:** result_validation
|
||||
**First error:** `"Active unknown violates reasoning pattern consistency: \"n_fincomp12\" is explanation but active pattern is decision"`
|
||||
|
||||
This error indicates an internal graph-state inconsistency: node `n_fincomp12` was classified as an "explanation" type node but the active reasoning pattern requires a "decision" pattern. The Start path produced a situation graph with this structural violation.
|
||||
The HTTP 500 is NOT established as a reasoning defect from 60B.97.
|
||||
|
||||
## Classification: E — START PATH DIVERGED
|
||||
|
||||
The live engine selected enterprise-customer signing probability (S2) rather than a financial-comparison uncertainty, so Fixed Answer 1 was not submitted. Additionally, Update 1 hit a result_validation error indicating a reasoning-pattern consistency violation in the graph state.
|
||||
Valid 60B.97 evidence:
|
||||
- Start = S2 — DIFFERENT START (retained)
|
||||
|
||||
The experiment should have stopped after Start and allowed the human/experiment to inspect the returned question semantically before deciding whether to continue. The canonical harness did not provide this capability at time of 60B.97 execution, so the Update portion of 60B.97 is invalid evidence.
|
||||
|
||||
### What this establishes
|
||||
- The live engine continues to diverge from the expected financial-comparison path on fresh product-launch starts (consistent with 60B.95 pattern).
|
||||
- A graph-state inconsistency exists: `n_fincomp12` (explanation) is active while the reasoning pattern is decision — a structural validity issue in the Start/Update pipeline.
|
||||
|
||||
### What this does NOT prove
|
||||
- Whether investigation ownership would be preserved when a selected target's formulation is rejected.
|
||||
- Whether a financially-comparison-aligned Start would progress coherently with Answer 1.
|
||||
- Whether the explanation-vs-decision pattern inconsistency is deterministic or intermittent.
|
||||
- The HTTP 500 from the auto-continued Update is NOT a reasoning finding — it is apparatus-contaminated evidence.
|
||||
|
||||
## Apparatus correction (60B.99)
|
||||
|
||||
The canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`) now supports:
|
||||
- `startOnly` mode: exactly one Start, zero Updates, persisted continuation state on disk
|
||||
- `continueOneUpdate` mode: loads captured Start state, requires explicit answer, exactly one Update
|
||||
- Normal mode (FIXTURE_MODE unset) unchanged
|
||||
|
||||
This enables future live experiments to implement a semantic post-Start gate.
|
||||
|
||||
## Compliance Checklist
|
||||
- **Production code changed:** NO
|
||||
@@ -76,5 +83,5 @@ The live engine selected enterprise-customer signing probability (S2) rather tha
|
||||
- **Playwright runs:** 0
|
||||
|
||||
## Documentation
|
||||
- `docs/experiment-60b97.md` — created (this file)
|
||||
- `docs/current-handoff.md` — appended experiment result entry
|
||||
- `docs/experiment-60b97.md` — updated with apparatus correction note
|
||||
- `docs/current-handoff.md` — appended experiment result entry + apparatus note
|
||||
|
||||
Reference in New Issue
Block a user