robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(harness): preserve null-question start captures

Browse Source
This commit is contained in:
robbond
2026-08-18 07:11:27 +01:00
parent 5d0ce0ddd3
commit 55b7551739
4 changed files with 259 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+8
View File
@@ -57,6 +57,14 @@
- no production changes during the experiment
- verify that the live LLM responds consistently to the null-question state and continues investigation on ntpt9ki
## Apparatus correction: 60B.101 — null-question start capture
The canonical `startOnly` harness was corrected to accept successful Start responses with `selectedQuestion = null`. Previously, any successful Start returning no graph-backed question (legitimate outcome meaning "target remains active but no askable question available") caused the harness to block and fail.
**Change:** The harness now checks `success === true` + valid `situationGraph` as the sole gate for startOnly success. `selectedQuestion` is preserved exactly (including null) in the continuation state file without coercion.
**Impact on 60B.100:** The evidence from 60B.100 was captured via direct curl because the harness blocked on null-question Start. That evidence is now marked as apparatus-contaminated and provisional observation only.
---
## Canonical harness gated apparatus (60B.99)
docs/experiment-60b100.md
+13 -5
View File
@@ -183,7 +183,7 @@ The deterministic selector **does** override the model's reconstruction question
**NO** (harness scenario string reverted to original after capture)
## Harness changed:
**NO** (transient modification reverted; evidence captured via curl)
**NO at time of experiment.** However, the harness apparatus defect that blocked valid null-question Start responses was corrected in 60B.101: `scripts/reproduce-multi-turn-investigation.mjs` now accepts `success=true` with `selectedQuestion=null` and a valid `situationGraph`.
## Ollama calls beyond permitted count:
0
@@ -192,11 +192,19 @@ The deterministic selector **does** override the model's reconstruction question
YES
## Documentation updated:
`docs/experiment-60b100.md` created
`docs/current-handoff.md` appended
`docs/experiment-60b100.md` corrected (this apparatus)
`docs/current-handoff.md` appended with 60B.101 correction note
---
## Final git status
## Apparatus note on evidence validity (60B.101)
Clean — all evidence files removed, harness reverted.
The canonical `startOnly` harness blocked when the Start response returned `selectedQuestion = null`. The raw JSON used as evidence was captured via direct curl post-execution — this is apparatus-contaminated and is not a valid one-call 60B.100 experiment result.
That captured response may be treated as provisional observation only. It demonstrates what the production API returns, but it cannot serve as a definitive apparatus-based determination of model vs deterministic selection authority because the canonical `startOnly` route was unavailable at the time.
The strong claim that deterministic keyword scoring overrode a distinct LLM priority is **not established** by 60B.100 alone.
Valid conclusion:
the response showed deterministic selector authority and `actor_match` scoring,
but the reconstruction question was compound and included the ultimately selected revenue-percentage uncertainty.