robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add Experiment 57J.31 entry to handoff

Browse Source
This commit is contained in:
robbond
2026-08-11 08:34:51 +01:00
parent 0348921542
commit 48e9bcf3eb
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3
View File
@@ -386,6 +386,9 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer ("Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers.") through the production `startCase()` → `updateCase()` path. **key finding:** `userSupportedMeaning` was null/empty in ALL 3 trials — no semantic meaning was extracted by the model. Yet admission outcomes diverged: Trial 2 (start with 7 nodes) admitted both unknowns; Trials 1 & 3 (start with 6 nodes) rejected at `proposal_compatibility` with identical "stronger reasoning category" errors despite null diagnostics. This confirms that start graph quality (6 vs 7 nodes cold-start variance) directly affects admission outcomes, and when `userSupportedMeaning` is empty the gate may still process hidden semantic fields. The admission variance cannot be explained by upstream model representation because no meaningful semantic content was produced in any trial — instability is downstream of model representation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.
### Experiment 57J.31 — Rejected Proposal Diagnostics Integration
**Objective:** Address the blocking diagnostic visibility gap from Experiment 57J.30 — when `applyValidatedProposal` fails at `proposal_compatibility`, no pre-validation proposal fields are exposed to developers, making causal attribution of rejection outcomes impossible. **Classification: E — FIX VALIDATED.** Added a compact `rejectedProposalSnapshot` to the orchestrator's diagnostics object in the `!applicationResult.success` path (lines ~690–725 of `lib/graph/orchestrator.js`). The snapshot is stage-gated (only for `proposal_compatibility`), contains only key proposal fields (`answerMeaning.userSupportedMeaning`, `answerMeaning.possibleInference`, `updatedNodes[].{nodeId,newValue}`, `resolvedUnknownNodeIds[]`, `addedNodes[].{id,kind,label,description,parentId,dependsOn,affects,childIds}`, `addedEdges[].{fromNodeId,toNodeId,relationship}`), and excludes raw model response/prompt content. No validation or mutation behavior changed — only the diagnostic surface is expanded. **Tested:** 7 new snapshot tests (all pass), 2 apply-proposal regression tests (both pass), 13 route-layer tests (all pass) = 84 passed, 0 failed by this change. Pre-existing orchestrator failure confirmed independent of this change. What remains unproven: whether the actual rejected trial from 57J.30 contained stronger `userSupportedMeaning` language vs. different structural elements — requires re-running Experiment 57J.30 now that the snapshot is visible in the HTTP response body. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 0 live calls needed (deterministic instrumentation). Production code changed: `lib/graph/orchestrator.js` only.
### Experiment 57J.30 — Proposal-Boundary Live Variance
**Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.