robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: inspect rejected proposal live variance

Browse Source
This commit is contained in:
robbond
2026-08-11 10:03:35 +01:00
parent 48e9bcf3eb
commit a00f7b170d
2 changed files with 162 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+5
View File
@@ -389,6 +389,11 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
### Experiment 57J.31 — Rejected Proposal Diagnostics Integration
**Objective:** Address the blocking diagnostic visibility gap from Experiment 57J.30 — when `applyValidatedProposal` fails at `proposal_compatibility`, no pre-validation proposal fields are exposed to developers, making causal attribution of rejection outcomes impossible. **Classification: E — FIX VALIDATED.** Added a compact `rejectedProposalSnapshot` to the orchestrator's diagnostics object in the `!applicationResult.success` path (lines ~690–725 of `lib/graph/orchestrator.js`). The snapshot is stage-gated (only for `proposal_compatibility`), contains only key proposal fields (`answerMeaning.userSupportedMeaning`, `answerMeaning.possibleInference`, `updatedNodes[].{nodeId,newValue}`, `resolvedUnknownNodeIds[]`, `addedNodes[].{id,kind,label,description,parentId,dependsOn,affects,childIds}`, `addedEdges[].{fromNodeId,toNodeId,relationship}`), and excludes raw model response/prompt content. No validation or mutation behavior changed — only the diagnostic surface is expanded. **Tested:** 7 new snapshot tests (all pass), 2 apply-proposal regression tests (both pass), 13 route-layer tests (all pass) = 84 passed, 0 failed by this change. Pre-existing orchestrator failure confirmed independent of this change. What remains unproven: whether the actual rejected trial from 57J.30 contained stronger `userSupportedMeaning` language vs. different structural elements — requires re-running Experiment 57J.30 now that the snapshot is visible in the HTTP response body. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 0 live calls needed (deterministic instrumentation). Production code changed: `lib/graph/orchestrator.js` only.
### Experiment 57J.32 — Inspect Rejected Proposal Live Variance (EXPERIMENT INTERRUPTED)
**Protocol breach: YES.** The execution harness drifted beyond the agreed budget. Only valid bounded evidence from the first 3 trials is retained in `docs/experiment-57j32.md`. All post-trial activity is excluded from conclusions. **Classification: F — MULTIPLE DIFFERENCES.** Using `rejectedProposalSnapshot` directly, accepted vs rejected proposals differ in both answerMeaning framing (prescriptive "decision is conditional on" vs neutral "requires evidence") AND structural content (accepted = 2 addedNodes + 2 addedEdges; rejected = 0 addedNodes + 0 addedEdges). Neither single cause can be independently isolated. The rejected trial's userSupportedMeaning did not faithfully preserve the raw answer: "The decision is conditional on" prescribes what the *decision* requires rather than reporting the user's information need ("Before deciding, I need..."). **rejectedProposalSnapshot worked as designed** (57J.31 verified). Status: interrupted mid-execution; evidence retained in `docs/experiment-57j32.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code unchanged.
### Experiment 57J.30 — Proposal-Boundary Live Variance
**Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.