robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate structural action contract live

Browse Source
This commit is contained in:
robbond
2026-08-12 08:04:41 +01:00
parent 1b3bbd59aa
commit c899ad620c
2 changed files with 156 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6
View File
@@ -233,6 +233,12 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult
---
### Experiment 57J.69 — structuralActionRequired Live Population and Contract Enforcement
**Classification: C — TRUE/NO-MUTATION CONTRADICTION.** One start + one update (fixed scenario: "We are considering relocating the engineering team to reduce operating costs." / fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."). Start HTTP 200 (5 nodes, 3 edges). Update 1 returned HTTP 422 at `proposal_compatibility`. Model populated `structuralActionRequired = true` and extracted `userSupportedMeaning` ("User is unsure...") with supportCategory="uncertain". Proposal contained only `[updatedNodes: [{nodeId: nqx00rq, newValue: null}]]` — no added nodes, no added edges, meaningfully zero mutation. Validator rejected with dual errors: new contract rule ("structuralActionRequired is true but proposal contains no graph mutation") + legacy guard ("answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"). **What this establishes:** (1) `structuralActionRequired` IS populated for savings-realism uncertainty; (2) v0.23 validator ENFORCES the strict declaration-to-mutation contract; (3) rejection fires at correct stage before any mutation. **What remains unproven:** whether model can produce true+mutation in a single attempt, stability across runs, prompt enforcement adequacy. The remaining gap: model knows it should act structurally but fails to produce the mutation itself — same prompt-enforcement gap from 57J.64. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j69.md`.
---
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.