robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: diagnose semantic-to-mutation action ownership

Browse Source
This commit is contained in:
robbond
2026-08-12 05:53:58 +01:00
parent f022d6f4ad
commit d7cb343838
2 changed files with 230 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6
View File
@@ -227,6 +227,12 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult
---
### Experiment 57J.64 — Semantic-to-Mutation Action Ownership Diagnosis (Read-Only Deterministic)
**Objective:** Why does the same explicit uncertainty sometimes produce graph structure and sometimes a semantic-only no-op? **Classification: E — MIXED.** Read-only architecture diagnosis across prompt contract, validator role, and deterministic engine boundaries. Three factors confirmed: (B) model noncompliance with complete contract — rule #6 "MUST" is clear but model occasionally emits zero mutation; (C) structured-field dependency gap — reliability correlates with supportCategory population because the deterministic fallback profile derives identical categories but feeds them only to validator cross-checks, not to action selection; (D) validator/recovery architecture gap — model owns all structural action decisions while code only rejects invalid output with no bounded repair path. No code changes. Confirmed: PROMPT-ONLY PATH EXHAUSTED. Next boundary is structured action-contract implementation (add `mutationIntent` field to output schema, validated deterministically). Full record in `docs/experiment-57j64.md`.
---
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.