robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: choose semantic-to-mutation contract fix

Browse Source
This commit is contained in:
robbond
2026-08-11 12:03:16 +01:00
parent 77f5ea26d4
commit 3b868b266e
2 changed files with 282 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+33
View File
@@ -513,3 +513,36 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
### Experiment 57J.38 — Semantic-to-Mutation Contract Fix Selection
**Objective:** Choose the smallest safe contract change preventing faithful semantic-only no-op proposals (populated `answerMeaning` with consequential uncertainty + zero structural mutation).
**Three options evaluated:**
- **A (prompt-only):** Single MUST rule in Additional Guidance clarifying structural mutation requirement. Fixes ambiguity but relies entirely on model compliance.
- **B (prompt + validator):** Same prompt rule PLUS a deterministic validator check targeting `userSupportedMeaning` text + empty structural fields. Produces specific actionable error.
- **C (schema cross-field constraint):** Zod `.refine()` rejecting any populated `answerMeaning` object with zero structural mutation.
**Key finding on Option C:** Breaks Case 5 (possibleInference only). Schema cannot distinguish "new consequential meaning" from "inference-only" without semantic analysis. Any populated answerMeaning object triggers rejection regardless of content type.
**Controlled case results summary:**
- Case 1 (genuinely new uncertainty): All options reject as expected. B has best diagnostic visibility.
- Case 2 (already represented): All options correctly reject.
- Case 3 (update/resolve existing): All options correctly allow structural update to existing node.
- Case 4 (answerMeaning null): All options preserve existing behavior.
- Case 5 (possibleInference only): A✓ / B✓ / C✗ (breaks — schema sees populated object, cannot distinguish inference from meaning).
**Recommended option: B — PROMPT + VALIDATOR CONTRACT**
Why: Fixes 57J.36 completely (prompt ambiguity + enforcement gap). No new semantic classifier needed. Preserves provider-agnostic design. Does not break valid cases (null answerMeaning, possibleInference-only). Specific error message provides actionable diagnostic where option A relies entirely on model compliance (which the evidence from 57J.36 shows is unreliable for this pattern).
Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
**READY FOR BOUNDED IMPLEMENTATION: YES**
Exact implementation boundary:
1. One MUST rule in prompt Additional Guidance (replaces line 132 of prompt-builder.js)
2. One deterministic check in `validateGraphUpdate()` after `hasMeaningfulChange` (utils.js)
3. Six regression tests: populated-meaning-zero-mutation rejection, prompt text verification, null-answerMeaning preserved, possibleInference-only not forced, update-existing-node valid, resolve-path valid
---