robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: locate semantic-to-mutation contract gap

Browse Source
This commit is contained in:
robbond
2026-08-11 11:56:36 +01:00
parent b341c9cf2f
commit 77f5ea26d4
2 changed files with 190 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+18
View File
@@ -495,3 +495,21 @@ Rejected proposal snapshot: `answerMeaning.userSupportedMeaning` preserved both
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed.
### Experiment 57J.37 — Rejected Proposal Diagnostics: Semantic-to-Mutation Contract Gap (Read-Only Diagnosis)
**Objective:** Read-only analysis of whether the graph-update prompt/validator contract requires structural representation of newly introduced unresolved uncertainty, or whether an empty mutation with populated `answerMeaning` is permitted by the model contract and merely rejected later as a no-op.
**Method:** Analyzed prompt instructions (`lib/graph/prompt-builder.js`), schema defaults (`lib/graph/schema.js`), validator logic (`lib/graph/utils.js` line 868–885), application pipeline (`lib/graph/apply-proposal.js` line 3174, 3252–3270), and existing test coverage. No Ollama calls. No live API.
**Findings:**
- **Prompt contract is AMBIGUOUS:** Rule #6 requires inspecting for new uncertainty but rule #7 ("Add new unknown nodes only when...") is a restriction, not a requirement. Additional Guidance explicitly permits semantic-only proposals via `answerMeaning`.
- **Schema contract PERMITS the combination:** `graphUpdateSchema` allows populated `answerMeaning` + zero structural mutation (all array fields default to `[]`). No cross-field constraint exists.
- **Validator contract REJECTS it:** `hasMeaningfulChange` checks only structural fields (addedNodes, updatedNodes status/value changes, addedEdges, removedEdgeIds). `answerMeaning` is not considered meaningful change.
- **Test coverage NOT COVERED:** No test for "grounded answerMeaning introduces new unresolved uncertainty + zero structural changes." The closest tests verify schema validity of `{}` and validator rejection of all-empty arrays, but neither tests the populated `answerMeaning` case.
**Classification: E — MIXED.** Three independent contract boundaries contribute: (1) prompt ambiguity between inspection and materialization; (2) schema permissiveness vs validator rejection mismatch; (3) model receives permissive guidance that leads to a rejected downstream gate.
**Who owns the failure:** MIXED — Prompt Contract (ambiguity) + Validator Contract (schema/validator mismatch). Model does NOT own this failure.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
docs/experiment-57j37.md
+172
View File
@@ -0,0 +1,172 @@
# Experiment 57J.37 — Semantic-to-Mutation Contract Gap Diagnosis (Read-Only)
## Objective
Answer: **When `answerMeaning.userSupportedMeaning` clearly contains newly introduced unresolved uncertainty, does the current graph-update prompt/validator contract require the proposal to represent that uncertainty structurally, or is an empty mutation still permitted by the model contract and merely rejected later as a no-op?**
This is a read-only deterministic diagnosis. No Ollama calls. No live API. No production code changes. No test changes.
## Retained Meaning (fixed)
```
Before deciding on relocation, the user requires two specific pieces of evidence: verification that projected office savings are realistic, and assurance that the move will not materially increase the loss of key engineers.
```
With `possibleInference = null`.
## Starting HEAD
`b341c9c` — experiment: rerun guarded multi-turn progress cleanly
---
## Part 1 — Prompt Contract
### Relevant new-uncertainty instructions in `lib/graph/prompt-builder.js`
| # | Instruction (verbatim excerpt) | Classification |
|---|-------------------------------|----------------|
| 6 | "Then inspect the answer for newly introduced consequential uncertainty." | MUST |
| 7 | "Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case." | MUST (restrictive) / AMBIGUOUS (obligative) |
| 8 | "Add at most 3 new unknown nodes." | MUST |
| 9 | "Every new unknown must be directly traceable to the user's answer and its description must state why that uncertainty matters." | MUST |
| 9a | "...explicitly include a short why-it-matters clause..." | MUST |
| 11 | "Do not add duplicate unknowns." | MUST |
| 16 | "If consequential unresolved unknowns exist, selectedQuestion **may** identify one valid candidate unknown..." | MAY |
| Additional-Guidance-1 | "If the answer only clarifies an existing unknown, prefer updatedNodes and resolvedUnknownNodeIds over creating duplicate nodes." | SHOULD (prefers) |
| Additional-Guidance-2 | "If you add a new unknown, do not leave it floating: connect it with an added edge..." | MUST (conditional) |
| Additional-Guidance-3 | "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved." | MAY (permits semantic-only) |
| Rule-21 | "Use empty arrays when there are no changes in a category." | MUST (defaulting) |
### Does prompt explicitly require structural representation of newly introduced unresolved uncertainty?
**PARTIAL**
**Why:** Instruction #6 creates an inspection obligation ("inspect the answer for newly introduced consequential uncertainty"). Instructions #7–#9 describe what to do *when* new unknowns are found, but #7 uses "Add new unknown nodes only when..." which is grammatically a **restriction** (you may not add unless...) rather than a clear **requirement** (you must add when...). Rule 16 uses "may" for selectedQuestion. The Additional Guidance explicitly permits semantic-only output ("Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved"). Thus, while the model is told to *inspect* for new uncertainty and shown what to do with it if found, there is no explicit MUST that forces structural materialization when new consequential uncertainty is detected.
---
## Part 2 — Schema Contract
**SCHEMA VALID**
The `graphUpdateSchema` (lib/graph/schema.js, line 178) permits:
```json
{
"answerMeaning": { "userSupportedMeaning": "<text>", ... },
"updatedNodes": [],
"resolvedUnknownNodeIds": [],
"addedNodes": [],
"addedEdges": []
}
```
All array fields have `.default([])`, and `answerMeaning` has `.default(null)` (nullable). The schema imposes no cross-field constraint requiring that a populated `answerMeaning` must be accompanied by non-empty structural mutation fields. Test at line 156-158 confirms empty object `{}` passes validation.
---
## Part 3 — Validator Contract
### Function: `validateGraphUpdate(graph, update)` in `lib/graph/utils.js`, lines 847–894
### Exact no-op condition (lines 868–885):
```javascript
const statusChanged = update.updatedNodes.some(
(u) => u.previousStatus !== null && u.newStatus !== u.previousStatus,
);
const valueChanged = update.updatedNodes.some(
(u) => (u.previousValue ?? null) !== (u.newValue ?? null),
);
const hasMeaningfulChange =
update.addedNodes.length > 0 ||
statusChanged ||
valueChanged ||
update.addedEdges.length > 0 ||
update.removedEdgeIds.length > 0;
if (!hasMeaningfulChange) {
errors.push("Update contains no meaningful change");
}
```
### Does `answerMeaning` count as meaningful change?
**NO.** The validator checks only structural fields. `answerMeaning` is not referenced in the `hasMeaningfulChange` computation.
### Is rejection of semantic-only no-op proposal correct under current graph semantics?
**YES**, under the *current* semantics where the graph is a strict mutation ledger and `answerMeaning` is metadata, not a structural change. The rejection is internally consistent: the graph structure didn't change, so the update is a no-op from the graph's perspective.
---
## Part 4 — Responsibility Boundary
### A — MODEL FAILED AN EXPLICIT CONTRACT
**NO.** No explicit "MUST materialize new consequential uncertainty as unknown nodes" instruction exists in the prompt. The model's inspection at rule #6 was fulfilled (it extracted meaning), but there is no mandatory bridge from "inspected" to "structurally represented."
### B — PROMPT CONTRACT IS AMBIGUOUS
**YES.** Rule #7 ("Add new unknown nodes only when...") reads as a restriction rather than a requirement. Instructions #8-#9 describe constraints *on* additions but don't mandate additions. Additional Guidance explicitly permits semantic-only proposals ("Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved").
### C — SCHEMA/VALIDATOR CONTRACT IS INCONSISTENT
**YES.** The schema semantically allows populated `answerMeaning` + zero mutation. The Additional Guidance tells the model it can use `answerMeaning` for this purpose. But the validator later rejects this exact combination as a no-op. The model receives permissive guidance that leads to a rejected outcome through a gate it cannot anticipate (no semantic meaning = meaningful change).
### D — EXISTING GRAPH MAY ALREADY CONTAIN THE MEANING
**PARTIAL.** The contract instructs: "Do not add duplicate unknowns" and "prefer updatedNodes... over creating duplicate nodes." If the cold-start graph already contained unknowns for these two evidence dimensions, an empty mutation would be defensible. However, without inspecting the 57J.36 cold-start graph state, this possibility cannot be confirmed or ruled out. The retained experiment record (57J.34) shows that cold-start produced a "single merged generic unknown" rather than two distinct evidence-dimension unknowns — suggesting partial overlap is possible but not complete.
---
## Part 5 — Test Coverage
### Existing test for: grounded answerMeaning introduces new unresolved uncertainty + proposal makes zero structural changes
**NOT COVERED**
The closest tests are:
1. `schema.test.js` line 156: "validates empty update (no-op proposal)" — validates `{}` passes the **schema** gate (confirms schema validity)
2. `utils.test.js` line 932: "rejects update with no meaningful change" — tests that all-empty structural arrays are rejected by the **validator**
3. `apply-proposal.test.js` line 705: same as #2 but via the application pipeline
None of these test the specific case of **populated `answerMeaning` + zero structural mutation**. The apply-proposal no-op test (line 705) uses an update with `updatedNodes` containing a null-status-change entry but **no `answerMeaning`** at all.
---
## Classification: E — MIXED
### Why:
Three independent contract boundaries contribute to the failure:
1. **Prompt contract (B):** Ambiguity between "inspect for new uncertainty" and "must materialize new uncertainty." Rule #7 is a restrictive clause, not an obligatory one. Additional Guidance explicitly permits semantic-only proposals.
2. **Schema contract (C — permissive):** Schema accepts the combination that later gets rejected. The test confirms `{}` passes schema validation, meaning populated `answerMeaning` + empty arrays is trivially schema-valid.
3. **Validator contract (C — rejecting):** The validator's "meaningful change" check explicitly excludes `answerMeaning`. The model follows permissive guidance and hits a downstream gate that contradicts the guidance.
The model is caught in a triple-bind: it correctly extracts meaning (as instructed), uses it exactly as permitted by the schema, receives permissive guidance about semantic-only proposals, and then gets rejected by an invariant not communicated to it.
---
## Who currently owns the failure: MIXED
- **Prompt Contract** owns the ambiguity between inspection and materialization
- **Validator Contract** owns the mismatch between schema-permitted inputs and validator-rejected outputs
- **Model** does NOT own this failure — no explicit instruction was violated
## What 57J.37 now legitimately establishes:
1. The prompt contract is ambiguous on whether newly introduced consequential uncertainty must be structurally materialized.
2. The schema contract explicitly permits populated `answerMeaning` + zero structural mutation (all array fields default to `[]`).
3. The validator contract does NOT consider `answerMeaning` as a meaningful change — only structural graph mutations count.
4. There is no existing test that covers the exact case of "grounded answerMeaning introduces new unresolved uncertainty + zero structural changes."
## What it does NOT establish:
1. Whether the cold-start graph from 57J.36 already contained nodes matching these two evidence dimensions (D possibility unverified).
2. Which single classification (B vs C) is primary — both boundaries are materially implicated.
3. A specific fix direction — this diagnoses the gap but does not prescribe resolution.
---
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434
Production code changed: NO
Prompt changed: NO
Tests changed: NO
Dev server disturbed: NO
Ollama calls made: 0