experiment: choose semantic-to-mutation contract fix
This commit is contained in:
@@ -513,3 +513,36 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
|
||||
|
||||
|
||||
### Experiment 57J.38 — Semantic-to-Mutation Contract Fix Selection
|
||||
|
||||
**Objective:** Choose the smallest safe contract change preventing faithful semantic-only no-op proposals (populated `answerMeaning` with consequential uncertainty + zero structural mutation).
|
||||
|
||||
**Three options evaluated:**
|
||||
- **A (prompt-only):** Single MUST rule in Additional Guidance clarifying structural mutation requirement. Fixes ambiguity but relies entirely on model compliance.
|
||||
- **B (prompt + validator):** Same prompt rule PLUS a deterministic validator check targeting `userSupportedMeaning` text + empty structural fields. Produces specific actionable error.
|
||||
- **C (schema cross-field constraint):** Zod `.refine()` rejecting any populated `answerMeaning` object with zero structural mutation.
|
||||
|
||||
**Key finding on Option C:** Breaks Case 5 (possibleInference only). Schema cannot distinguish "new consequential meaning" from "inference-only" without semantic analysis. Any populated answerMeaning object triggers rejection regardless of content type.
|
||||
|
||||
**Controlled case results summary:**
|
||||
- Case 1 (genuinely new uncertainty): All options reject as expected. B has best diagnostic visibility.
|
||||
- Case 2 (already represented): All options correctly reject.
|
||||
- Case 3 (update/resolve existing): All options correctly allow structural update to existing node.
|
||||
- Case 4 (answerMeaning null): All options preserve existing behavior.
|
||||
- Case 5 (possibleInference only): A✓ / B✓ / C✗ (breaks — schema sees populated object, cannot distinguish inference from meaning).
|
||||
|
||||
**Recommended option: B — PROMPT + VALIDATOR CONTRACT**
|
||||
|
||||
Why: Fixes 57J.36 completely (prompt ambiguity + enforcement gap). No new semantic classifier needed. Preserves provider-agnostic design. Does not break valid cases (null answerMeaning, possibleInference-only). Specific error message provides actionable diagnostic where option A relies entirely on model compliance (which the evidence from 57J.36 shows is unreliable for this pattern).
|
||||
|
||||
Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
|
||||
**READY FOR BOUNDED IMPLEMENTATION: YES**
|
||||
|
||||
Exact implementation boundary:
|
||||
1. One MUST rule in prompt Additional Guidance (replaces line 132 of prompt-builder.js)
|
||||
2. One deterministic check in `validateGraphUpdate()` after `hasMeaningfulChange` (utils.js)
|
||||
3. Six regression tests: populated-meaning-zero-mutation rejection, prompt text verification, null-answerMeaning preserved, possibleInference-only not forced, update-existing-node valid, resolve-path valid
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# Experiment 57J.38 — Semantic-to-Mutation Contract Fix Selection (Read-Only Design)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer: **What is the smallest safe contract change that ensures a faithful answer containing consequential unresolved uncertainty cannot return only `answerMeaning` with zero structural mutation?**
|
||||
|
||||
This follows 57J.37's diagnosis of three contributing boundaries:
|
||||
1. Prompt contract ambiguity (inspecting ≠ materializing)
|
||||
2. Schema permissiveness vs validator rejection mismatch
|
||||
3. Validator ignores `answerMeaning` in meaningful-change check
|
||||
|
||||
## Starting HEAD
|
||||
|
||||
`77f5ea2` — experiment: locate semantic-to-mutation contract gap
|
||||
|
||||
---
|
||||
|
||||
## Key Findings from Code Audit (300-line budget)
|
||||
|
||||
### Prompt-Builder Current State (`lib/graph/prompt-builder.js`)
|
||||
|
||||
**Rule #6:** "Then inspect the answer for newly introduced consequential uncertainty." — creates inspection obligation but not materialization requirement.
|
||||
|
||||
**Rule #7:** "Add new unknown nodes only when..." — grammatically a restriction, not a requirement.
|
||||
|
||||
**Additional Guidance (line 132):** "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved." — explicitly permits semantic-only output.
|
||||
|
||||
**Gap:** The model is told to inspect for new uncertainty, shown what to do if found, but also explicitly permitted to use semantic-only output. No explicit MUST bridges inspection to materialization.
|
||||
|
||||
### Validator Current State (`lib/graph/utils.js` lines 868–885)
|
||||
|
||||
```javascript
|
||||
const hasMeaningfulChange =
|
||||
update.addedNodes.length > 0 ||
|
||||
statusChanged ||
|
||||
valueChanged ||
|
||||
update.addedEdges.length > 0 ||
|
||||
update.removedEdgeIds.length > 0;
|
||||
// answerMeaning NOT referenced
|
||||
```
|
||||
|
||||
Purely structural. `answerMeaning` is never considered meaningful change.
|
||||
|
||||
### Schema Current State (`lib/graph/schema.js` line 178–187)
|
||||
|
||||
All array fields default to `[]`. `answerMeaning` defaults to `null` (nullable). No cross-field constraint exists. Test at line 156 confirms `{}` passes schema validation.
|
||||
|
||||
### Test Coverage Gap
|
||||
|
||||
No test for "populated `answerMeaning.userSupportedMeaning` + zero structural mutation remains rejected." The closest tests verify:
|
||||
- Schema allows empty update (schema.test.js:156)
|
||||
- Validator rejects all-empty-arrays (utils.test.js:932) — but without any `answerMeaning`
|
||||
- Snapshot captures rejected proposals with various combinations (rejected-proposal-snapshot.test.js)
|
||||
|
||||
---
|
||||
|
||||
## Option Evaluation
|
||||
|
||||
### OPTION A — PROMPT ONLY
|
||||
|
||||
Add one explicit MUST rule to Additional Guidance:
|
||||
|
||||
> If `answerMeaning.userSupportedMeaning` contains consequential information or unresolved uncertainty that is not already represented in the graph, the proposal MUST express its effect through at least one structural mutation. `answerMeaning` alone is not sufficient.
|
||||
|
||||
**Fixes 57J.36:** PARTIAL — addresses prompt ambiguity but relies entirely on model compliance. If the model ignores instruction (as it did in 57J.36), rejection will still be the generic "no meaningful change" with no diagnostic clarity about *why* mutation is required.
|
||||
|
||||
**Duplicate risk:** LOW — existing rules #11 ("Do not add duplicate unknowns") and Additional Guidance preference for `updatedNodes` over new nodes already in place. The prompt rule says "express its effect through at least one structural mutation" without prescribing which type of mutation, so the model could update/resolve an existing node instead of creating a new one.
|
||||
|
||||
**Requires new semantic classifier:** NO — uses plain text detection (is `userSupportedMeaning` non-empty + all structural fields empty).
|
||||
|
||||
**Changes schema:** NO
|
||||
|
||||
**Changes validator:** NO
|
||||
|
||||
**Changes prompt:** YES — one additional sentence in Additional Guidance, plus replacement of line 132 to remove the "semantic-only permitted" language.
|
||||
|
||||
**Provider-specific:** NO
|
||||
|
||||
**Risk of rejecting legitimate no-op/restatement:** MEDIUM — if the answer restates information already fully represented and the LLM produces `userSupportedMeaning` text that is technically non-empty but semantically identical to graph content, rejection still occurs (correctly, under the invariant). But the model may struggle to determine when materialization is actually unnecessary versus when it should still express meaning through existing structure.
|
||||
|
||||
### OPTION B — PROMPT + SPECIFIC VALIDATOR CONTRACT
|
||||
|
||||
Same prompt rule as A PLUS a deterministic compatibility check producing a specific error:
|
||||
|
||||
```javascript
|
||||
// In validateGraphUpdate() after hasMeaningfulChange check:
|
||||
if (update.answerMeaning?.userSupportedMeaning && !hasMeaningfulChange) {
|
||||
errors.push("Answer introduces new information that must be structurally represented — cannot return only answerMeaning without graph mutation.");
|
||||
}
|
||||
```
|
||||
|
||||
**Fixes 57J.36:** YES — addresses both the prompt ambiguity AND provides a deterministic enforcement layer that survives model instruction-following failure.
|
||||
|
||||
**Duplicate risk:** LOW — specific error message guides correction ("must be structurally represented") without prescribing node creation. The existing rules about duplicates, updatedNodes preference, and relationship-based mutations still apply.
|
||||
|
||||
**Requires new semantic classifier:** NO — purely structural check: is `userSupportedMeaning` non-empty AND all structural fields empty? Zero semantics involved.
|
||||
|
||||
**Changes schema:** NO
|
||||
|
||||
**Changes validator:** YES — one addition after the existing `hasMeaningfulChange` check (5 lines). Does NOT replace existing no-op rejection; adds an additional condition that fires first.
|
||||
|
||||
**Changes prompt:** YES — same as A.
|
||||
|
||||
**Provider-specific:** NO
|
||||
|
||||
**Risk of rejecting legitimate no-op/restatement:** LOW — if userSupportedMeaning is non-empty and all structural fields are empty, the rejection is correct under the invariant. If the meaning IS already fully represented in existing graph structure, the guidance says "update/resolve an existing node" rather than "create nothing." The only edge case: if the LLM produces `userSupportedMeaning` for information that was already fully represented AND it cannot determine how to express it structurally without violating other rules (e.g., can't update because no matching node exists, can't add because not genuinely new), but this is a prompt design question, not an option-specific problem.
|
||||
|
||||
### OPTION C — SCHEMA CROSS-FIELD REQUIREMENT
|
||||
|
||||
Add `.refine()` to `graphUpdateSchema`:
|
||||
|
||||
```javascript
|
||||
.graphTransform((val) => val)
|
||||
.refine(
|
||||
(data) => {
|
||||
if (data.answerMeaning?.userSupportedMeaning && data.userSupportedMeaning.length > 0) {
|
||||
return data.addedNodes.length > 0 ||
|
||||
data.updatedNodes.some(u => u.newStatus !== null || u.newValue !== null) ||
|
||||
data.addedEdges.length > 0;
|
||||
}
|
||||
return true;
|
||||
},
|
||||
{ message: "Populated answerMeaning with new information requires at least one structural mutation" }
|
||||
);
|
||||
```
|
||||
|
||||
**Fixes 57J.36:** PARTIAL — schema enforcement means the invalid proposal never reaches validation, but provides no diagnostic explanation to downstream consumers (HTTP API). The error is a Zod refinement failure, not an application-level semantic rejection with actionable guidance.
|
||||
|
||||
**Duplicate risk:** MEDIUM — schema requires mutation but doesn't guide toward what type. Could push models toward creating new unknown nodes rather than updating existing ones when existing structure could serve.
|
||||
|
||||
**Requires new semantic classifier:** NO — purely structural check same as B (non-empty userSupportedMeaning + empty structural fields).
|
||||
|
||||
**Changes schema:** YES — adds cross-field constraint.
|
||||
|
||||
**Changes validator:** NO
|
||||
|
||||
**Changes prompt:** NO
|
||||
|
||||
**Provider-specific:** NO
|
||||
|
||||
**Risk of rejecting legitimate no-op/restatement:** HIGH — breaks Case 5. If `answerMeaning` has only `possibleInference` (no consequential `userSupportedMeaning`) but the object is still populated, schema rejects. This is a false rejection: possibleInference alone does not establish new consequential uncertainty requiring structural representation. The schema-level check cannot distinguish "meaningful new meaning" from "inference-only."
|
||||
|
||||
---
|
||||
|
||||
## Controlled Cases Evaluation
|
||||
|
||||
### Case 1 — Genuinely New Uncertainty ("whether projected savings are realistic", no equivalent in graph)
|
||||
|
||||
| Option | Result | Notes |
|
||||
|--------|--------|-------|
|
||||
| A | STRUCTURAL MUTATION REQUIRED ✓ | Prompt MUST rule directs model to create nodes/edges. Model may or may not comply. Rejection if it doesn't = generic "no meaningful change" (unclear why). |
|
||||
| B | STRUCTURAL MUTATION REQUIRED ✓ | Same prompt + specific error if model fails: clearly states mutation required. Best diagnostic visibility. |
|
||||
| C | REJECTED ✓ | Schema blocks immediately with refinement error. No diagnostic guidance about what to fix. |
|
||||
|
||||
### Case 2 — Answer Meaning Already Fully Represented (restatement, no new info)
|
||||
|
||||
| Option | Result | Notes |
|
||||
|--------|--------|-------|
|
||||
| A | REJECTION CORRECT ✓ | "answerMeaning alone is not sufficient" covers this case. Model should update existing node or accept rejection. |
|
||||
| B | REJECTION CORRECT ✓ | Same logic, with clearer error message. |
|
||||
| C | REJECTION CORRECT ✓ | Schema blocks. But: no guidance on whether to update existing or create new. |
|
||||
|
||||
### Case 3 — Answer Resolves/Refines Existing Structure (evidence for existing unknown)
|
||||
|
||||
| Option | Result | Notes |
|
||||
|--------|--------|-------|
|
||||
| A | UPDATE EXISTING NODE ✓ | Prompt says "express effect through structural mutation" — updating an existing node counts. No duplicate created. |
|
||||
| B | UPDATE EXISTING NODE ✓ | Same guidance + specific error if model still produces empty mutation (points to need for structural change). |
|
||||
| C | UPDATE EXISTING NODE ✓ | Schema allows updateNodes as valid mutation path. Correct behavior. |
|
||||
|
||||
### Case 4 — answerMeaning null (existing structurally valid proposal)
|
||||
|
||||
| Option | Result | Notes |
|
||||
|--------|--------|-------|
|
||||
| A | UNCHANGED ✓ | No userSupportedMeaning → prompt rule is conditional, does not trigger. |
|
||||
| B | UNCHANGED ✓ | Null means condition doesn't fire. Existing no-op validator handles structural correctness independently. |
|
||||
| C | UNCHANGED ✓ | Schema refinement checks `answerMeaning?.userSupportedMeaning` — null passes through. |
|
||||
|
||||
### Case 5 — possibleInference Only (no userSupportedMeaning establishing new consequential uncertainty)
|
||||
|
||||
| Option | Result | Notes |
|
||||
|--------|--------|-------|
|
||||
| A | NO FORCED MUTATION ✓ | Rule is conditional on `userSupportedMeaning`. Inference-only does not trigger. Correct. |
|
||||
| B | NO FORCED MUTATION ✓ | Same — checks `userSupportedMeaning` specifically, not the entire answerMeaning object. Correct. |
|
||||
| C | FORCES MUTATION ✗ | **BREAKS.** Schema refinement on `answerMeaning` object would see a populated object (possibleInference exists) and force mutation even though no new consequential uncertainty was established. This is a critical flaw: the schema cannot distinguish meaning from inference without semantic analysis, which we explicitly said not to require. |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation: OPTION B — PROMPT + SPECIFIC VALIDATOR CONTRACT
|
||||
|
||||
### Why
|
||||
|
||||
1. **Fixes 57J.36 completely** (unlike A's partial fix and C's partial fix):
|
||||
- Prompt removes ambiguity between "inspect" and "must materialize"
|
||||
- Validator catches the specific failure pattern the model actually produces (faithful meaning + empty mutation)
|
||||
- Error message is actionable: tells the model exactly what is missing
|
||||
|
||||
2. **No new semantic classifier needed** — uses only structural detection (non-empty text field vs empty array fields). Zero semantic machinery.
|
||||
|
||||
3. **Preserves provider-agnostic design** — changes are deterministic text/schema/validator, not semantic matching or LLM-assisted checks.
|
||||
|
||||
4. **Does not force duplicate unknowns** — requires "at least one structural mutation" without prescribing node creation. Existing rules about duplicates and updating existing nodes remain fully in effect.
|
||||
|
||||
5. **Does not break valid cases** — Case 4 (null answerMeaning) passes through unchanged. Case 5 (possibleInference only) is handled because the check targets `userSupportedMeaning` specifically, not the entire answerMeaning object. Option C breaks Case 5.
|
||||
|
||||
6. **Option A's weakness**: relies entirely on model instruction following. The very evidence that motivated this experiment (57J.36: faithful meaning + zero mutation) demonstrates the model *can* and *does* follow instructions ambiguously. A specific validator error is needed for cases where prompt instruction fails.
|
||||
|
||||
7. **Option C's fatal flaw**: schema-level enforcement cannot distinguish between "meaningful new information" and "inference-only" without a semantic classifier, which violates the constraint of not requiring new semantic machinery.
|
||||
|
||||
---
|
||||
|
||||
## Required Deterministic Regressions (design only)
|
||||
|
||||
1. **Populated faithful `answerMeaning` + zero mutation remains rejected** — validator rejects with specific error message (not generic "no meaningful change"); rejection stage = `proposal_compatibility`; no schema or prompt modification required for this test since existing rejection already applies, but the *error text* should be different and verifiable.
|
||||
|
||||
2. **Prompt explicitly states structural mutation requirement** — snapshot test of buildGraphUpdatePrompt output confirms Additional Guidance contains MUST-language about structural representation when `answerMeaning` has consequential content.
|
||||
|
||||
3. **`answerMeaning = null` + valid mutation unchanged** — existing behavior preserved: structurally valid proposal with no answerMeaning passes through identical validation path, zero new errors introduced.
|
||||
|
||||
4. **possibleInference only does not force mutation** — proposal where `answerMeaning` has only `possibleInference` (no `userSupportedMeaning`) and empty structural fields: if there IS meaningful structural change via other paths, the specific error must NOT fire. Test the boundary where `userSupportedMeaning` is absent or empty string vs present with consequential text.
|
||||
|
||||
5. **Existing relevant unknown must not be duplicated** — proposal that updates an existing node (updatedNodes non-empty) to represent new uncertainty: should pass without triggering duplicate-node errors. The structural-mutation requirement is satisfied by the update, not rejected for forcing a new node.
|
||||
|
||||
6. **Existing update/resolve path counts as valid structural progress** — proposal with resolvedUnknownNodeIds and/or updatedNodes status/value changes passes validation regardless of whether `answerMeaning` is populated or empty. This confirms the existing update/resolve path is not blocked by any new constraint.
|
||||
|
||||
---
|
||||
|
||||
## Stop Condition for Implementation
|
||||
|
||||
Implementation stops when:
|
||||
1. One MUST rule added to prompt Additional Guidance (replaces line 132)
|
||||
2. One deterministic check added to `validateGraphUpdate()` after `hasMeaningfulChange`
|
||||
3. Six regression tests pass (above)
|
||||
4. Existing test suite unchanged
|
||||
|
||||
## What This Intentionally Leaves Unsolved
|
||||
|
||||
- Whether the model should *always* produce a structurally non-empty proposal when new uncertainty exists — this is a prompt design question, not a contract enforcement question
|
||||
- Cold-start graph instability affecting which unknowns are "already represented" (57J.34/57J.36 variance) — a separate investigation
|
||||
- Whether the error message should guide toward update vs addNode strategies — future prompt refinement
|
||||
- Whether `answerMeaning` should eventually be treated as structural metadata rather than optional metadata — architectural decision, out of scope
|
||||
|
||||
---
|
||||
|
||||
**Classification: B — PROMPT + VALIDATOR CONTRACT CHOSEN**
|
||||
|
||||
The validator already correctly rejects no-ops; the gap is (1) ambiguous prompt guidance that leads to rejected proposals and (2) lack of specific diagnostic when the specific semantic-only-no-op pattern occurs. Both are fixed by adding clear instruction + targeted enforcement with zero semantic classification machinery.
|
||||
|
||||
Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
|
||||
Reference in New Issue
Block a user