experiment: diagnose resolution contract mismatch
This commit is contained in:
@@ -3002,3 +3002,7 @@ Experiment 60B.46 reran the exact 60B.44 closure case through one bounded live u
|
||||
---
|
||||
|
||||
Experiment 60B.47 tested whether resolving the customer-signing factor with a **negative** answer (opposite polarity to 60B.46) produces symmetric lifecycle closure. **Classification: G — DIFFERENT FIRST FAILURE.** The engine's semantic reasoning was correct: same customer factor (`n_enterprise_customer_signing`) identified, negative meaning accurately preserved (`userSupportedMeaning` captured "will not sign" + £700k revenue lost), same decision (`n_product_launch_decision`) targeted. However, the proposal was rejected at `proposal_compatibility` with HTTP 422 because the model included `n_product_launch_decision` in `updatedNodes` but omitted it from `resolvedUnknownNodeIds` — a structural inconsistency when resolving multiple unknowns in one turn. No graph mutation occurred. No new unknowns invented (zero addedNodes/edges). The semantic path is symmetric; the structural output contract is not yet symmetric under negative framing. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md.
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
# Experiment 60B.48 — Resolution contract mismatch diagnosis
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/closure-metadata-capture-v0.39`
|
||||
|
||||
## Purpose
|
||||
|
||||
Diagnose **exactly why** the model in 60B.47 produced a proposal where:
|
||||
|
||||
```
|
||||
updatedNodes:
|
||||
n_enterprise_customer_signing → resolved
|
||||
n_product_launch_decision → resolved
|
||||
|
||||
resolvedUnknownNodeIds:
|
||||
n_enterprise_customer_signing (included)
|
||||
n_product_launch_decision (OMITTED ← causes rejection)
|
||||
```
|
||||
|
||||
This is a **read-only code-path and contract diagnosis**. No production code, tests, prompts, or API calls.
|
||||
|
||||
## Established facts (from 60B.47)
|
||||
|
||||
- Negative meaning preserved correctly (`userSupportedMeaning` accurate).
|
||||
- Same customer factor reused (`n_enterprise_customer_signing`).
|
||||
- Same decision targeted (`n_product_launch_decision`).
|
||||
- No addedNodes, no addedEdges.
|
||||
- Validation rejected at `proposal_compatibility` stage.
|
||||
- Error: `"Unknown node updated to resolved must also appear in resolvedUnknownNodeIds: \"n_product_launch_decision\""`
|
||||
|
||||
## Investigation path
|
||||
|
||||
### 1. Contract ownership
|
||||
|
||||
**updatedNodes[].newStatus** — MODEL GENERATED
|
||||
The model generates this field directly as part of its JSON output from the prompt contract (prompt-builder.js lines 76-93 define the field shape; rules #5, #164-#178 govern its usage). No deterministic code modifies these values before validation.
|
||||
|
||||
**resolvedUnknownNodeIds** — MODEL GENERATED
|
||||
The model generates this field directly as part of its JSON output. Rule #168 says: "When an answer resolves an existing unknown, include that existing node ID in resolvedUnknownNodeIds and update that node." This addresses the case where the model knows about resolution but doesn't explicitly tie it to updatedNodes[].newStatus.
|
||||
|
||||
**Are they generated independently?**
|
||||
PARTIAL — The model generates both fields in one JSON emission. But there is no prompt rule that makes them *structurally dependent*. They are semantically linked by the model's understanding of "resolution" but structurally independent in the output contract.
|
||||
|
||||
**Does deterministic code reconcile them before validation?**
|
||||
NO — `reconcileResolutionSemantics()` reconciles ONE direction only (resolvedUnknownNodeIds → updatedNodes). It never adds a node from updatedNodes into resolvedUnknownNodeIds.
|
||||
|
||||
### 2. Prompt contract analysis
|
||||
|
||||
Locating exact rules in `prompt-builder.js`:
|
||||
|
||||
**Rule #5:** "Resolve the answered unknown first when the answer supports it."
|
||||
→ Generic resolution guidance. Does not mention resolvedUnknownNodeIds or updatedNodes relationship.
|
||||
|
||||
**Rule #168:** "When an answer resolves an existing unknown, include that existing node ID in resolvedUnknownNodeIds and update that node rather than creating only a parallel observation."
|
||||
→ Says: put node ID in resolvedUnknownNodeIds AND update the node. But does NOT say: if you set newStatus="resolved" in updatedNodes, the node MUST also be in resolvedUnknownNodeIds.
|
||||
|
||||
**Rule #165:** "If the answer only clarifies an existing unknown, prefer updatedNodes and resolvedUnknownNodeIds over creating duplicate nodes."
|
||||
→ Says to use both fields together for clarification cases. Does not define their structural relationship.
|
||||
|
||||
**Does the prompt explicitly require the bidirectional tie?**
|
||||
NO — There is no explicit rule that says: "if any node in updatedNodes has newStatus='resolved', then every such node MUST also appear in resolvedUnknownNodeIds."
|
||||
|
||||
**Rule quality: MISSING**
|
||||
The relationship between these two fields is never formally defined as an invariant in the prompt. The model must infer it from partial guidance (rule #168 implies both should be used together, but doesn't mandate their structural consistency).
|
||||
|
||||
### 3. Reconciliation analysis
|
||||
|
||||
Locating `reconcileResolutionSemantics()` in `apply-proposal.js` at line 312:
|
||||
|
||||
```javascript
|
||||
function reconcileResolutionSemantics(graph, proposal) {
|
||||
const nextProposal = cloneJsonSafe(proposal);
|
||||
const errors = [];
|
||||
const graphNodeById = new Map(graph.nodes.map((node) => [node.id, node]));
|
||||
const updatedNodeById = new Map(
|
||||
nextProposal.updatedNodes.map((nodeUpdate) => [nodeUpdate.nodeId, nodeUpdate]),
|
||||
);
|
||||
|
||||
// DIRECTION 1: resolvedUnknownNodeIds → updatedNodes (ONE-WAY)
|
||||
for (const resolvedUnknownNodeId of nextProposal.resolvedUnknownNodeIds) {
|
||||
const existingNode = graphNodeById.get(resolvedUnknownNodeId);
|
||||
if (!existingNode) { /* error */ continue; }
|
||||
if (existingNode.kind !== "unknown") { /* error */ continue; }
|
||||
|
||||
const existingUpdate = updatedNodeById.get(resolvedUnknownNodeId);
|
||||
if (!existingUpdate) {
|
||||
// Auto-create synthetic update for node in resolvedUnknownNodeIds but not in updatedNodes
|
||||
const syntheticUpdate = buildResolvedUnknownUpdate(existingNode);
|
||||
nextProposal.updatedNodes.push(syntheticUpdate);
|
||||
updatedNodeById.set(resolvedUnknownNodeId, syntheticUpdate);
|
||||
continue;
|
||||
}
|
||||
|
||||
// If existingUpdate's newStatus is NOT "resolved", force it to "resolved"
|
||||
if (existingUpdate.newStatus !== "resolved") {
|
||||
existingUpdate.newStatus = "resolved";
|
||||
/* ... copy previous status/value */
|
||||
}
|
||||
}
|
||||
|
||||
// DIRECTION 2: updatedNodes → resolvedUnknownNodeIds (NO OP — validation only)
|
||||
for (const update of nextProposal.updatedNodes) {
|
||||
const existingNode = graphNodeById.get(update.nodeId);
|
||||
if (
|
||||
existingNode?.kind === "unknown" &&
|
||||
update.newStatus === "resolved" &&
|
||||
!nextProposal.resolvedUnknownNodeIds.includes(update.nodeId)
|
||||
) {
|
||||
// ADDS ERROR — does NOT fix
|
||||
errors.push(`Unknown node updated to resolved must also appear in resolvedUnknownNodeIds: "${update.nodeId}"`);
|
||||
}
|
||||
}
|
||||
|
||||
return { proposal: nextProposal, errors };
|
||||
}
|
||||
```
|
||||
|
||||
**Choice: B — LEAVES MISMATCH UNCHANGED (for the mismatch direction)**
|
||||
|
||||
Exact behaviour when `updatedNodes` contains a node with `newStatus="resolved"` but `resolvedUnknownNodeIds` omits it:
|
||||
1. The validation loop at lines 359-370 detects the inconsistency.
|
||||
2. It pushes an error string to the errors array.
|
||||
3. It does NOT add the node to `resolvedUnknownNodeIds`.
|
||||
4. The errors array is returned alongside the (unmodified) proposal.
|
||||
5. The caller (`applyValidatedProposal` at line 3518) adds these errors to `proposalCompatibilityErrors`.
|
||||
6. Since `errors.length > 0`, the proposal fails at the `proposal_compatibility` stage.
|
||||
|
||||
**One-way reconciliation confirmed:** `resolvedUnknownNodeIds → updatedNodes` (auto-fix). Reverse direction only reports error, does not auto-fix.
|
||||
|
||||
### 4. Validator semantics
|
||||
|
||||
The invariant is enforced at `apply-proposal.js` lines 359-370 within `reconcileResolutionSemantics()`. This function serves dual role: reconciliation + validation. The specific check (lines 361-368) ensures every unknown node marked resolved in `updatedNodes` also appears in `resolvedUnknownNodeIds`.
|
||||
|
||||
**Is this invariant semantically necessary?**
|
||||
YES — `resolvedUnknownNodeIds` is the canonical list of which unknowns are considered "resolved by this answer." If an unknown's status is set to "resolved" but it's absent from that list, downstream deterministic code (unknown clearing, decision closure, question selection) may not see it as resolved. The invariant ensures both lists agree on what was resolved.
|
||||
|
||||
**Why:** `resolvedUnknownNodeIds` drives:
|
||||
- Post-mutation unknown clearing logic (activeUnknownNodeId resolution)
|
||||
- Decision sufficiency checks
|
||||
- Question elimination (resolved unknowns are excluded from candidate pools)
|
||||
|
||||
If a node is resolved via `updatedNodes.newStatus="resolved"` but not in `resolvedUnknownNodeIds`, some downstream paths would see it as unresolved while others see it as resolved — creating inconsistent state.
|
||||
|
||||
### 5. Positive vs negative comparison
|
||||
|
||||
**60B.46 (positive):** The model apparently emitted both nodes in `resolvedUnknownNodeIds`. This allowed reconciliation to auto-create synthetic updates for any missing `updatedNodes` entries, and the proposal passed validation cleanly.
|
||||
|
||||
**60B.47 (negative):** The model only included `n_enterprise_customer_signing` in `resolvedUnknownNodeIds`, omitting `n_product_launch_decision`. Both nodes appeared in `updatedNodes` with `newStatus="resolved"`. Reconciliation auto-fixed one direction (nothing to fix for customer since it was already in both lists), but reported an error for the decision node's missing entry.
|
||||
|
||||
| Field | 60B.46 | 60B.47 |
|
||||
|---|---|---|
|
||||
| customer updated to resolved | YES (in updatedNodes) | YES (in updatedNodes) |
|
||||
| decision updated to resolved | YES (in updatedNodes, possibly via reconciliation synthetic) | YES (in updatedNodes) |
|
||||
| customer in resolvedUnknownNodeIds | YES | YES |
|
||||
| decision in resolvedUnknownNodeIds | YES (model provided) | NO (model omitted) |
|
||||
| proposal accepted | YES | NO (422 proposal_compatibility) |
|
||||
|
||||
**Difference source: MODEL OUTPUT VARIANCE + DETERMINISTIC ASYMMETRY**
|
||||
|
||||
Both factors contributed:
|
||||
- **MODEL OUTPUT VARIANCE:** The model included `n_product_launch_decision` in `resolvedUnknownNodeIds` for the positive case but not for the negative case. This is stochastic variance in how the model handles multi-node resolution lists.
|
||||
- **DETERMINISTIC ASYMMETRY:** The reconciliation function only processes one direction (`resolvedUnknownNodeIds → updatedNodes`). If 60B.46's model had also omitted the decision from `resolvedUnknownNodeIds`, it would have failed identically to 60B.47. The deterministic asymmetry in the fix means model variance has different outcomes depending on which field the model happens to get "right."
|
||||
|
||||
### 6. Candidate assessment
|
||||
|
||||
**Candidate A — PROMPT CLARIFICATION**
|
||||
Strengthen the prompt rule tying `newStatus="resolved"` to `resolvedUnknownNodeIds`.
|
||||
- Prevents 60B.47 mismatch: PARTIAL (depends on future model compliance)
|
||||
- Preserves semantic invariant: YES
|
||||
- Depends on model compliance: HIGH
|
||||
- Changes schema: NO
|
||||
- Implementation scope: SMALL (prompt text change only)
|
||||
- Principal risk: Stochastic model may still omit or produce inconsistent output; no deterministic fallback
|
||||
|
||||
**Candidate B — DETERMINISTIC NORMALISATION**
|
||||
Before validation, deterministically add every unknown node updated to `resolved` into `resolvedUnknownNodeIds`.
|
||||
- Prevents 60B.47 mismatch: YES (structural invariant enforced deterministically)
|
||||
- Preserves semantic invariant: YES (normalisation aligns output with what the model already attempted to do)
|
||||
- Depends on model compliance: LOW (model's intent is captured; code fixes the omission)
|
||||
- Changes schema: NO
|
||||
- Implementation scope: SMALL (~4 lines in reconcileResolutionSemantics, replacing error push with list update)
|
||||
- Principal risk: Minimal — if model intentionally omits a node from resolvedUnknownNodeIds, this overrides it. But there is no legitimate semantic reason to resolve a node without listing it as resolved.
|
||||
|
||||
**Candidate C — REMOVE DUPLICATED REPRESENTATION**
|
||||
Schema/contract redesign so resolution has one source of truth.
|
||||
- Prevents 60B.47 mismatch: YES (eliminates the dual-representation problem)
|
||||
- Preserves semantic invariant: YES (single source eliminates inconsistency)
|
||||
- Depends on model compliance: LOW
|
||||
- Changes schema: YES (requires prompt contract and proposal schema changes)
|
||||
- Implementation scope: LARGE (affects all downstream consumers, tests, migration)
|
||||
- Principal risk: Migration complexity; breaking existing proposals; over-engineering for a bounded fix
|
||||
|
||||
**Candidate D — KEEP CURRENT STRICT REJECTION**
|
||||
Treat inconsistent model proposals as invalid and rely on retries/future model behaviour.
|
||||
- Prevents 60B.47 mismatch: NO (same rejection will recur with probabilistic delay)
|
||||
- Preserves semantic invariant: YES
|
||||
- Depends on model compliance: HIGH
|
||||
- Changes schema: NO
|
||||
- Implementation scope: NONE
|
||||
- Principal risk: Same failure pattern repeats; no deterministic guarantee of eventual success
|
||||
|
||||
**Candidate E — COMBINATION**
|
||||
A + B: Prompt clarification PLUS deterministic normalisation.
|
||||
- Minimum viable: B alone suffices for structural correctness. A reinforces intent.
|
||||
- Prevents 60B.47 mismatch: YES
|
||||
- Preserves semantic invariant: YES
|
||||
- Depends on model compliance: LOW
|
||||
- Changes schema: NO
|
||||
- Implementation scope: SMALL
|
||||
- Principal risk: Minimal
|
||||
|
||||
### 7. Critical distinction
|
||||
|
||||
**Choice: E — MULTIPLE FACTORS**
|
||||
|
||||
Three contributing factors, in order of impact:
|
||||
1. **DETERMINISTIC NORMALISATION GAP (primary):** `reconcileResolutionSemantics` reconciles only one direction. The reverse gap is not auto-fixed.
|
||||
2. **PROMPT COMPLIANCE GAP (secondary):** No explicit rule mandates the bidirectional structural tie between `updatedNodes[].newStatus="resolved"` and `resolvedUnknownNodeIds`.
|
||||
3. **MODEL OUTPUT VARIANCE (symptom):** The model sometimes includes both nodes in `resolvedUnknownNodeIds`, sometimes doesn't — depending on answer polarity/framing.
|
||||
|
||||
### 8. Minimum corrective boundary
|
||||
|
||||
**Choice: B — deterministic reconciliation**
|
||||
|
||||
Add every unknown node updated to `resolved` into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()`, before the validation loop. This:
|
||||
- Preserves the invariant that resolved unknowns are represented consistently
|
||||
- Does not weaken semantic validation (validator still catches mismatches)
|
||||
- Does not depend on stochastic model compliance
|
||||
- Preserves accepted 60B.46 positive closure (both nodes already in resolvedUnknownNodeIds → no change to output)
|
||||
- Makes negative closure structurally valid (adds missing node deterministically)
|
||||
- Avoids schema change
|
||||
|
||||
The exact change would be in `reconcileResolutionSemantics()` at line ~359, before the error-pushing loop:
|
||||
|
||||
```javascript
|
||||
// NEW: Normalise updatedNodes → resolvedUnknownNodeIds (reverse direction)
|
||||
for (const update of nextProposal.updatedNodes) {
|
||||
const existingNode = graphNodeById.get(update.nodeId);
|
||||
if (
|
||||
existingNode?.kind === "unknown" &&
|
||||
update.newStatus === "resolved" &&
|
||||
!nextProposal.resolvedUnknownNodeIds.includes(update.nodeId)
|
||||
) {
|
||||
nextProposal.resolvedUnknownNodeIds.push(update.nodeId);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then keep the existing error-pushing loop as an assertion (detecting post-normalisation mismatch should now be impossible, but it remains as defensive code).
|
||||
|
||||
### 9. Validation of boundary candidates
|
||||
|
||||
**Would positive closure remain valid:** YES — In 60B.46's case, both nodes were already in `resolvedUnknownNodeIds`, so the normalisation adds nothing (duplicate check prevents double-inclusion).
|
||||
|
||||
**Would negative closure become structurally valid:** YES — The missing `n_product_launch_decision` would be added deterministically before validation.
|
||||
|
||||
**Would validator remain strict:** YES — The existing error-pushing code remains as a post-normalisation assertion. If any future scenario produces a mismatch (should be impossible after normalisation), it is still rejected.
|
||||
|
||||
## Findings summary
|
||||
|
||||
| Checkpoint | Finding |
|
||||
|---|---|
|
||||
| Contract ownership | Both fields are MODEL GENERATED, structurally independent in the prompt |
|
||||
| Prompt contract | MISSING — no explicit rule tying `newStatus="resolved"` to `resolvedUnknownNodeIds` membership |
|
||||
| Reconciliation | B — LEAVES MISMATCH UNCHANGED for reverse direction; only reconciles resolvedUnknownNodeIds → updatedNodes |
|
||||
| Validator semantics | YES, semantically necessary — prevents inconsistent downstream resolution state |
|
||||
| Positive vs negative | MODEL OUTPUT VARIANCE + DETERMINISTIC ASYMMETRY |
|
||||
| Critical distinction | E — MULTIPLE FACTORS (normalisation gap primary, prompt gap secondary) |
|
||||
| Minimum boundary | B — deterministic reconciliation (add missing nodes to resolvedUnknownNodeIds before validation) |
|
||||
|
||||
## No production code changed. No tests modified. No Ollama calls. No live API calls. Pure code-path and contract diagnosis.
|
||||
Reference in New Issue
Block a user