confidence-engine/docs/experiment-57j54.md

Experiment 57J.54 — Uncertainty Identity vs Relatedness Diagnosis

Branch: feature/structured-semantic-fidelity-v0.20 Starting HEAD: 19c00f3 (experiment: test structured-fidelity multi-turn progress) Experiment commit: pending

Objective

Answer exactly: Under the current v0.20 graph-update contract, why can an explicit unresolved uncertainty such as "whether projected office savings are realistic" be represented by modifying a broader existing cost unknown instead of creating/refining a node that actually represents that uncertainty?

Controlled Distinction

Broad projected-cost uncertainty (Concept A): "What will total costs at the new location be, including relocation, facilities and payroll?"

Savings-realism uncertainty (Concept B): "Are the projected office savings realistic?"

Verdict: OVERLAPPING BUT DISTINCT

These are not fully distinct because Concept B is a sub-question of Concept A's domain. Concept A asks "how much will it cost?" and Concept B asks "is one component of the cost projection valid?" They overlap in that both concern projected financial outcomes at the new location. But they are not the same uncertainty: Concept A is about magnitude/estimation accuracy across all cost categories; Concept B is about assumption validity for a specific cost category (office savings). One can be fully resolved (we know total will be £X) while the other remains open (savings may be overstated).

The graph cannot currently represent this without either:

• A dedicated unknown for Concept B (direct), or
• Absorbing it into Concept A's node (indirect, losing specificity).

Part 1 — Prompt Contract

Same-vs-related distinction explicit: PARTIAL

Relevant rules from lib/graph/prompt-builder.js:

• Rule #6: structural mutation required for consequential information/uncertainty
• Rule #7: new unknown only for "new decision, claim, object, measure, dependency, or unresolved term"
• Rule #11: "Do not add duplicate unknowns."
• Additional Guidance (line 137): "first check whether an existing unresolved node already represents the same uncertainty; if so, update/refine that existing structure rather than adding a duplicate; if no such node exists, add a new unknown"
• Rule #12: "Do not expand unrelated branches."

Analysis: The prompt instructs the model to distinguish "same uncertainty" from "merely related uncertainty" but provides no structural mechanism to enforce this distinction. Rule #11 says "do not add duplicate unknowns" — but this only triggers when the model chooses to add a new unknown node (which then gets checked against existing ones). When the model chooses update/refine over add, this rule never applies. The additional guidance line 137 tells the model to check for "the same uncertainty" but gives no criteria for distinguishing "same" from "related." Rule #7's list ("new decision, claim, object, measure, dependency, or unresolved term") is exhaustive in structure but not semantic — it lists categories that justify new nodes but does not define when an existing node already covers the uncertainty.

Part 2 — Validator Contract

Same-vs-related distinction enforced: NO

Mechanism: validateSemanticDuplicateUnknowns() in lib/graph/apply-proposal.js (line 378) compares added nodes against unresolved existing unknowns for exact normalized string overlap on label/description text. It uses .includes(text) — i.e., the added node's label or description must appear as a substring of the existing unknown's label or description.

This mechanism cannot distinguish Concept A from Concept B because:

1. The model chose updatedNodes (not addedNodes), so this function never runs for the savings-realism question.
2. Even if it did run, exact string matching would not flag "savings realism" as a duplicate of "total costs at new location" since neither text contains the other.

The validator has no invariant that says: "If an existing unknown is a superset concept and the answer introduces a sub-question within that superset but outside its direct scope, a new unknown may be required." This distinction is purely semantic and falls entirely on model compliance.

Part 3 — Structural Consequences

57J.53 Update 1 representation: MATERIAL INFORMATION LOSS

Why: The engine replaced a focused uncertainty ("is this specific savings assumption valid?") with a broad status flag ("this cost projection is provisional"). The node's semantic content did not change — only its status field changed from unknown to provisional. This means:

1. Query capability lost: The graph cannot answer "What evidence bears on whether savings are realistic?" because the node's label/description still says "Projected total costs at the new location including relocation, facility, and payroll adjustments." The specific savings-realism question is not retrievable from any node field.
2. Dependency tracking lost: If someone later adds evidence about savings realism (as Update 2 did), there is no structural target for that evidence beyond a "supports" edge to the central statement — not to the cost unknown node where the concern actually resides.
3. Scope drift possible: Future reasoning steps might treat provisional status as meaning "this cost estimate needs verification" rather than "I specifically doubt whether these savings assumptions hold." The distinction is subtle but material for downstream investigation.

Can later reasoning directly ask what evidence bears on savings realism: PARTIAL

PARTIAL because the selected question from Update 1 ("What would clarify realism of projected office savings from the relocation in this situation?") preserved the specific language, so at least one textual anchor survives — but this is in the selectedQuestion.label, not in the graph structure itself. If the selectedQuestion field is transient, no persistent graph-level anchor for the savings-realism concern remains.

Part 4 — Evidence Linkage Consequence

Is Update 2's parallel £2M metric consequence of the same representation issue: YES — SAME ROOT CAUSE

Why structurally: Because Update 1 represented the savings-realism uncertainty via status degradation rather than a dedicated node, there was no structural anchor for the evidence to attach to. When Update 2 arrives with concrete savings data ("£2M/year based on lease/business rates/etc."), the model sees:

• Central statement node (target of "supports" edge — but that's generic)
• noymlfr node with only a provisional status flag (not a clear "savings realism unknown")
• No dedicated savings-realism unknown

The £2M metric was correctly added as evidence, but without a dedicated savings-realism unknown from Update 1, the model had no structurally obvious target for the linkage. It connected to the central statement instead — which is valid but incomplete. The lack of cross-linkage between the new evidence and the existing uncertainty is a direct downstream consequence of Update 1's weak representation.

Part 5 — Architecture Ownership

Classification: B — PROMPT SEMANTIC-IDENTITY GAP

Why: The prompt correctly instructs the model to distinguish "same uncertainty" from "merely related uncertainty" (Additional Guidance, line 137), but this instruction is fundamentally underspecified. It tells the model to do the right thing (check whether an existing node represents the same uncertainty) without giving it a criterion for when a broad cost node covers a specific savings-valuation concern. The gap is in the prompt's semantic identity definition — it conflates "overlapping topic domain" with "same uncertainty" without distinguishing them structurally or semantically.

The prompt does NOT need keyword logic (anti-keyword rule confirmed: the distinction is inherently semantic). It needs clearer boundary conditions between:

• "This broad node already covers my concern" (update/refine)
• "This broad node overlaps my domain but asks a different question about it" (add new unknown)

Part 6 — Smallest Next Boundary

Smallest next boundary: B — prompt-only clarification

The single semantic distinction the prompt must make:

When the answer expresses uncertainty about a specific assumption or sub-component within an existing uncertain topic, treat this as a new unresolved term under rule #7 (the assumption itself is the unresolved term), even if the broader domain appears covered. "Same uncertainty" means the question being asked is structurally equivalent — both are asking for the same factual resolution. "Overlapping but distinct" means one asks about scope/magnitude while the other asks about a specific variable's validity or realism within that scope, and resolving the magnitude does not resolve the variable's validity.

This can be stated as an addition to Additional Guidance under rule #6 for explicitly unresolved uncertainty — no schema change, validator change, or graph-model change required. It simply tightens the criterion the model uses to judge "same uncertainty" vs "related but distinct."

---

Convergence

Does this require keyword/synonym logic: NO

Does this require new semantic taxonomy: NO

Does this require graph schema change: NO

Does this require validator change: NO (the current validator works correctly for what it checks — exact string duplicates. The gap is upstream in model instruction, not validation.)

Does this require prompt change: YES

Should the candidate-tie stall be handled in this same change: NO (explicitly excluded)

What this establishes:

• The root cause of 57J.53's Update 1 behavior is a prompt-level semantic-identity gap, not a validator or graph-model defect.
• "Same uncertainty" and "overlapping but distinct" are both real distinctions the system needs to make, and the current contract does not distinguish them clearly enough to enforce consistently.

What this does NOT establish:

• Whether the model can actually comply with tighter prompt guidance (requires testing).
• Whether similar gaps exist in other structured categories beyond uncertainty identity.
• Any resolution of the question-selection tie stall from Update 2.
• The full scope of information loss across all existing unknown nodes that might absorb sub-concerns.

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Schema changed: NO

Tests changed: NO

Ollama calls: 0

Dev server disturbed: NO