confidence-engine/docs/experiment-56h.md

Experiment 56H — Validate Unresolved Uncertainty After Harness Repair

Date: 2026-08-09
Branch: feature/reasoning-fidelity-v0.8
Starting reasoning commit: e6f7842 (establish canonical live reasoning experiment harness)
Harness repair commit: c40d8c6 (fix canonical live experiment harness import)
Type: Live experiment — observation only
Status: PASS

---

Objective

When the user says "I'm not really sure.", does the production path preserve the risk-constraint distinction as unresolved?

Hypothesis

The production path will preserve the user's uncertainty:

• userSupportedMeaning will not invent a preference or hard-constraint position;
• compatibility/guard logic will prevent unsupported resolution;
• the risk-constraint unknown will remain unresolved.

Fixed Case — Regression C

• Source: "I want the business to grow, but I don't want to take on more risk."
• Previous question: "Is avoiding additional risk a hard constraint or a preference/trade-off?"
• Answer: "I'm not really sure."
• Expected preserved meaning: User is uncertain about whether avoiding additional risk is a hard constraint or preference/trade-off.
• Expected uncertainty: Full — no position taken.

Graph Setup

Pre-update graph state:

• n-risk-constraint (unknown/unknown) — active unknown, status=unknown
• obs-source-statement (observation/supported) — source observation
• 1 edge connecting source to risk unknown

Results

#	Checkpoint	Result
1	userSupportedMeaning extracted	null — no semantic content extracted from the non-answer
2	possibleInference derived	null
3	rawAnswerCategory (deterministic)	cannot_determine
4	proposedMeaningCategory (from userSupportedMeaning)	none
5	proposalValidation.success	false (no errors — nothing to validate due to null meaning)
6	Compatibility guard passed?	false — guard did not pass
7	Graph mutation applied?	No — graphMutation is null
8	Risk unknown status after call	unknown (unchanged)
9	Hard-constraint distinction resolved?	NO

Verdict

PASS - uncertainty preserved

The final authoritative graph state does not establish either:

• risk is a hard constraint; nor
• risk is not a hard constraint;

from "I'm not really sure." alone. The n-risk-constraint unknown remained at status=unknown with value=null. No graph mutation occurred.

Analysis Against Regression C Contract

What the model extracted (userSupportedMeaning)

null — no semantic content extracted from a non-answer response ("I'm not really sure.").

The LLM did not invent any preference, constraint position, or leaning. This is the correct behaviour for a genuine non-answer. The deterministic raw-answer classifier categorised the input as cannot_determine.

What prevented graph mutation

The pre-mutation safeguard chain rejected the proposal:

• No meaningful userSupportedMeaning was extracted from the non-answer (null).
• Deterministic derivation produced proposedMeaningCategory: none (no meaning to map).
• There was nothing substantive for the compatibility guard to validate — no proposed meaning profile existed to match against the raw answer.
• No graph mutation occurred. n-risk-constraint remained unknown with value=null.

Key observation

The non-answer ("I'm not really sure.") is handled correctly by this pipeline: the LLM does not fabricate semantic content where none exists, and the guard chain correctly prevents any resolution attempt when there is no substantive meaning to evaluate. The risk-constraint distinction remains unresolved as expected.

What this established

• After harness repair (commit c40d8c6), Regression C passes through the real production path. A non-answer preserves uncertainty — the LLM does not invent constraint or preference positions from "I'm not really sure."
• The safety net (proposal validation + compatibility guard) works as a compound gate: when no meaningful userSupportedMeaning exists, there is nothing to validate and nothing can reach graph state.
• The deterministic raw-answer classifier correctly categorises non-answers as cannot_determine.

What remains untested

• Whether the LLM's handling of "I'm not really sure." is stable across repeated identical runs.
• Whether a near-answer (e.g., "I'm leaning toward..." or "It depends on...") would trigger different behaviour.
• Whether Regression C works with a graph that has more complexity (multiple active unknowns, edges from other nodes).
• Stability across other models — this test used only qwen-claude:latest.
• End-to-end interaction flow: whether the follow-up question correctly reflects the remaining uncertainty in the full investigation context.

Configuration

• Host: http://192.168.1.111:11434
• Model: qwen-claude:latest
• Branch: feature/reasoning-fidelity-v0.8
• Harness: tests/graph/live-update-experiment-helper.cjs (canonical)
• Runner: experiment-56h-runner.mjs (temporary, removed after capture)
• Production code changed: NO
• Live calls: 1

Previous Attempt

Experiment 56G was blocked by apparatus failure (broken import path in the canonical helper). This repair was completed by commit c40d8c6. Experiment 56H succeeds where 56G could not.

---

Status pending Rob's review.