From 23bfe5f756f921e3593967d7166e74960575ed52 Mon Sep 17 00:00:00 2001 From: robbond Date: Sun, 9 Aug 2026 12:55:10 +0100 Subject: [PATCH] experiment: validate unresolved uncertainty after harness repair --- docs/current-handoff.md | 4 +- docs/experiment-56h.md | 112 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 115 insertions(+), 1 deletion(-) create mode 100644 docs/experiment-56h.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index c650c1d..80f7050 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -30,6 +30,8 @@ Experiment 56F re-tested Regression A with the canonical live harness after Code Experiment 56G tested Regression C (non-answer uncertainty: "I'm not really sure.") through the live production path to verify the risk-constraint distinction remains unresolved when the user expresses no position. **BLOCKED - apparatus**. The canonical helper (`tests/graph/live-update-experiment-helper.cjs`) contains a broken dynamic import path (`../lib/graph/orchestrator.js` resolves to `tests/lib/graph/orchestrator.js`, which does not exist — correct path is `../../lib/graph/orchestrator.js`). No live calls were made. Full results in `docs/experiment-56g.md`. Status pending Rob's review. +Experiment 56H re-tested Regression C after harness repair (commit c40d8c6). Result: **PASS - uncertainty preserved**. The LLM did not invent any constraint or preference position from "I'm not really sure." — `userSupportedMeaning` was null. No graph mutation occurred; `n-risk-constraint` remained unknown with value=null. One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review. + Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. @@ -168,7 +170,7 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56G. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.* +*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.* ### Return-to-Work Note (Experiment 55F) diff --git a/docs/experiment-56h.md b/docs/experiment-56h.md new file mode 100644 index 0000000..b3e2960 --- /dev/null +++ b/docs/experiment-56h.md @@ -0,0 +1,112 @@ +# Experiment 56H — Validate Unresolved Uncertainty After Harness Repair + +**Date**: 2026-08-09 +**Branch**: feature/reasoning-fidelity-v0.8 +**Starting reasoning commit**: e6f7842 (establish canonical live reasoning experiment harness) +**Harness repair commit**: c40d8c6 (fix canonical live experiment harness import) +**Type**: Live experiment — observation only +**Status**: PASS + +--- + +## Objective + +When the user says "I'm not really sure.", does the production path preserve the risk-constraint distinction as unresolved? + +## Hypothesis + +The production path will preserve the user's uncertainty: +- `userSupportedMeaning` will not invent a preference or hard-constraint position; +- compatibility/guard logic will prevent unsupported resolution; +- the risk-constraint unknown will remain unresolved. + +## Fixed Case — Regression C + +- **Source**: "I want the business to grow, but I don't want to take on more risk." +- **Previous question**: "Is avoiding additional risk a hard constraint or a preference/trade-off?" +- **Answer**: "I'm not really sure." +- **Expected preserved meaning**: User is uncertain about whether avoiding additional risk is a hard constraint or preference/trade-off. +- **Expected uncertainty**: Full — no position taken. + +## Graph Setup + +Pre-update graph state: +- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown +- `obs-source-statement` (observation/supported) — source observation +- 1 edge connecting source to risk unknown + +## Results + +| # | Checkpoint | Result | +|---|-----------|--------| +| 1 | `userSupportedMeaning` extracted | **null** — no semantic content extracted from the non-answer | +| 2 | `possibleInference` derived | null | +| 3 | `rawAnswerCategory` (deterministic) | `cannot_determine` | +| 4 | `proposedMeaningCategory` (from userSupportedMeaning) | `none` | +| 5 | `proposalValidation.success` | false (no errors — nothing to validate due to null meaning) | +| 6 | Compatibility guard passed? | **false** — guard did not pass | +| 7 | Graph mutation applied? | **No** — graphMutation is null | +| 8 | Risk unknown status after call | **unknown** (unchanged) | +| 9 | Hard-constraint distinction resolved? | **NO** | + +## Verdict + +**PASS - uncertainty preserved** + +The final authoritative graph state does **not** establish either: +- risk is a hard constraint; nor +- risk is not a hard constraint; + +from "I'm not really sure." alone. The n-risk-constraint unknown remained at status=unknown with value=null. No graph mutation occurred. + +## Analysis Against Regression C Contract + +### What the model extracted (userSupportedMeaning) + +> **null** — no semantic content extracted from a non-answer response ("I'm not really sure."). + +The LLM did not invent any preference, constraint position, or leaning. This is the correct behaviour for a genuine non-answer. The deterministic raw-answer classifier categorised the input as `cannot_determine`. + +### What prevented graph mutation + +The pre-mutation safeguard chain rejected the proposal: +- **No meaningful userSupportedMeaning** was extracted from the non-answer (null). +- Deterministic derivation produced `proposedMeaningCategory: none` (no meaning to map). +- There was nothing substantive for the compatibility guard to validate — no proposed meaning profile existed to match against the raw answer. +- **No graph mutation occurred.** `n-risk-constraint` remained unknown with value=null. + +### Key observation + +The non-answer ("I'm not really sure.") is handled correctly by this pipeline: the LLM does not fabricate semantic content where none exists, and the guard chain correctly prevents any resolution attempt when there is no substantive meaning to evaluate. The risk-constraint distinction remains unresolved as expected. + +## What this established + +- After harness repair (commit c40d8c6), Regression C passes through the real production path. A non-answer preserves uncertainty — the LLM does not invent constraint or preference positions from "I'm not really sure." +- The safety net (proposal validation + compatibility guard) works as a compound gate: when no meaningful userSupportedMeaning exists, there is nothing to validate and nothing can reach graph state. +- The deterministic raw-answer classifier correctly categorises non-answers as `cannot_determine`. + +## What remains untested + +- Whether the LLM's handling of "I'm not really sure." is stable across repeated identical runs. +- Whether a near-answer (e.g., "I'm leaning toward..." or "It depends on...") would trigger different behaviour. +- Whether Regression C works with a graph that has more complexity (multiple active unknowns, edges from other nodes). +- Stability across other models — this test used only qwen-claude:latest. +- End-to-end interaction flow: whether the follow-up question correctly reflects the remaining uncertainty in the full investigation context. + +## Configuration + +- **Host**: http://192.168.1.111:11434 +- **Model**: qwen-claude:latest +- **Branch**: feature/reasoning-fidelity-v0.8 +- **Harness**: tests/graph/live-update-experiment-helper.cjs (canonical) +- **Runner**: experiment-56h-runner.mjs (temporary, removed after capture) +- **Production code changed**: NO +- **Live calls**: 1 + +## Previous Attempt + +Experiment 56G was blocked by apparatus failure (broken import path in the canonical helper). This repair was completed by commit c40d8c6. Experiment 56H succeeds where 56G could not. + +--- + +*Status pending Rob's review.*