From 144b7c53f5ca76ee7300f72947f8c8c914f48671 Mon Sep 17 00:00:00 2001 From: robbond Date: Sun, 9 Aug 2026 12:20:57 +0100 Subject: [PATCH] experiment: validate unresolved uncertainty live path --- docs/current-handoff.md | 4 +++- docs/experiment-56g.md | 48 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 docs/experiment-56g.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index c1fb654..c650c1d 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -28,6 +28,8 @@ Experiment 56E tested whether the weak-priority answer ("Risk matters more to me Experiment 56F re-tested Regression A with the canonical live harness after Codex commit `4aa1492` (refine raw-answer boundary for answer meaning). Result: **PASS - strengthening safely rejected**. The LLM still produced semantic strengthening in `userSupportedMeaning` ("Avoiding additional risk is a strongly weighted preference/trade-off rather than a hard constraint") — the same class of over-resolution as 56E. However, the pre-mutation safeguard chain correctly rejected the proposal: deterministic derivation produced `proposedMeaningCategory: hard_constraint` which mismatched `rawAnswerCategory: relative_importance`, causing `proposalValidation.success: false` and preventing compatibility guard from passing. No graph mutation occurred — `n-risk-constraint` remained unresolved (status=unknown, value=null). One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review. +Experiment 56G tested Regression C (non-answer uncertainty: "I'm not really sure.") through the live production path to verify the risk-constraint distinction remains unresolved when the user expresses no position. **BLOCKED - apparatus**. The canonical helper (`tests/graph/live-update-experiment-helper.cjs`) contains a broken dynamic import path (`../lib/graph/orchestrator.js` resolves to `tests/lib/graph/orchestrator.js`, which does not exist — correct path is `../../lib/graph/orchestrator.js`). No live calls were made. Full results in `docs/experiment-56g.md`. Status pending Rob's review. + Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. @@ -166,7 +168,7 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56E. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.* +*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56G. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.* ### Return-to-Work Note (Experiment 55F) diff --git a/docs/experiment-56g.md b/docs/experiment-56g.md new file mode 100644 index 0000000..8a3691f --- /dev/null +++ b/docs/experiment-56g.md @@ -0,0 +1,48 @@ +# Experiment 56G — Validate Unresolved Uncertainty Through Live Production Path + +**Date**: 2026-08-09 +**Branch**: feature/reasoning-fidelity-v0.8 +**Type**: Live experiment — BLOCKED by apparatus failure +**Status**: BLOCKED - apparatus + +--- + +## Objective + +Answer: When the user says "I'm not really sure.", does the production path preserve that uncertainty instead of resolving or strengthening the risk-constraint distinction? + +## Fixed Case — Regression C + +- **Source**: "I want the business to grow, but I don't want to take on more risk." +- **Previous question**: "Is avoiding additional risk a hard constraint or a preference/trade-off?" +- **Answer**: "I'm not really sure." +- **Expected preserved meaning**: User is uncertain about whether avoiding additional risk is a hard constraint or preference/trade-off. +- **Expected uncertainty**: Full — no position taken. + +## Apparatus Failure + +The canonical helper (`tests/graph/live-update-experiment-helper.cjs`) contains a broken import path: + +```js +const { updateCase } = await import("../lib/graph/orchestrator.js"); +``` + +From its location at `tests/graph/`, this resolves to `tests/lib/graph/orchestrator.js` — which does not exist. The correct relative path is `../../lib/graph/orchestrator.js`. + +The canonical helper cannot invoke the production path without a fix to this import. + +## Result + +**BLOCKED - apparatus** + +No live calls were made. No experiment data captured. + +## Evidence + +- File exists: `./lib/graph/orchestrator.js` (project root) +- File missing: `tests/lib/graph/orchestrator.js` +- Broken path: `../lib/graph/orchestrator.js` from `tests/graph/live-update-experiment-helper.cjs` + +--- + +*Status pending Rob's review. Requires canonical helper import path fix before this experiment can proceed.*