robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update reasoning refinement handoff after experiment 56E

Browse Source
This commit is contained in:
robbond
2026-08-09 11:25:43 +01:00
parent 168ef69074
commit c06aecc3f7
1 changed files with 19 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+19 -1
View File
@@ -22,6 +22,10 @@
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
Experiment 56D confirmed that Regression B (conditional trade-off resolution) works end-to-end through the real `updateCase()` production path. Deterministic derivation correctly identifies conditional semantics, passes all guards, and produces a valid graph update with emergent threshold unknown — no regression detected from commit `3e78d57`. Status pending Rob's review.
Experiment 56E tested whether the weak-priority answer ("Risk matters more to me.") survives the full `updateCase()` production path without strengthening beyond relative importance. Result: **FAIL - semantic interpretation**. The LLM extracted userSupportedMeaning as "Avoiding additional risk is a preference/trade-off rather than a hard constraint" — asserting that risk is not a hard constraint, which goes beyond what the answer establishes (only relative importance). The deterministic guard passed because it saw the already-strengthened meaning. n-risk-constraint was incorrectly treated as resolved to "preference/trade-off". No emergent unknown created. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
@@ -160,7 +164,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56E. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 55F)
@@ -177,3 +181,17 @@ The first implementation pass added proposal-level `answerMeaning` with a pre-mu
### Experiment 56B Summary — Regression B Live Run After Normalisation
Commit `36faf70` added normalization for `conditional_qualification → conditional_tradeoff`, but a live Regression B run returned a *different* variant: `supportCategory: "conditional_preference"`. The existing normalisation map does not cover this value. Two independent Zod errors occurred: (1) `conditional_preference` not in the supportCategory enum, and (2) `resolutionGuidance` was free-text instead of an enum value. **Run-to-run model variation confirmed** — the same fixed input produced `conditional_qualification` in Ex 56A and `conditional_preference` in Ex 56B. The pre-mutation guard remains unreachable because proposal_validation rejects first. Failure classification: `FAIL — normalization / proposal contract`. Branch: `feature/reasoning-fidelity-v0.8`. File to inspect when resuming: `docs/experiment-56b.md`. Status pending Rob's review.
### Experiment 56D Summary — Regression B via Real Production Path
Tested whether deterministic derivation refinement from commit `3e78d57` (refine answer meaning derivation for negation and qualification) works end-to-end through the real `updateCase()` production path. Input: source "I want the business to grow, but I don't want to take on more risk." Answer "I'd normally avoid more risk, but for the right opportunity I might accept some." — the canonical conditional trade-off case (Regression B).
**Result: PASS.** Five of five checkpoints confirmed across one live Ollama call at `http://192.168.1.111:11434` with `qwen-claude:latest`:
1. `userSupportedMeaning` correctly extracted conditional semantics — separated default preference (avoid risk) from qualification (override for right opportunity).
2. Deterministic profile derivation produced `conditional_tradeoff` category despite LLM returning null for `supportCategory`.
3. Pre-mutation guard passed with zero errors — the normalized/derived meaning is compatible.
4. Graph mutation proposed: `n-risk-constraint` resolved from unknown→resolved; emergent unknown `n-opportunity-criteria` created (unknown/unknown) capturing the threshold definition need.
5. Follow-up question correctly targets the emergent conditional/threshold unknown.
**Key observation**: The LLM does not auto-populate `supportCategory` — it is consistently null in `answerMeaning`. The deterministic derivation layer in `readDiagnostics` (and the inline pipeline) is the sole mechanism by which meaning profile category gets determined. This confirms the design: LLM produces raw meaning; deterministic logic categorizes it. No regression detected. Full results in `docs/experiment-56d.md`. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.