experiment: test end-to-end clarification chain
This commit is contained in:
@@ -24,7 +24,7 @@ Experiment 54R tested whether a consequential disagreement actually requires use
|
||||
|
||||
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
|
||||
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
|
||||
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
|
||||
|
||||
@@ -146,8 +146,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54V. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54W. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54V)
|
||||
### Return-to-Work Note (Experiment 54W)
|
||||
|
||||
54U showed a fixed clarification target can become one clean question in three tested cases; the correction clarifies the conclusion covers only those targets, not broader wording quality. 54V tested whether the user's answer resolves only that target without rewriting everything else: hard constraint resolved cleanly (true/null), affordability definition resolved correctly (true/null), and an incomplete answer ("depends on the opportunity") remained unresolved rather than being forced into a stronger category (false/uncertainty). All three cases preserved boundaries. Graph updates, next-question choice, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
|
||||
54R–54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No drift was detected across the full chain on this single pair of scenarios, though A2's output lost the "preference/trade-off vs hard constraint" granularity from earlier experiments — the chain still succeeded because the coarser representation remained workable. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-chain.test.js` for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
Reference in New Issue
Block a user