experiment: test consequence of clarification target broadening

This commit is contained in:
2026-08-08 07:29:36 +01:00
parent 4302e3c435
commit 1d1dceefa3
3 changed files with 538 additions and 7 deletions
+6 -4
View File
@@ -36,7 +36,9 @@ Experiment 37 corrected the routing defect from Experiment 36 and tested a cross
**Commit:** pending (experiment: validate cold-start project recovery) — to be committed this session.
Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. Two of three targets stayed fully specific; one reproduced the 54W-style broadening on preference-versus-constraint distinctions. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. This confirms broadening did not alter downstream resolution in this scenario; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.
@@ -148,8 +150,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 3853, 54A54W. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 3853, 54A54Y. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54W)
### Return-to-Work Note (Experiment 54Y)
54R54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No drift was detected across the full chain on this single pair of scenarios, though A2's output lost the "preference/trade-off vs hard constraint" granularity from earlier experiments — the chain still succeeded because the coarser representation remained workable. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-chain.test.js` for the full experiment and results. Status pending Rob's review.
Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.