experiment: test resolution from preserved answer meaning
This commit is contained in:
@@ -44,7 +44,9 @@ Experiment 54Z tested whether convergence between precise and broadened targets
|
||||
|
||||
Experiment 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer). Two of the four tested answers showed loss of nuance: one was over-resolved (weak priority set targetResolved=true with inferred "not a constraint" meaning) and one retained the correct target category while losing conditional qualification ("might accept some for the right opportunity" became "preference or trade-off rather than a hard constraint"). The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur. 4 live calls completed at ~62s total. The answer-resolution step appears biased toward resolution for weak priority statements. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
Experiment 55B separated answer meaning from target-resolution judgement using independent calls per case. Three fixed answers tested (weak priority, conditional trade-off, non-answer) through two modes each: Mode A (meaning-only, no resolution decision) and Mode B (resolution via the same 54V/55A instruction). All three meanings were preserved accurately in Mode A (3/3 meaning_preserved). The only information loss appeared during resolution: Case 2's conditional qualification ("for the right opportunity I might accept some") was preserved by Mode A but flattened to a flat "preference or trade-off" statement during resolution judgement. This supports separating meaning extraction from resolution judgement in production, though further testing is needed before specific implementation changes. Additionally, Case 1 (weak priority) resolved correctly in 55B but over-resolved in 55A — suggesting the resolution step may exhibit run-to-run instability under the same configuration. 6 live calls completed at ~104s total. No production code changed. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
Experiment 55B separated answer meaning from target-resolution judgement using independent calls per case. Three fixed answers tested (weak priority, conditional trade-off, non-answer) through two modes each: Mode A (meaning-only, no resolution decision) and Mode B (resolution via the same 54V/55A instruction). Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement. Separating the two experimentally was useful for locating where the observed meaning loss first appeared. Additionally, Case 1 (weak priority) resolved correctly in 55B but over-resolved in 55A — this does not establish that the weak-priority problem is solved; it indicates run-to-run variation. 6 live calls completed at ~104s total. No production code changed. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
Experiment 55C chained actual preserved meaning from Stage 1 into Stage 2 resolution, testing whether carrying semantic state forward removes the conditionality loss observed in 55B. Three cases tested (weak priority, conditional trade-off, non-answer) through two stages each = 6 live calls at ~117s total. Case 2 conditional qualification survived through both stages and resolved correctly (targetResolved=true with condition retained). Case 3 non-answer uncertainty preserved through both stages. Case 1 over-resolved in Stage 2 because Stage 1 itself strengthened "risk matters more" into language about "preference/trade-off rather than absolute constraint." Compared to 55B, the weak-priority case did not remain honestly unresolved — Stage 1 distorted it first, then Stage 2 resolved that distortion. No two-stage design is proven superior; meaning can be lost at either stage. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.
|
||||
|
||||
@@ -156,11 +158,11 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55B. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55C. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
|
||||
|
||||
### Return-to-Work Note (Experiment 55B)
|
||||
### Return-to-Work Note (Experiment 55C)
|
||||
|
||||
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 8c12931. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. 55C chained actual preserved meaning into the resolution step to test whether carrying semantic state forward removes that loss. The weak-priority case did not remain honestly unresolved — Stage 1 strengthened it, then Stage 2 resolved the distortion. The conditional trade-off qualification survived through both stages; non-answer uncertainty survived too. Preserving meaning before resolution changed the conditionality-loss pattern (it was eliminated) but introduced a new one: if Stage 1 distorts, Stage 2 amplifies it. This does not yet justify production architecture changes. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 22325d5. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
### Experiment 55A Summary — Clarification Uncertainty Preservation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user