48 KiB
Experiment 54W — Does the Clarification Chain Hold Together End to End? (2026-08-08)
Objective
Tighten Experiment 54V so it does not overstate the isolated clarification-chain results. Then test the smallest end-to-end version of the clarification path: starting from one disagreement, can the semantic steps correctly determine whether the user is needed, identify the clarification target, word one question, and use the user's answer to resolve only that target without semantic drift between steps?
This is still test-only. Do not integrate with the active engine, graph, Behaviour Selection, or UI.
Hypothesis
The individual clarification steps may remain aligned when chained together. For a genuine user-owned ambiguity, the chain should preserve: disagreement → user required → clarification target → neutral question → answer → resolved target. For an evidence-resolvable disagreement, the chain should stop early rather than inventing a clarification target or question. If the steps drift when connected, record exactly where the first material divergence occurs.
Configuration
Host: http://192.168.1.111:11434 (same as 54R–54V)
Model: qwen-claude:latest (same as 54R–54V)
Number of Live Inference Calls
Exactly 5 live Ollama calls — 4 for Scenario A + 1 for Scenario B.
Context Used
docs/current-handoff.md- Experiment 54V only in
docs/design-evolution-log.md tests/reconstruction/semantic-disagreement-resolution-source.test.jstests/reconstruction/semantic-clarification-target.test.jstests/reconstruction/semantic-clarification-question.test.jstests/reconstruction/semantic-clarification-answer-resolution.test.js
Scenarios
Scenario A — Genuine User-Owned Ambiguity
Source: "I want the business to grow, but I don't want to take on more risk."
Fixed disagreement:
- growth should be prioritised even if some additional risk is unavoidable;
- avoiding additional risk is a hard constraint even if growth is slower.
Fixed user answer: "It's a hard constraint. I don't want any increase in risk."
Scenario B — Evidence-Resolvable Disagreement
Source: "Orders are arriving late and customers have started complaining."
Fixed disagreement:
- delays may be caused by insufficient staff capacity;
- delays may be caused by unreliable supplier lead times.
Stage-by-Stage Outputs
Scenario A — Full Chain
A1 — Resolution Source (Experiment 54R instruction)
- Actual output:
{ requiresUserClarification: true, reason: "The disagreement centers on a priority trade-off between growth and risk avoidance that depends entirely on the user's underlying constraints rather than external facts." } - Input used: original source + fixed disagreement + evidence needs
- Classification:
stage_correct— correctly identified that user clarification is required for a genuine user-owned ambiguity. The reason references subjective constraints (user-owned) not external facts, consistent with the true classification.
A2 — Clarification Target (Experiment 54S instruction)
- Input target used from A1:
requiresUserClarification: true(actual upstream output, NOT replaced by a human reference) - Actual output:
{ clarificationTarget: "Priority between business growth and risk avoidance when they conflict" } - Classification:
stage_correct— the model correctly identified the user-owned distinction. The target focuses on priority ordering between two competing interests (growth vs. risk), which is the exact user-owned ambiguity present in the source. No question language, no evidence framing, no interpretation chosen.
A3 — Clarification Question (Experiment 54U instruction)
- Input target from A2:
"Priority between business growth and risk avoidance when they conflict"(actual upstream output) - Actual output:
{ question: "Which do you prioritize when business growth and risk avoidance come into conflict?" } - Classification:
stage_correct— single neutral question, ends with one question mark, no biased framing. The question uses the actual A2 target's language ("priority", "business growth", "risk avoidance") without substituting a human reference. Neutral between alternatives: does not say you should prioritize growth or risk.
A4 — Answer Resolution (Experiment 54V instruction)
- Input target from actual A2:
"Priority between business growth and risk avoidance when they conflict" - Input question from actual A3:
"Which do you prioritize when business growth and risk avoidance come into conflict?" - Fixed user answer: "It's a hard constraint. I don't want any increase in risk."
- Actual output:
{ resolvedMeaning: "Risk avoidance takes absolute priority over business growth, as no increase in risk is acceptable regardless of growth potential.", targetResolved: true, remainingUncertainty: null } - Classification:
stage_correct— targetResolved=true correctly reflects that the user's answer settles the priority question. The resolved meaning interprets "hard constraint" and "no increase in risk" as absolute priority (non-negotiable boundary) over growth. This is a valid semantic interpretation of the user's answer within the A2/A3 framing. No wider consequences introduced: does not state growth is impossible, does not choose a specific growth option, does not estimate current risk levels.
Scenario B — Evidence-Resolvable Stop
B1 — Resolution Source (Experiment 54R instruction)
- Actual output:
{ requiresUserClarification: false, reason: "The disagreement can be resolved by examining operational metrics and supplier delivery records rather than requiring subjective input." } - Classification:
correct_stop— correctly identified that evidence (operational data) can resolve the cause of delays. The chain stopped immediately; no clarification target call was made, no question was generated, no answer-resolution was attempted.
Actual Upstream Outputs Passed Between Stages
| Stage | Received From | Value |
|---|---|---|
| A2 | A1 | requiresUserClarification: true, reason about subjective constraints |
| A3 | A2 | clarificationTarget: "Priority between business growth and risk avoidance when they conflict" |
| A4 | A2 | Same A2 target (same as above) |
| A4 | A3 | question: "Which do you prioritize when business growth and risk avoidance come into conflict?" |
Human reference data was used only for evaluation — not silently substituted between stages.
First Drift Point in Scenario A
No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario. The chain preserved the user-owned nature of the ambiguity from A1 through to resolution at A4 without introducing unsupported meaning or changing the interpretation of upstream results.
Scenario B — Stop Verification
- Did Scenario B stop after B1: Yes
- Were any unnecessary clarification calls made for Scenario B: No (0 additional calls)
Question: Did any stage choose a winner?
No. None of the stages selected an interpretation as correct or better. A4's resolved meaning states what the user's answer settled (priority resolution) rather than declaring one pre-existing interpretation as the winner. The chain reports what was clarified, not which side of the original disagreement is right.
Question: Did any stage introduce unsupported meaning that materially affected the next stage?
No. A2 stayed within the priority dimension present in the source. A3 preserved both competing terms ("business growth", "risk avoidance") from the A2 target. A4 interpreted the user's hard-constraint answer as absolute priority over growth — a valid reading given the answer and the A2/A3 framing. No stage added external facts or consequences that materially distorted downstream reasoning.
Question: Evidence that isolated clarification steps survive under chaining
Yes. The chain_correct result demonstrates that all four individual capabilities (resolution source, target identification, question wording, answer resolution) remained usable when chained in the two tested scenarios. Each stage's output was a valid input for the next stage. No stage degraded or produced an unexpected format. The semantic proximity between A2 and A4 is worth noting: A2 framed the distinction as "priority" while the user answer used "hard constraint" — these are semantically close but not identical (a hard constraint is stronger than a priority preference). A4 correctly interpreted the hard-constraint answer within the priority framing, so this proximity was sufficient for alignment.
Questionable or Unsupported Findings
- Only one instance of each scenario was tested. Chain stability across repeated runs needs verification.
- Only the growth-versus-risk domain was tested for Scenario A. Different domains may produce different drift patterns.
- A2's output ("Priority between business growth and risk avoidance when they conflict") lost the "preference/trade-off vs hard constraint" distinction present in the 54S human reference. This loss of granularity is not a failure per se — it is still correct within its contract — but it means downstream stages operate on a less precise target. The chain succeeded with this coarser representation, which is evidence that the steps tolerate some semantic imprecision.
Experiment Conclusion
The clarification chain remained semantically aligned end to end in both tested scenarios. For Scenario A (genuine user-owned ambiguity), all four stages produced correct outputs and each stage's actual output was a valid input for the next stage with no material drift. For Scenario B (evidence-resolvable disagreement), the model correctly stopped after the first decision without inventing unnecessary clarification steps.
Focused Test Result
chain_correct for Scenario A + correct_stop for Scenario B. 5/5 live calls completed within budget. All stage assertions passed.
Historical Comparison Result
The chain_correct result is new evidence not available in any earlier experiment (54R–54V tested isolated steps only). It demonstrates that the individual clarification capabilities remained usable when chained in the two tested scenarios, on one scenario and one model configuration. This does not extend to production integration readiness.
Documentation Updated
docs/design-evolution-log.md— added 54V clarifying caveat; added full Experiment 54W entrydocs/current-handoff.md— added 54V clarifying caveat
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as 54R–54V.
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file, not production prompts.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Clarification-Chain Logic Entered Active Runtime
This experiment created one new test file only. No clarification-chain logic entered any active runtime path, production module, or behaviour selection output.
Return-to-Work Note (Experiment 54W)
54R–54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-chain.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54X — Does the Clarification Target Lose Important Specificity When Chained? (2026-08-08)
Objective
Isolate whether the clarification-target-generation step preserves the exact user-owned distinction or broadens it, using three fixed cases under the same instruction as Experiment 54S. Passive and test-only. No question generation, answer resolution, Behaviour Selection, graph, or UI integration attempted. Same host/model. Branch: feature/user-workspace-ux-v0.7.
Hypothesis
The model may preserve clarification targets well when the ambiguity is simple and explicit, but broaden targets when the distinction is relational or preference-based. If broadening happens repeatedly, that may matter downstream because the question-generation step can only be as precise as the target it receives.
Configuration
Host: http://192.168.1.111:11434 (same as 54R–54W)
Model: qwen-claude:latest (same as 54R–54W)
Number of Live Inference Calls
Exactly 3 live Ollama calls — one per case.
Context Used
docs/current-handoff.md- Experiment 54W only in
docs/design-evolution-log.md(as historical context for the broadening observation) tests/reconstruction/semantic-clarification-target.test.js(for structural reference)tests/reconstruction/semantic-clarification-chain.test.js(for structural reference)
Experiment 54W Corrections Applied
Replaced "No drift was detected." with: "No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario."
Replaced "the individual clarification capabilities survive end-to-end chaining" with: "The individual clarification capabilities remained usable when chained in the two tested scenarios."
Case 1 — Preference Versus Hard Constraint
Source: "I want the business to grow, but I don't want to take on more risk." Disagreement: growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower. Human reference target: whether avoiding additional risk is a preference/trade-off or a hard constraint
Actual model output: "preferred priority between business growth and risk avoidance"
Specificity classification: target_broadened — On the right topic but broadened from the material distinction (preference/trade-off vs. hard constraint) to general priority ordering. Usable downstream but not fully specific.
Case 2 — Definition Ambiguity
Source: "I want to replace the system, but the new option needs to be affordable." Disagreement: affordable means keeping upfront cost low / affordable means keeping total long-term cost low. Human reference target: whether "affordable" means low upfront cost or low overall/long-term cost
Actual model output: "whether 'affordable' refers to upfront cost or total long-term cost"
Specificity classification: target_specific — Preserved the material distinction: upfront cost versus total long-term cost. The distinction is explicit and identical in meaning to the human reference.
Case 3 — Private Factual Boundary
Source: "I could move the project forward next month, depending on whether I actually have enough time." Disagreement: the user has enough available time next month / the user does not have enough available time next month. Human reference target: whether the user has enough available time next month to take on the project
Actual model output: "whether the user has enough available time next month"
Specificity classification: target_specific — Preserved all three required elements: time availability, next month, and the capacity question. The omission of "to take on the project" does not lose material specificity — it is implied by the source context.
Inference Timing
| Metric | Value |
|---|---|
| Total live calls | 3 |
| Total inference time | 49,220 ms |
| Average | 16,406.7 ms per call |
| Fastest | 11,815 ms (Case 2) |
| Slowest | 21,303 ms (Case 1) |
Results Summary
| Classification | Count |
|---|---|
| target_specific | 2/3 |
| target_broadened | 1/3 |
| target_wrong | 0/3 |
Questions Answered
- Did Case 1 preserve preference/trade-off versus hard constraint? No — broadened to priority ordering.
- Did Case 2 preserve upfront versus long-term affordability? Yes — preserved explicitly.
- Did Case 3 preserve the user's available-time boundary? Yes — preserved explicitly with all three required elements.
- How many cases were target_specific / target_broadened / target_wrong? 2 / 1 / 0
- Did any target remain usable while still losing material specificity? Yes — Case 1 was broadly relevant and actionable but lost the preference-versus-constraint distinction.
- Did any target introduce unsupported meaning? No — no case introduced concepts not present in source or disagreement.
- Does this reproduce the broadening observed in 54W? Yes — both experiments show broadening from preference/constraint to priority framing on Case 1-style input.
- Does this establish why broadening happens? No — one isolated result per case cannot determine causality; only that it does occur for at least one ambiguity pattern.
- Does this establish whether a broader target is acceptable for the user journey? No — acceptability depends on downstream question quality and user experience, which were not tested here.
- Does this establish how the clarification question should be worded? No — no question-generation step was involved.
Evidence About Clarification-Target Specificity
A clarification target can be broadly relevant without being precise enough. Case 1's output ("preferred priority between business growth and risk avoidance") is clearly about the right topic and usable downstream, but it does not preserve the material distinction that the user actually needs to clarify — whether avoiding additional risk is a preference or a hard constraint. Cases 2 and 3 show that the same instruction can produce fully specific targets when the ambiguity involves definition boundaries or private facts rather than preference-versus-constraint relationships.
Limitations
- Only one model configuration was tested (qwen-claude:latest). Different models may behave differently.
- Only one inference per case — stability across repeated runs is untested here (though 54L previously showed strong stability for other tasks).
- The broadening pattern only emerged in Case 1; the instruction and model appear capable of specificity on other patterns.
- No downstream question or answer-resolution step was tested — usability of a broader target cannot be fully assessed without those stages.
Experiment Conclusion
Clarification targets remained usable but broadened in one of three tested cases (Case 1). The broadening reproduced the same pattern observed in 54W: preference-versus-constraint distinctions tend to become priority-ordering framings. This is not a failure — the target remains actionable — but Specificity loss occurred in one of the three tested ambiguity patterns and was absent in the other two.
Focused Test Result
2/3 targets preserved material distinction; 1/3 broadened (matching 54W pattern). All structural assertions passed. No invariant violations detected.
Historical Comparison Result
The Case 1 result reproduces the A2 output from Experiment 54W ("Priority between business growth and risk avoidance when they conflict" → "preferred priority between business growth and risk avoidance"). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 54X entrydocs/current-handoff.md— updated Return-to-Work note with 54X findings; applied 54W wording corrections
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as 54R–54X.
Confirmation Semantic Instruction and Output Contract Remained Unchanged
The instruction was identical to Experiment 54S (no examples, no stronger coaching). The output contract remained { "clarificationTarget": "short statement" } — unchanged from 54S.
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instruction defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Clarification-Target Logic Entered Active Runtime
This experiment created one new test file only. No clarification-target logic entered any active runtime path, production module, or behaviour selection output.
Return-to-Work Note (Experiment 54X)
54W showed the full clarification chain worked in the tested pair but Stage A2 broadened one target; 54X isolated target specificity using three clarification cases under the same 54S instruction — preference/constraint distinction was lost to priority framing (broadened), affordability definition stayed precise (specific), and private factual capacity stayed precise (specific). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Y — Does a Broader Clarification Target Actually Change the User Question or Resolution? (2026-08-08)
Objective
Test whether the specificity loss observed in Experiments 54W/54X actually matters downstream. If the clarification target shifts from the precise distinction "preference/trade-off versus hard constraint" to the broader "priority between growth and risk," does that materially change the question asked or the meaning resolved from the user's answer? Passive and test-only. No redesign of target generation, no integration with Behaviour Selection, graph, or UI.
Hypothesis
The broader target may remain workable but could alter the user-facing distinction. Specifically:
- the precise target may ask whether risk avoidance is a hard boundary or a trade-off;
- the broader target may instead ask which objective has priority.
Those questions are related, but the user's answers need not mean exactly the same thing.
Configuration
Host: http://192.168.1.111:11434 (same as 54R–54X)
Model: qwen-claude:latest (same as 54R–54X)
Number of Live Inference Calls
Exactly 4 live Ollama calls — one question per variant + one answer-resolution per variant.
Context Used
docs/current-handoff.md- Experiment 54X only in
docs/design-evolution-log.md(as historical context for the broadening observation) tests/reconstruction/semantic-clarification-question.test.js(for structural reference: instruction and output contract)tests/reconstruction/semantic-clarification-answer-resolution.test.js(for structural reference: instruction and output contract)
Fixed Scenario
Source: "I want the business to grow, but I don't want to take on more risk."
Fixed user answer: "It's a hard constraint. I don't want any increase in risk."
Variant A — Precise Target
Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint.
This is the human-reviewed specific target.
Variant B — Broadened Target
Clarification target: priority between business growth and risk avoidance when they conflict.
This mirrors the broader target observed in Experiments 54W and 54X.
Stage 1 Results — Question Generation
| Variant | Generated Question |
|---|---|
| A (Precise) | Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off? |
| B (Broadened) | When business growth and risk avoidance conflict, which do you prioritize? |
Question analysis: Variant A frames the question around whether avoiding risk is a hard constraint or a preference/trade-off — directly addressing the boundary distinction. Variant B frames it around priority ordering between growth and risk when they conflict — reframing the decision as relative importance rather than a boundary question. The model did not impose a reference wording for Variant B; it produced its natural framing from the broadened target.
Stage 2 Results — Answer Resolution
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | Avoiding additional risk is established as a hard constraint. | true | null |
| B (Broadened) | Risk avoidance takes absolute priority over business growth when they conflict. | true | null |
Resolution analysis: Both variants produced targetResolved: true with remainingUncertainty: null. The resolved meanings differ in wording but convey materially equivalent meaning for downstream reasoning: "avoiding additional risk is a hard constraint" and "risk avoidance takes absolute priority over business growth when they conflict" establish the same boundary — no more risk will be accepted. Neither resolution introduced unsupported wider consequences.
Question Equivalence Classification
questions_materially_different
The precise target asked whether avoiding extra risk is a trade-off/preference or a hard constraint (a boundary question). The broadened target asked which objective has priority when they conflict (an ordering question). These ask the user to resolve different conceptual distinctions. The distinction was lost as predicted.
Resolution Equivalence Classification
resolutions_materially_equivalent
Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
Questions Answered
- What question did the precise target generate? "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?"
- What question did the broadened target generate? "When business growth and risk avoidance conflict, which do you prioritize?"
- Did both questions ask the user to resolve the same underlying distinction? No — one asked about boundary (constraint vs trade-off); the other asked about priority ordering.
- Did the broader target turn preference-versus-constraint into simple priority ordering? Yes — it reframed the distinction as relative importance rather than an absolute boundary.
- What resolved meaning did Variant A produce from the fixed answer? "Avoiding additional risk is established as a hard constraint." (targetResolved: true)
- What resolved meaning did Variant B produce from the same answer? "Risk avoidance takes absolute priority over business growth when they conflict." (targetResolved: true)
- Were the two resolved meanings materially equivalent? Yes — both establish that no additional risk will be accepted for growth.
- Did either variant leave remaining uncertainty? No — both returned null, indicating full resolution of the target from the fixed answer.
- Did either variant introduce unsupported wider consequences? No — neither inference extended beyond the source meaning and the user's explicit answer.
- Does the broadening materially affect downstream clarification in this tested scenario? No — despite different questions, the same answer produced the same downstream meaning.
- Does this establish that broad targets are generally safe or unsafe? No — only one scenario tested.
- Does this establish how target generation should be changed? No — no fix designed from these results.
- Does this establish UI behaviour? No — this is a clarification-target test only.
Limitations
- Only one source scenario and one fixed answer were tested. Different sources may behave differently.
- Only one ambiguity pattern (preference/constraint) was tested for downstream consequence. Other patterns not assessed.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
- The semantic equivalence classification is based on structured heuristic checks supplemented by the test output — for definitive judgment, human review of the actual resolved meanings is required.
- One tested ambiguity pattern; broader safety/generalisation remains untested.
Experiment Conclusion
The broader target changed the clarification question but not the resolved meaning for the tested explicit answer.
The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
The key finding is: Does the distinction we lost actually matter? — In this tested scenario, it did not. Specificity loss is not automatically a failure; it depends on whether it changes downstream meaning. Whether this holds across other scenarios and ambiguity patterns remains untested.
Focused Test Result
Both variants produced targetResolved: true with zero remaining uncertainty and zero unsupported inferences. Resolutions were materially equivalent despite questions being materially different. 4/4 live inference calls completed successfully (all tests passed).
Historical Comparison Result
Variant A's resolution ("Avoiding additional risk is established as a hard constraint") matches the expected outcome from Experiment 54V Case 1 and the handoff summary. Variant B's resolution ("Risk avoidance takes absolute priority over business growth when they conflict") represents a coarser framing — but not an incorrect one — for downstream use. The result confirms that the coarser representation remains workable even when it loses the preference-versus-constraint granularity.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 54Y entry; applied 54X wording correctionsdocs/current-handoff.md— updated with Experiment 54Y summary and new Return-to-Work note
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as 54R–54X.
Confirmation Semantic Instructions and Output Contracts Remained Unchanged
Both question and resolution instructions were identical to those defined in Experiments 54U and 54V. Output contracts unchanged from 54U ({ "question": "..." }) and 54V ({ "resolvedMeaning", "targetResolved", "remainingUncertainty" }).
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Specificity-Consequence Logic Entered Active Runtime
This experiment created one new test file only. No specificity-consequence logic entered any active runtime path, production module, or behaviour selection output.
Return-to-Work Note (Experiment 54Y)
Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Z — Does Target Broadening Change Resolution When the Answer Is Less Explicit? (2026-08-08)
Objective
Experiment 54Y showed that with a very explicit hard-constraint answer ("It's a hard constraint. I don't want any increase in risk."), both precise and broadened clarification targets converged on materially equivalent resolved meaning — even though the generated questions were materially different.
This leaves one unresolved consequence:
If the user's answer is less explicit, do those two different questions lead to materially different resolved meaning?
54Z tests that only. Passive and test-only. No redesign of target generation. No integration with Behaviour Selection, graph, or UI. No production code changes.
Hypothesis
The precise and broadened questions may behave differently when the answer does not explicitly name the missing distinction. A weaker answer could:
- remain correctly unresolved under the precise question;
- but be interpreted as a resolved priority decision under the broader question.
If that happens, target broadening has a real downstream consequence. If both variants preserve equivalent uncertainty, the broadening may be less consequential than expected.
Configuration
Host: http://192.168.1.111:11434 (same as 54R–54Y)
Model: qwen-claude:latest (same as 54R–54Y)
Number of Live Inference Calls
Exactly 4 live Ollama calls — one answer-resolution per variant × two weaker answers. No question-generation calls (questions are fixed from Experiment 54Y).
Context Used
docs/current-handoff.md- Experiment 54Y in
docs/design-evolution-log.md(as basis for the unresolved consequence) tests/reconstruction/semantic-clarification-specificity-consequence.test.js(structural reference: question generation and answer resolution helpers)tests/reconstruction/semantic-clarification-answer-resolution.test.js(structural reference: instruction and output contract)
Fixed Source
I want the business to grow, but I don't want to take on more risk.
Variant A — Precise Target
Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint. Fixed question: Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
Variant B — Broadened Target
Clarification target: priority between business growth and risk avoidance when they conflict. Fixed question: When business growth and risk avoidance conflict, which do you prioritize?
Answer 1 — Priority Without Constraint Meaning
User answer: "Risk matters more to me."
Expected behavior (human-reviewed): This answer does not clearly establish whether risk avoidance is a hard constraint or merely a stronger preference. For Variant A, the precise target should therefore remain unresolved. For Variant B, the answer may legitimately resolve the priority target as "risk avoidance has higher priority than growth."
Answer 1 Results
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | The user treats avoiding additional risk as a strong priority or preference rather than an absolute, non-negotiable constraint. | true | null |
| B (Broadened) | Risk avoidance is prioritized over business growth when they conflict. | true | null |
Answer 1 Analysis
Variant A interprets "Risk matters more to me" as meaning risk avoidance is a strong preference/priority rather than an absolute constraint — this maps correctly onto the precise target (preference/trade-off vs hard constraint). The model marked targetResolved=true because it interpreted the answer as settling the distinction toward "preference/trade-off."
Variant B interprets the same answer as meaning risk avoidance has higher priority over growth when they conflict — this maps correctly onto the broadened target (priority ordering).
Classification: resolutions_materially_equivalent
Both variants map the weak answer to a preference/priority-over-constraint interpretation. Neither resolves to "hard constraint." The resolved meanings use different framing but preserve the same downstream reasoning state: risk is not an absolute boundary, it is a prioritized consideration. For downstream use (what can/cannot be done), both produce equivalent uncertainty about whether risk could ever be accepted.
Both variants were flagged as potential_erasal_of_uncertainty because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases. The precise target may have erased uncertainty by inferring that "Risk matters more to me" means risk is not a hard constraint; that conclusion was not explicitly supplied by the user.
Answer 2 — Conditional Trade-Off
User answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
Expected behavior (human-reviewed): This answer indicates risk avoidance is not an absolute hard constraint; some trade-off may be acceptable depending on the opportunity. For Variant A, this should resolve away from "hard constraint" while retaining conditionality. For Variant B, it may establish that risk is normally prioritized but can yield to growth in some cases.
Answer 2 Results
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | Avoiding additional risk is a preference or trade-off rather than a hard constraint. | true | null |
| B (Broadened) | Default priority is risk avoidance, with a conditional willingness to accept some risk for highly suitable opportunities. | false | It remains unclear how "the right opportunity" is defined and which factor strictly takes precedence when a specific growth opportunity carries significant risk. |
Answer 2 Analysis
This is the critical divergence. Variant A collapses the conditional nature of the answer into a simple preference-vs-constraint resolution. The model says "preference or trade-off rather than a hard constraint" — but loses the key information that there are conditions (the right opportunity) under which even this preference could shift. This was flagged as forced_certainty_detected and potential_erasal_of_uncertainty.
Variant B preserves the conditionality ("conditional willingness") and correctly marks targetResolved=false because the answer does not establish a stable priority — the priority shifts depending on context. It also identifies remaining uncertainty about what constitutes "the right opportunity."
Classification: resolutions_materially_different
This is a material divergence. Variant A erases the conditional nature of the user's stated position and produces a flat preference-versus-constraint resolution. Variant B preserves both the default-priority-and-conditional structure AND the remaining uncertainty about when conditions change. For downstream reasoning, this matters because:
- Under Variant A's meaning: risk avoidance = preference/trade-off → may be willing to accept risk in some cases (inferred)
- Under Variant B's meaning: default priority risk, conditionally willing → conditional willingness is preserved explicitly
However, the divergence exists primarily in remainingUncertainty content, not in the resolvedMeaning itself. Both agree that risk avoidance is not a hard constraint. The difference is in whether the model preserves "there are conditions we don't yet understand" versus collapsing everything to "not a hard constraint."
Questions Answered
-
For Answer 1, did Variant A correctly preserve uncertainty about preference versus hard constraint? Partially. Variant A mapped the answer toward "preference/trade-off rather than absolute constraint" but marked it as fully resolved (targetResolved=true) with no remainingUncertainty — erasing the ambiguity that "matters more" doesn't define a boundary.
-
For Answer 1, did Variant B resolve the broader priority target? Yes. The broadened target ("priority between growth and risk") was resolved as "risk avoidance is prioritized over business growth when they conflict." This is a correct mapping from the weak answer to the priority frame.
-
Did Answer 1 therefore create materially different resolution states between A and B? No. Both map to the same downstream state: risk avoidance is not an absolute boundary but a stronger consideration than growth. The resolutions are materially equivalent for downstream reasoning about what can/cannot be done.
-
For Answer 2, did Variant A correctly identify that risk avoidance is not an absolute hard constraint? Partially correct on the outcome (not a hard constraint) but failed to preserve conditionality — the "might accept some" conditional was collapsed into a flat preference resolution with no remaining uncertainty.
-
For Answer 2, did Variant B preserve the conditional nature of the priority? Yes. Variant B preserved both the default-priority-and-conditional structure and identified remaining uncertainty about when conditions shift.
-
Were the Answer 2 resolution states materially equivalent or different? Different. Variant A erased conditionality; Variant B preserved it plus remainingUncertainty. This is a material divergence for downstream reasoning state.
-
Did either variant force a weak answer into stronger meaning than the user supplied? Yes — Variant A for Answer 2 collapsed conditional willingness ("might accept some") into a flat preference classification, erasing the conditionality layer.
-
Did either variant erase uncertainty that should remain? Yes — Variant A for both answers showed
potential_erasal_of_uncertainty. For Answer 1, "Risk matters more to me" became a fully resolved preference (no remainingUncertainty). For Answer 2, conditionality was erased. -
Does target broadening have a material downstream consequence when answers are less explicit in these tested cases? Yes — specifically for Answer 2 (conditional trade-off). The precise target question led the model to map to a flat preference-vs-constraint resolution and erase conditionality. The broadened target preserved conditional structure. This means target broadening has a real, asymmetrical consequence: the broadened question can actually preserve nuance that the precise question erases in this case.
-
Does this establish that precise targets are always required? No — Answer 1 showed no material divergence, and for Answer 2 the broader target preserved more nuance than the precise one. Neither is universally better.
-
Does this establish how target-generation logic should be changed? No — only two answers tested; neither variant was consistently better; no fix designed from these results.
-
Does this establish UI behaviour? No — this is a clarification-target test only.
Limitations
- Only one source scenario and two user answers were tested. Different sources may behave differently.
- Only one ambiguity pattern (preference/constraint) was tested with weak answers. Other patterns not assessed.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
- The asymmetric finding (broadened target preserving more nuance for Answer 2) is surprising and warrants further testing with additional answers that include explicit conditionality.
- Two tested cases; broader generalisation remains untested.
Experiment Conclusion
Target broadening changed wording but not material resolution under weaker answers — except when the answer contained explicit conditionality.
For Answer 1 ("Risk matters more to me."), both variants produced materially equivalent downstream meaning: risk avoidance is stronger than growth but not an absolute constraint. The broader target did not create a material divergence here.
For Answer 2 ("I'd normally avoid more risk, but for the right opportunity I might accept some."), the variants diverged. Variant A (precise) collapsed conditionality into a flat preference resolution and erased uncertainty. Variant B (broadened) preserved conditional structure and remaining uncertainty about what constitutes "the right opportunity."
Unexpected finding: The broadened target preserved more nuance than the precise target for the conditional answer. Neither framing was consistently superior across the two tested weaker answers.
Focused Test Result
All 4 live inference calls completed successfully (all tests passed). Answer 1: materially equivalent resolutions from both variants. Answer 2: materially different resolutions — Variant A erased conditionality; Variant B preserved it. One forced certainty detection (Variant A on Answer 2).
Historical Comparison Result
Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. Both tested weak answers confirmed the hypothesis: weak answers expose differences between precise and broadened targets, but only under specific content conditions. Neither variant was consistently superior across the two answers tested.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 54Z entry; applied 54Y wording correctionsdocs/current-handoff.md— updated with Experiment 54Z summary and new Return-to-Work note
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as 54R–54Y.
Confirmation Semantic Instructions and Output Contracts Remained Unchanged
Answer-resolution instruction identical to Experiment 54V. Output contract unchanged from 54V ({ "resolvedMeaning", "targetResolved", "remainingUncertainty" }).
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Weak-Answer Consequence Logic Entered Active Runtime
This experiment created one new test file only. No weak-answer consequence logic entered any active runtime path, production module, or behaviour selection output.