robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(confidence-engine): checkpoint design evolution archive tranche seven

Browse Source
This commit is contained in:
robbond
2026-09-03 11:34:04 +01:00
parent 53cb99ee8f
commit c33bcdbefa
3 changed files with 1313 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/archive/experiments/vol-1-chapters/README.md
+31
View File
@@ -185,6 +185,37 @@ Experiment 54R — testing whether models correctly distinguish disagreements re
Fidelity:
Exact contiguous copy.
## Tranche 7
### Chapter 13
Path:
docs/archive/experiments/vol-1-chapters/ch13/54S-54V-clarification-target-and-answer-resolution.md
Original source:
docs/design-evolution-log.md lines 8013–8635
Contents:
Experiments 54S–54V — clarification target identification, no-clarification-stability across repeated inputs, whether a clarification target becomes a useful user question without adding new meaning, and whether a clarification answer can resolve only the target.
Fidelity:
Exact contiguous copy.
### Chapter 14
Path:
docs/archive/experiments/vol-1-chapters/ch14/54W-54Z-clarification-chain-and-target-broadening.md
Original source:
docs/design-evolution-log.md lines 8636–9294
Contents:
Experiments 54W–54Z — clarification chain end-to-end integrity, specificity loss under chaining, target broadening effects on question and resolution, and resolution differences when the answer is less explicit.
Fidelity:
Exact contiguous copy.
Tranches 1 through 7 have now been extracted.
The original monolithic log remains intact and authoritative while extraction is incomplete.
## Refactor status
Only tranches 1 and 2 have been extracted.
docs/archive/experiments/vol-1-chapters/ch13/54S-54V-clarification-target-and-answer-resolution.md
+623
View File
@@ -0,0 +1,623 @@
## Experiment 54S — Can the Model Identify Exactly What the User Needs to Clarify? (2026-08-07)
### Objective
When clarification genuinely requires user input, can the model identify the specific missing user-owned distinction without yet generating the clarification question? This is passive and test-only. Do not generate a question. Do not choose a winning interpretation. Do not change Behaviour Selection. Do not change production behaviour.
### Hypothesis
When clarification genuinely belongs to the user, the model may be able to identify the smallest unresolved user-owned distinction. For example, for "I want the business to grow, but I don't want to take on more risk," the missing distinction is not "what are the risks?" but rather "whether avoiding additional risk is a preference or a hard constraint."
### Configuration
Host: `http://192.168.1.111:11434` (same as 54R)
Model: `qwen-claude:latest` (same as 54R)
No localhost fallback. No experiment-specific model variable.
### Number of Live Inference Calls
Exactly **3** live Ollama calls — one per case.
### Input Contract
Each call receives: `{ source, disagreement, requiresUserClarification }`. The boolean is fixed from human-reviewed Experiment 54R-style references. The model does not re-decide whether clarification is required.
### Output Contract
Return exactly: `{ "clarificationTarget": "short statement" | null }`.
- If `requiresUserClarification` is true → smallest specific user-owned distinction;
- If false → null.
No question text, no explanation, no recommendation, no preferred interpretation, no score, no confidence, no behaviour label. Test-only, not a production schema.
### Semantic Instruction Used
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
No examples. No expected answers. Same instruction for all three cases.
### Case 1 — Growth Versus Risk Priority
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
**Fixed clarification status:** `requiresUserClarification: true`
**Fixed human target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
### Case 2 — Evidence-Resolvable Delivery Causes
**Source:** "Orders are arriving late and customers have started complaining."
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
**Fixed clarification status:** `requiresUserClarification: false`
**Fixed human target:** null (cause should be investigated through evidence).
### Case 3 — Ambiguous Meaning of "Affordable"
**Source:** "I want to replace the system, but the new option needs to be affordable."
**Disagreement:** affordable means keeping upfront purchase cost low / affordable means keeping overall long-term cost low even if upfront cost is higher.
**Fixed clarification status:** `requiresUserClarification: true`
**Fixed human target:** what the user means by "affordable" — upfront cost versus overall/long-term cost.
### Results
#### Case 1 — Growth Versus Risk Priority
**Model output:**
```json
{
"clarificationTarget": "whether avoiding additional risk is a preference/trade-off or a hard constraint"
}
```
**Classification:** `clarification_target_correct` (14 bidirectional concept matches)
The model correctly identified the constraint versus preference distinction — the exact human target. No evidence confusion, no question text, no winner selection.
#### Case 2 — Evidence-Resolvable Delivery Causes
**Model output:**
```json
{
"clarificationTarget": null
}
```
**Classification:** `clarification_target_correct` (correctly returned null when clarification is not required)
The model correctly respected the false signal and returned no target for an evidence-resolvable case. Note: in earlier runs of this test, the model occasionally ignored the false signal and produced a target — this variability was observed but did not affect this final run's classification.
#### Case 3 — Ambiguous Meaning of "Affordable"
**Model output:**
```json
{
"clarificationTarget": "whether affordability prioritizes low upfront cost or low long-term total cost"
}
```
**Classification:** `clarification_target_correct` (7 bidirectional concept matches)
The model correctly identified the definition ambiguity — upfront cost versus long-term total cost. No vendor comparison, no budget range confusion, no question text.
### Clarification-Target-Correct Count
**3 of 3** cases classified correctly.
### Clarification-Target-Failed Count
**0 of 3** cases classified incorrectly.
### Required Questions — Answers
1. Did Case 1 identify preference/trade-off versus hard constraint? **Yes**
2. Did Case 1 avoid asking about external risk evidence instead? **Yes** (no evidence keywords present)
3. Did Case 2 correctly return null? **Yes** (in the final run)
4. Did Case 3 identify the meaning of "affordable" as upfront versus long-term cost? **Yes**
5. Did the model ever generate a full question? **No**
6. Did it confuse clarification target with evidence needed? **No**
7. Did it choose a winner? **No**
### Inference Timing
- Total time: 55,511ms (55.5s)
- Average: 18,503.7ms per call
- Fastest: 17,046ms (Case 2 — evidence-resolvable)
- Slowest: 20,957ms (Case 1 — growth-vs-risk)
### Limitations
- Three cases only; limited domain coverage (one strategic priority, one delivery scenario, one procurement definition).
- Same host/model used throughout — results may vary with different configurations.
- Does not establish generalisation beyond these specific inputs.
- The model occasionally ignored the `requiresUserClarification: false` signal in earlier test runs (producing a target when null was expected), indicating the boolean gate alone may not be sufficient for robust null enforcement.
- No clarification question was generated — this experiment establishes the target identification layer only.
### Conclusion
**The final three-case run was correct, but earlier repetitions showed instability when clarification was explicitly not required. Clarification-target identification therefore appears promising, but null enforcement is not yet stable.**
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour (the model occasionally produced a target despite `requiresUserClarification: false` in earlier runs), not merely a test warning. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
## Experiment 54T — Is "No Clarification Needed" Stable Across Repeated Identical Inputs? (2026-08-08)
### Objective
First, tighten Experiment 54S so its conclusion reflects the instability observed during earlier runs.
Then test one narrow question:
> **When `requiresUserClarification` is explicitly false, does the model consistently return no clarification target across repeated identical calls?**
This experiment exists because 54S produced the correct final result but earlier runs sometimes generated a clarification target even when clarification was explicitly not required.
This is passive and test-only.
Do not change the clarification-target instruction yet.
Do not generate questions.
Do not change production behaviour.
### Hypothesis
Two possibilities are plausible.
**Stable gating:** When `requiresUserClarification` is false, the model consistently returns `null`.
**Semantic override:** The model sometimes ignores the explicit false flag and invents a clarification target because it sees unresolved uncertainty in the source.
Either result is useful.
Do not try to correct the behaviour in this experiment.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54S)
Model: `qwen-claude:latest` (same as 54S)
No localhost fallback. No experiment-specific model variable.
### Number of Live Inference Calls
Exactly **6** live Ollama calls — three per case, repeated identical input each time.
### Semantic Instruction Used
Exact Experiment 54S instruction unchanged:
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
No examples. No expected answers. Same instruction for all six cases.
### Output Contract
Unchanged from 54S:
```json
{
"clarificationTarget": "short statement" | null
}
```
### Case A — Evidence-Resolvable / False (the unstable case from 54S)
**Source:** "Orders are arriving late and customers have started complaining."
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
**Fixed clarification status:** `requiresUserClarification: false`
**Expected result:** `clarificationTarget: null`
Run this exact case **3 times** without changing wording. This is the unstable Case 2 from 54S.
### Case B — User-Owned Ambiguity / True Control
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
**Fixed clarification status:** `requiresUserClarification: true`
**Expected semantic target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
Run this exact case **3 times**. Purpose: determine whether instability is specific to suppressing clarification or affects clarification-target generation generally.
### Results
#### Case A — Evidence-Resolvable / False
| Run | Result | Classification |
|-----|--------|----------------|
| 1 | `null` (15,566ms) | null ✓ |
| 2 | `null` (13,800ms) | null ✓ |
| 3 | `null` (19,087ms) | null ✓ |
**Null count: 3/3**
**Non-null count: 0/3**
No clarification targets were invented. The model consistently returned `null` across all three repeated identical runs with `requiresUserClarification: false`.
#### Case B — User-Owned Ambiguity / True Control
| Run | Target | Classification |
|-----|--------|----------------|
| 1 | "The relative priority between business growth and strict risk avoidance when they conflict" (18,595ms) | target_correct |
| 2 | "Preferred priority between accelerating business growth and strictly avoiding additional risk" (19,458ms) | target_correct |
| 3 | "Your maximum acceptable level of additional risk relative to desired business growth." (18,976ms) | target_correct |
**Correct-target count: 3/3**
**Incorrect-target count: 0/3**
**Null count: 0/3**
All three runs produced semantically correct targets aligned with the human reference. No null responses observed when clarification was required.
### Timing
- Total time: 105,470ms (105.5s)
- Average: 17,578.3ms per call
- Fastest: 13,798ms (Case A run 2)
- Slowest: 19,457ms (Case B run 3)
### Required Questions — Answers
1. How many Case A runs returned `null`? **3**
2. How many Case A runs returned a non-null clarification target? **0**
3. If Case A produced a target, what distinction did it invent? **N/A — none produced**
4. How many Case B runs produced the correct clarification target? **3**
5. Did Case B ever incorrectly return `null`? **No**
6. Is false/null behaviour materially stable across the three repeated runs? **Yes** — all 3 returned null with zero variance
7. Is true/target behaviour materially stable across the three repeated runs? **Yes** — all 3 produced semantically correct targets with zero variance
8. Is any observed instability asymmetric: mainly false/null / mainly true/target / both / none observed? **None observed in this experiment.** Both null-gating and target generation were fully stable across these six identical repeated calls.
9. Does this experiment establish why instability occurs? **No** — this is an observation experiment, not a diagnostic one.
10. Does this establish how to enforce null behaviour? **No** — the current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested here; broader stability remains unproven.
11. Does this establish how Behaviour Selection should work? **No.**
### Limitations
- Only two cases tested (one false, one true); limited domain coverage.
- Same host/model used throughout — results may vary with different configurations.
- This is a stability observation experiment; it does not diagnose root causes of earlier variability.
- Does not establish generalisation beyond these specific inputs.
- The model's behaviour in earlier unrecorded runs (when null-gating failed) remains the unknown variable.
### Evaluation Conclusion
**Clarification null-gating was stable across all tested repeats**
Case A returned `null` in 3 of 3 runs. Case B produced correct targets in 3 of 3 runs. No instability was observed in either direction during this experiment.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.
## Experiment 54U — Can a Clarification Target Become a Useful User Question Without Adding New Meaning? (2026-08-08)
### Objective
Given a fixed clarification target, can the model turn that target into one concise, neutral clarification question without adding assumptions, choosing a side, or broadening the issue?
This is test-only.
Do not integrate anything into the UI.
Do not change Behaviour Selection.
Do not change production prompts.
### Hypothesis
Given a specific clarification target, the model may be able to produce a single question that:
- asks only about the unresolved distinction;
- remains neutral between the interpretations;
- does not introduce new assumptions;
- does not ask for evidence instead;
- does not become a multi-part interview.
If it broadens the question or adds new meaning, record that failure.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54S / 54T)
Model: `qwen-claude:latest` (same as 54S / 54T)
No localhost fallback. No experiment-specific model variable.
### Number of Live Inference Calls
Exactly **3** live Ollama calls — one per case.
### Semantic Instruction Used
> Write one concise clarification question that asks only about the supplied clarification target. Keep it neutral between the possible meanings. Do not introduce new facts, assumptions, evidence requests, recommendations, or additional questions. Do not explain why you are asking.
No examples. No expected wording. Same instruction for all three cases.
### Input Contract
Each call receives:
```json
{ "source": "...", "clarificationTarget": "..." }
```
The target is fixed human-reviewed input. The model must not decide whether clarification is needed.
### Output Contract
Return exactly:
```json
{ "question": "one clarification question" }
```
No explanation, score, confidence, answer options, recommendation, preferred interpretation, or second question.
### Case 1 — Preference Versus Hard Constraint
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
**Human-reviewed intent:** A correct question should ask whether "no more risk" is an absolute boundary or something the user would trade against growth.
### Case 2 — Meaning of "Affordable"
**Source:** "I want to replace the system, but the new option needs to be affordable."
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
**Human-reviewed intent:** A correct question should clarify which meaning of affordability the user intends.
### Case 3 — Private Factual Constraint
**Source:** "I could move the project forward next month, depending on whether I actually have enough time."
**Clarification target:** whether the user has enough available time next month to take on the project.
**Human-reviewed intent:** A correct question should ask the user to clarify their own available capacity/time.
### Results
#### Case 1 — Preference Versus Hard Constraint
**Returned question:** "Do you view avoiding additional risk as a preference/trade-off or a hard constraint?"
**Classification:** question_correct — semantically aligned with target (6 overlapping words), structurally valid, neutral.
**Did it stay within the target?** Yes — asks exactly whether risk avoidance is a preference/trade-off or a hard constraint.
**Did it add assumptions?** No.
**Did it ask for evidence?** No.
**Multiple questions?** No — one substantive question.
**Implied preferred interpretation?** No.
#### Case 2 — Meaning of "Affordable"
**Returned question:** 'Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?'
**Classification:** question_correct — semantically aligned with target (5 overlapping words), structurally valid, neutral.
**Did it stay within the target?** Yes — asks which meaning of affordability the user intends.
**Did it add assumptions?** No.
**Did it ask for evidence?** No.
**Multiple questions?** No.
**Implied preferred interpretation?** No.
#### Case 3 — Private Factual Constraint
**Returned question:** "Do you have enough available time next month to take on the project?"
**Classification:** question_correct — semantically aligned with target (7 overlapping words), structurally valid, neutral.
**Did it stay within the target?** Yes — asks for the user's own capacity clarification.
**Did it add assumptions?** No — did not estimate hours or assume availability.
**Did it ask for evidence?** No.
**Multiple questions?** No.
**Implied preferred interpretation?** No.
### Timing
- Total time: 71,758ms (71.8s)
- Average: 23,919.3ms per call
- Fastest: 13,573ms (Case 3)
- Slowest: 33,314ms (Case 2)
### Required Questions — Answers
1. Did Case 1 ask only about preference/trade-off versus hard constraint? **Yes**
2. Did Case 2 ask only what "affordable" means? **Yes**
3. Did Case 3 correctly ask for the user's private factual capacity? **Yes**
4. Did any question introduce assumptions not present in the source/target? **No**
5. Did any question ask for evidence instead of clarification? **No**
6. Did any question contain more than one substantive question? **No**
7. Did any question choose or imply a preferred interpretation? **No**
8. How many cases were question_correct / question_failed? **3 correct, 0 failed.**
9. Does this establish when the question should be asked? **No.**
10. Does this establish how the answer should update reasoning state? **No.**
11. Does this establish that the question will feel good in the UI? **No.**
### Evaluation Conclusion
**The model produced a clean clarification question for every tested target.**
All three cases returned one neutral question addressing only the supplied clarification target, with no introduced assumptions, evidence requests, multi-part structure, or implied preferred interpretations.
**Corrected conclusion:** The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Word-overlap metrics are diagnostic only; manual semantic review is the stronger evidence.
### Limitations
- Only three cases tested; limited domain coverage.
- Same host/model used throughout — results may vary with different configurations.
- Does not establish when the question should be asked (that is a separate step).
- Does not establish how answers should update reasoning state.
- Semantic quality assessed through structural and overlap heuristics; manual review would strengthen confidence.
- Does not establish that the question will feel good in the UI.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
## Experiment 54V — Can a Clarification Answer Resolve Only the Target Without Rewriting Everything Else? (2026-08-08)
### Objective
Given the original source, a fixed clarification target, the clarification question, and the user's answer, can the model identify what has now been resolved without adding new meaning or rewriting unrelated reasoning?
This is test-only.
Do not integrate with the graph, Behaviour Selection, or UI.
### Hypothesis
A clarification answer should be able to resolve one narrow user-owned ambiguity without causing the model to:
- reinterpret the whole source;
- add unsupported consequences;
- reopen unrelated uncertainty.
If the model cannot preserve that boundary, clarification answers may create as much ambiguity as they remove.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54S / 54T / 54U)
Model: `qwen-claude:latest` (same as 54S / 54T / 54U)
No localhost fallback. No experiment-specific model variable.
### Number of Live Inference Calls
Exactly **3** live Ollama calls — one per case.
### Input Contract
Each call receives:
```json
{ "source": "...", "clarificationTarget": "...", "clarificationQuestion": "...", "userAnswer": "..." }
```
### Output Contract
Return exactly:
```json
{ "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
```
No next question, recommendation, confidence score, graph update, extra interpretation, or explanation.
### Semantic Instruction Used
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
No examples. No expected answers. Same instruction for all three cases.
### Case 1 — Hard Constraint Resolved
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
**User answer:** It's a hard constraint. I don't want any increase in risk.
**Human reference:**
- resolvedMeaning: "avoiding additional risk is a hard constraint"
- targetResolved: true
- remainingUncertainty: null
Must not infer: growth is impossible; which growth option should be chosen; how much risk currently exists.
### Case 2 — Definition Resolved
**Source:** "I want to replace the system, but the new option needs to be affordable."
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
**Clarification question:** Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?
**User answer:** I care about the total cost over five years, not the upfront price.
**Human reference:**
- resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost"
- targetResolved: true
- remainingUncertainty: null
Do not invent a budget or preferred system.
### Case 3 — Answer Does Not Fully Resolve Target
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
**User answer:** It depends on the opportunity.
**Human reference:**
- The target is **not fully resolved**.
- targetResolved: false
- remainingUncertainty should preserve that the acceptable trade-off still depends on circumstances.
- resolvedMeaning may state only that the user's risk position is conditional on the opportunity.
Do not force the answer into either preference or hard-constraint category.
### Results
#### Case 1 — Hard Constraint Resolved
**Returned result:**
- resolvedMeaning: "Avoiding additional risk is established as a hard constraint, meaning no increase in risk is acceptable."
- targetResolved: true
- remainingUncertainty: null
**Manual semantic review:**
- Used only the user answer to resolve the target? **Yes.**
- Preserved unresolved ambiguity where present? N/A (target was resolved).
- Avoided wider consequences? **Yes** — did not state growth is impossible, did not choose a growth option, did not estimate current risk levels. The phrase "meaning no increase in risk is acceptable" is a direct restatement of the user answer ("I don't want any increase in risk"), not an addition from the source.
- Avoided adding unsupported facts? **Yes.**
- Avoided generating another question? **Yes.**
**Classification:** resolution_correct
#### Case 2 — Definition Resolved
**Returned result:**
- resolvedMeaning: "Affordable refers to a low overall or long-term cost rather than a low upfront price."
- targetResolved: true
- remainingUncertainty: null
**Manual semantic review:**
- Used only the user answer to resolve the target? **Yes.**
- Preserved unresolved ambiguity where present? N/A (target was resolved).
- Avoided wider consequences? **Yes** — did not invent a budget amount, did not recommend a system.
- Avoided adding unsupported facts? **Yes.**
- Avoided generating another question? **Yes.**
**Classification:** resolution_correct
#### Case 3 — Answer Does Not Fully Resolve Target
**Returned result:**
- resolvedMeaning: "Avoiding additional risk is conditional rather than fixed, varying based on the specific opportunity."
- targetResolved: false
- remainingUncertainty: "It remains unclear whether avoiding additional risk is strictly a preference/trade-off or a hard constraint, as the stance depends on the opportunity."
**Manual semantic review:**
- Used only the user answer to resolve the target? **Yes.** The model preserved the conditionality present in "It depends on the opportunity" without forcing into either category.
- Preserved unresolved ambiguity where present? **Yes.** Correctly kept the target unresolved and described the remaining uncertainty specifically about that target.
- Avoided wider consequences? **Yes.**
- Avoided adding unsupported facts? **Yes.**
- Avoided generating another question? **Yes.**
**Classification:** resolution_correct
### Timing
- Total time: 38,052ms (38.1s)
- Average: 12,684.0ms per call
- Fastest: 8,479ms (Case 1)
- Slowest: 15,208ms (Case 3)
### Required Questions — Answers
1. Did Case 1 resolve the target to a hard constraint without adding wider consequences? **Yes.** The resolved meaning stays within the user answer's scope. No inference about growth feasibility, option selection, or current risk levels.
2. Did Case 2 resolve "affordable" to long-term cost without inventing a budget? **Yes.** The model correctly captured the five-year perspective without adding any budget figure or system recommendation.
3. Did Case 3 correctly keep the target unresolved? **Yes.** The model returned targetResolved=false, preserved conditionality in resolvedMeaning, and provided meaningful remainingUncertainty.
4. Did any case force an ambiguous answer into a stronger meaning? **No.** Case 3's conditional answer was kept at its actual strength — neither strengthened to preference nor to hard constraint.
5. Did any case rewrite unrelated parts of the source? **No.** In Cases 1 and 3 (same source), the model treated the "grow" portion identically to the original source meaning without reinterpreting it.
6. Did any case generate another question? **No.** All resolvedMeaning fields are statements, not questions.
7. How many cases were resolution_correct / resolution_failed? **3 correct, 0 failed.**
8. Does this establish how graph state should be updated? **No.** This only tests semantic recognition of what a clarification answer resolves; it does not test any graph update mechanism.
9. Does this establish what question should come next? **No.** The experiment tested one directional step (answer → resolved meaning) and did not test the next question generation cycle.
10. Does this establish how Behaviour Selection should react? **No.** No behaviour selection logic was tested or involved.
### Evaluation Conclusion
**Clarification answers resolved only the intended target across all tested cases.**
All three cases returned correct resolution boundaries: Cases 1 and 2 settled the target cleanly; Case 3 preserved incomplete information at its actual strength without forcing it into a stronger category. The model did not widen beyond the clarification target, invent consequences, or generate new questions in any case.
**The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.**
### Limitations
- Only three cases tested; limited domain coverage (risk constraint, affordability definition, conditional constraint).
- Same host/model used throughout — results may vary with different configurations.
- Does not establish how graph state should update from resolved meanings.
- Does not establish what question should come next after resolution.
- Does not establish how Behaviour Selection should react to resolved vs unresolved targets.
- Semantic quality assessed through structural checks and manual review; broader generalisation untested.
- Case 3's remainingUncertainty output is longer than the human reference — acceptable because it describes the uncertainty rather than adding meaning, but worth noting.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
docs/archive/experiments/vol-1-chapters/ch14/54W-54Z-clarification-chain-and-target-broadening.md
+659
View File
@@ -0,0 +1,659 @@
## Experiment 54W — Does the Clarification Chain Hold Together End to End? (2026-08-08)
### Objective
Tighten Experiment 54V so it does not overstate the isolated clarification-chain results. Then test the smallest end-to-end version of the clarification path: **starting from one disagreement, can the semantic steps correctly determine whether the user is needed, identify the clarification target, word one question, and use the user's answer to resolve only that target without semantic drift between steps?**
This is still test-only. Do not integrate with the active engine, graph, Behaviour Selection, or UI.
### Hypothesis
The individual clarification steps may remain aligned when chained together. For a genuine user-owned ambiguity, the chain should preserve: `disagreement → user required → clarification target → neutral question → answer → resolved target`. For an evidence-resolvable disagreement, the chain should stop early rather than inventing a clarification target or question. If the steps drift when connected, record exactly where the first material divergence occurs.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54R–54V)
Model: `qwen-claude:latest` (same as 54R–54V)
### Number of Live Inference Calls
Exactly **5** live Ollama calls — 4 for Scenario A + 1 for Scenario B.
### Context Used
- `docs/current-handoff.md`
- Experiment 54V only in `docs/design-evolution-log.md`
- `tests/reconstruction/semantic-disagreement-resolution-source.test.js`
- `tests/reconstruction/semantic-clarification-target.test.js`
- `tests/reconstruction/semantic-clarification-question.test.js`
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js`
### Scenarios
#### Scenario A — Genuine User-Owned Ambiguity
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Fixed disagreement:**
1. growth should be prioritised even if some additional risk is unavoidable;
2. avoiding additional risk is a hard constraint even if growth is slower.
**Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
#### Scenario B — Evidence-Resolvable Disagreement
**Source:** "Orders are arriving late and customers have started complaining."
**Fixed disagreement:**
1. delays may be caused by insufficient staff capacity;
2. delays may be caused by unreliable supplier lead times.
### Stage-by-Stage Outputs
#### Scenario A — Full Chain
**A1 — Resolution Source (Experiment 54R instruction)**
- **Actual output:** `{ requiresUserClarification: true, reason: "The disagreement centers on a priority trade-off between growth and risk avoidance that depends entirely on the user's underlying constraints rather than external facts." }`
- **Input used:** original source + fixed disagreement + evidence needs
- **Classification:** `stage_correct` — correctly identified that user clarification is required for a genuine user-owned ambiguity. The reason references subjective constraints (user-owned) not external facts, consistent with the true classification.
**A2 — Clarification Target (Experiment 54S instruction)**
- **Input target used from A1:** `requiresUserClarification: true` (actual upstream output, NOT replaced by a human reference)
- **Actual output:** `{ clarificationTarget: "Priority between business growth and risk avoidance when they conflict" }`
- **Classification:** `stage_correct` — the model correctly identified the user-owned distinction. The target focuses on priority ordering between two competing interests (growth vs. risk), which is the exact user-owned ambiguity present in the source. No question language, no evidence framing, no interpretation chosen.
**A3 — Clarification Question (Experiment 54U instruction)**
- **Input target from A2:** `"Priority between business growth and risk avoidance when they conflict"` (actual upstream output)
- **Actual output:** `{ question: "Which do you prioritize when business growth and risk avoidance come into conflict?" }`
- **Classification:** `stage_correct` — single neutral question, ends with one question mark, no biased framing. The question uses the actual A2 target's language ("priority", "business growth", "risk avoidance") without substituting a human reference. Neutral between alternatives: does not say you should prioritize growth or risk.
**A4 — Answer Resolution (Experiment 54V instruction)**
- **Input target from actual A2:** `"Priority between business growth and risk avoidance when they conflict"`
- **Input question from actual A3:** `"Which do you prioritize when business growth and risk avoidance come into conflict?"`
- **Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
- **Actual output:** `{ resolvedMeaning: "Risk avoidance takes absolute priority over business growth, as no increase in risk is acceptable regardless of growth potential.", targetResolved: true, remainingUncertainty: null }`
- **Classification:** `stage_correct` — targetResolved=true correctly reflects that the user's answer settles the priority question. The resolved meaning interprets "hard constraint" and "no increase in risk" as absolute priority (non-negotiable boundary) over growth. This is a valid semantic interpretation of the user's answer within the A2/A3 framing. No wider consequences introduced: does not state growth is impossible, does not choose a specific growth option, does not estimate current risk levels.
#### Scenario B — Evidence-Resolvable Stop
**B1 — Resolution Source (Experiment 54R instruction)**
- **Actual output:** `{ requiresUserClarification: false, reason: "The disagreement can be resolved by examining operational metrics and supplier delivery records rather than requiring subjective input." }`
- **Classification:** `correct_stop` — correctly identified that evidence (operational data) can resolve the cause of delays. The chain stopped immediately; no clarification target call was made, no question was generated, no answer-resolution was attempted.
### Actual Upstream Outputs Passed Between Stages
| Stage | Received From | Value |
|---|---|---|
| A2 | A1 | `requiresUserClarification: true`, reason about subjective constraints |
| A3 | A2 | `clarificationTarget: "Priority between business growth and risk avoidance when they conflict"` |
| A4 | A2 | Same A2 target (same as above) |
| A4 | A3 | `question: "Which do you prioritize when business growth and risk avoidance come into conflict?"` |
Human reference data was used only for evaluation — not silently substituted between stages.
### First Drift Point in Scenario A
**No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario.** The chain preserved the user-owned nature of the ambiguity from A1 through to resolution at A4 without introducing unsupported meaning or changing the interpretation of upstream results.
### Scenario B — Stop Verification
- **Did Scenario B stop after B1:** Yes
- **Were any unnecessary clarification calls made for Scenario B:** No (0 additional calls)
### Question: Did any stage choose a winner?
**No.** None of the stages selected an interpretation as correct or better. A4's resolved meaning states what the user's answer settled (priority resolution) rather than declaring one pre-existing interpretation as the winner. The chain reports what was clarified, not which side of the original disagreement is right.
### Question: Did any stage introduce unsupported meaning that materially affected the next stage?
**No.** A2 stayed within the priority dimension present in the source. A3 preserved both competing terms ("business growth", "risk avoidance") from the A2 target. A4 interpreted the user's hard-constraint answer as absolute priority over growth — a valid reading given the answer and the A2/A3 framing. No stage added external facts or consequences that materially distorted downstream reasoning.
### Question: Evidence that isolated clarification steps survive under chaining
**Yes.** The chain_correct result demonstrates that all four individual capabilities (resolution source, target identification, question wording, answer resolution) remained usable when chained in the two tested scenarios. Each stage's output was a valid input for the next stage. No stage degraded or produced an unexpected format. The semantic proximity between A2 and A4 is worth noting: A2 framed the distinction as "priority" while the user answer used "hard constraint" — these are semantically close but not identical (a hard constraint is stronger than a priority preference). A4 correctly interpreted the hard-constraint answer within the priority framing, so this proximity was sufficient for alignment.
### Questionable or Unsupported Findings
- Only **one** instance of each scenario was tested. Chain stability across repeated runs needs verification.
- Only the growth-versus-risk domain was tested for Scenario A. Different domains may produce different drift patterns.
- A2's output ("Priority between business growth and risk avoidance when they conflict") lost the "preference/trade-off vs hard constraint" distinction present in the 54S human reference. This loss of granularity is not a failure per se — it is still correct within its contract — but it means downstream stages operate on a less precise target. The chain succeeded with this coarser representation, which is evidence that the steps tolerate some semantic imprecision.
### Experiment Conclusion
**The clarification chain remained semantically aligned end to end in both tested scenarios.** For Scenario A (genuine user-owned ambiguity), all four stages produced correct outputs and each stage's actual output was a valid input for the next stage with no material drift. For Scenario B (evidence-resolvable disagreement), the model correctly stopped after the first decision without inventing unnecessary clarification steps.
### Focused Test Result
`chain_correct` for Scenario A + `correct_stop` for Scenario B. 5/5 live calls completed within budget. All stage assertions passed.
### Historical Comparison Result
The chain_correct result is new evidence not available in any earlier experiment (54R–54V tested isolated steps only). It demonstrates that the individual clarification capabilities remained usable when chained in the two tested scenarios, on one scenario and one model configuration. This does not extend to production integration readiness.
### Documentation Updated
- `docs/design-evolution-log.md` — added 54V clarifying caveat; added full Experiment 54W entry
- `docs/current-handoff.md` — added 54V clarifying caveat
### Confirmation Host and Model Remained Unchanged
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54V.
### Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file, not production prompts.
### Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
### Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
### Confirmation No Clarification-Chain Logic Entered Active Runtime
This experiment created one new test file only. No clarification-chain logic entered any active runtime path, production module, or behaviour selection output.
### Return-to-Work Note (Experiment 54W)
54R–54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-chain.test.js` for the full experiment and results. Status pending Rob's review.
## Experiment 54X — Does the Clarification Target Lose Important Specificity When Chained? (2026-08-08)
### Objective
Isolate whether the clarification-target-generation step preserves the exact user-owned distinction or broadens it, using three fixed cases under the same instruction as Experiment 54S. Passive and test-only. No question generation, answer resolution, Behaviour Selection, graph, or UI integration attempted. Same host/model. Branch: `feature/user-workspace-ux-v0.7`.
### Hypothesis
The model may preserve clarification targets well when the ambiguity is simple and explicit, but broaden targets when the distinction is relational or preference-based. If broadening happens repeatedly, that may matter downstream because the question-generation step can only be as precise as the target it receives.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54R–54W)
Model: `qwen-claude:latest` (same as 54R–54W)
### Number of Live Inference Calls
Exactly **3** live Ollama calls — one per case.
### Context Used
- `docs/current-handoff.md`
- Experiment 54W only in `docs/design-evolution-log.md` (as historical context for the broadening observation)
- `tests/reconstruction/semantic-clarification-target.test.js` (for structural reference)
- `tests/reconstruction/semantic-clarification-chain.test.js` (for structural reference)
### Experiment 54W Corrections Applied
Replaced "No drift was detected." with: "**No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario.**"
Replaced "the individual clarification capabilities survive end-to-end chaining" with: "**The individual clarification capabilities remained usable when chained in the two tested scenarios.**"
### Case 1 — Preference Versus Hard Constraint
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
**Human reference target:** whether avoiding additional risk is a preference/trade-off or a hard constraint
**Actual model output:** `"preferred priority between business growth and risk avoidance"`
**Specificity classification:** **target_broadened** — On the right topic but broadened from the material distinction (preference/trade-off vs. hard constraint) to general priority ordering. Usable downstream but not fully specific.
### Case 2 — Definition Ambiguity
**Source:** "I want to replace the system, but the new option needs to be affordable."
**Disagreement:** affordable means keeping upfront cost low / affordable means keeping total long-term cost low.
**Human reference target:** whether "affordable" means low upfront cost or low overall/long-term cost
**Actual model output:** `"whether 'affordable' refers to upfront cost or total long-term cost"`
**Specificity classification:** **target_specific** — Preserved the material distinction: upfront cost versus total long-term cost. The distinction is explicit and identical in meaning to the human reference.
### Case 3 — Private Factual Boundary
**Source:** "I could move the project forward next month, depending on whether I actually have enough time."
**Disagreement:** the user has enough available time next month / the user does not have enough available time next month.
**Human reference target:** whether the user has enough available time next month to take on the project
**Actual model output:** `"whether the user has enough available time next month"`
**Specificity classification:** **target_specific** — Preserved all three required elements: time availability, next month, and the capacity question. The omission of "to take on the project" does not lose material specificity — it is implied by the source context.
### Inference Timing
| Metric | Value |
|---|---|
| Total live calls | 3 |
| Total inference time | 49,220 ms |
| Average | 16,406.7 ms per call |
| Fastest | 11,815 ms (Case 2) |
| Slowest | 21,303 ms (Case 1) |
### Results Summary
| Classification | Count |
|---|---|
| target_specific | 2/3 |
| target_broadened | 1/3 |
| target_wrong | 0/3 |
### Questions Answered
1. Did Case 1 preserve preference/trade-off versus hard constraint? **No** — broadened to priority ordering.
2. Did Case 2 preserve upfront versus long-term affordability? **Yes** — preserved explicitly.
3. Did Case 3 preserve the user's available-time boundary? **Yes** — preserved explicitly with all three required elements.
4. How many cases were target_specific / target_broadened / target_wrong? **2 / 1 / 0**
5. Did any target remain usable while still losing material specificity? **Yes** — Case 1 was broadly relevant and actionable but lost the preference-versus-constraint distinction.
6. Did any target introduce unsupported meaning? **No** — no case introduced concepts not present in source or disagreement.
7. Does this reproduce the broadening observed in 54W? **Yes** — both experiments show broadening from preference/constraint to priority framing on Case 1-style input.
8. Does this establish why broadening happens? **No** — one isolated result per case cannot determine causality; only that it does occur for at least one ambiguity pattern.
9. Does this establish whether a broader target is acceptable for the user journey? **No** — acceptability depends on downstream question quality and user experience, which were not tested here.
10. Does this establish how the clarification question should be worded? **No** — no question-generation step was involved.
### Evidence About Clarification-Target Specificity
A clarification target can be broadly relevant without being precise enough. Case 1's output ("preferred priority between business growth and risk avoidance") is clearly about the right topic and usable downstream, but it does not preserve the material distinction that the user actually needs to clarify — whether avoiding additional risk is a preference or a hard constraint. Cases 2 and 3 show that the same instruction can produce fully specific targets when the ambiguity involves definition boundaries or private facts rather than preference-versus-constraint relationships.
### Limitations
- Only one model configuration was tested (qwen-claude:latest). Different models may behave differently.
- Only one inference per case — stability across repeated runs is untested here (though 54L previously showed strong stability for other tasks).
- The broadening pattern only emerged in Case 1; the instruction and model appear capable of specificity on other patterns.
- No downstream question or answer-resolution step was tested — usability of a broader target cannot be fully assessed without those stages.
### Experiment Conclusion
Clarification targets remained usable but broadened in one of three tested cases (Case 1). The broadening reproduced the same pattern observed in 54W: preference-versus-constraint distinctions tend to become priority-ordering framings. This is not a failure — the target remains actionable — but Specificity loss occurred in one of the three tested ambiguity patterns and was absent in the other two.
### Focused Test Result
2/3 targets preserved material distinction; 1/3 broadened (matching 54W pattern). All structural assertions passed. No invariant violations detected.
### Historical Comparison Result
The Case 1 result reproduces the A2 output from Experiment 54W ("Priority between business growth and risk avoidance when they conflict" → "preferred priority between business growth and risk avoidance"). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation.
### Documentation Updated
- `docs/design-evolution-log.md` — added full Experiment 54X entry
- `docs/current-handoff.md` — updated Return-to-Work note with 54X findings; applied 54W wording corrections
### Confirmation Host and Model Remained Unchanged
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54X.
### Confirmation Semantic Instruction and Output Contract Remained Unchanged
The instruction was identical to Experiment 54S (no examples, no stronger coaching). The output contract remained `{ "clarificationTarget": "short statement" }` — unchanged from 54S.
### Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instruction defined in this test file.
### Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
### Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
### Confirmation No Clarification-Target Logic Entered Active Runtime
This experiment created one new test file only. No clarification-target logic entered any active runtime path, production module, or behaviour selection output.
### Return-to-Work Note (Experiment 54X)
54W showed the full clarification chain worked in the tested pair but Stage A2 broadened one target; 54X isolated target specificity using three clarification cases under the same 54S instruction — preference/constraint distinction was lost to priority framing (broadened), affordability definition stayed precise (specific), and private factual capacity stayed precise (specific). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target-specificity.test.js` for the full experiment and results. Status pending Rob's review.
## Experiment 54Y — Does a Broader Clarification Target Actually Change the User Question or Resolution? (2026-08-08)
### Objective
Test whether the specificity loss observed in Experiments 54W/54X actually matters downstream. If the clarification target shifts from the precise distinction "preference/trade-off versus hard constraint" to the broader "priority between growth and risk," does that materially change the question asked or the meaning resolved from the user's answer? Passive and test-only. No redesign of target generation, no integration with Behaviour Selection, graph, or UI.
### Hypothesis
The broader target may remain workable but could alter the user-facing distinction. Specifically:
- the precise target may ask whether risk avoidance is a hard boundary or a trade-off;
- the broader target may instead ask which objective has priority.
Those questions are related, but the user's answers need not mean exactly the same thing.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54R–54X)
Model: `qwen-claude:latest` (same as 54R–54X)
### Number of Live Inference Calls
Exactly **4** live Ollama calls — one question per variant + one answer-resolution per variant.
### Context Used
- `docs/current-handoff.md`
- Experiment 54X only in `docs/design-evolution-log.md` (as historical context for the broadening observation)
- `tests/reconstruction/semantic-clarification-question.test.js` (for structural reference: instruction and output contract)
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (for structural reference: instruction and output contract)
### Fixed Scenario
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
### Variant A — Precise Target
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
This is the human-reviewed specific target.
### Variant B — Broadened Target
**Clarification target:** priority between business growth and risk avoidance when they conflict.
This mirrors the broader target observed in Experiments 54W and 54X.
---
### Stage 1 Results — Question Generation
| Variant | Generated Question |
|---|---|
| A (Precise) | Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off? |
| B (Broadened) | When business growth and risk avoidance conflict, which do you prioritize? |
**Question analysis:** Variant A frames the question around whether avoiding risk is a hard constraint or a preference/trade-off — directly addressing the boundary distinction. Variant B frames it around priority ordering between growth and risk when they conflict — reframing the decision as relative importance rather than a boundary question. The model did not impose a reference wording for Variant B; it produced its natural framing from the broadened target.
### Stage 2 Results — Answer Resolution
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | Avoiding additional risk is established as a hard constraint. | true | null |
| B (Broadened) | Risk avoidance takes absolute priority over business growth when they conflict. | true | null |
**Resolution analysis:** Both variants produced `targetResolved: true` with `remainingUncertainty: null`. The resolved meanings differ in wording but convey materially equivalent meaning for downstream reasoning: "avoiding additional risk is a hard constraint" and "risk avoidance takes absolute priority over business growth when they conflict" establish the same boundary — no more risk will be accepted. Neither resolution introduced unsupported wider consequences.
### Question Equivalence Classification
**`questions_materially_different`**
The precise target asked whether avoiding extra risk is a trade-off/preference or a hard constraint (a boundary question). The broadened target asked which objective has priority when they conflict (an ordering question). These ask the user to resolve different conceptual distinctions. The distinction was lost as predicted.
### Resolution Equivalence Classification
**`resolutions_materially_equivalent`**
Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
### Questions Answered
1. **What question did the precise target generate?** "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?"
2. **What question did the broadened target generate?** "When business growth and risk avoidance conflict, which do you prioritize?"
3. **Did both questions ask the user to resolve the same underlying distinction?** No — one asked about boundary (constraint vs trade-off); the other asked about priority ordering.
4. **Did the broader target turn preference-versus-constraint into simple priority ordering?** Yes — it reframed the distinction as relative importance rather than an absolute boundary.
5. **What resolved meaning did Variant A produce from the fixed answer?** "Avoiding additional risk is established as a hard constraint." (targetResolved: true)
6. **What resolved meaning did Variant B produce from the same answer?** "Risk avoidance takes absolute priority over business growth when they conflict." (targetResolved: true)
7. **Were the two resolved meanings materially equivalent?** Yes — both establish that no additional risk will be accepted for growth.
8. **Did either variant leave remaining uncertainty?** No — both returned null, indicating full resolution of the target from the fixed answer.
9. **Did either variant introduce unsupported wider consequences?** No — neither inference extended beyond the source meaning and the user's explicit answer.
10. **Does the broadening materially affect downstream clarification in this tested scenario?** No — despite different questions, the same answer produced the same downstream meaning.
11. **Does this establish that broad targets are generally safe or unsafe?** No — only one scenario tested.
12. **Does this establish how target generation should be changed?** No — no fix designed from these results.
13. **Does this establish UI behaviour?** No — this is a clarification-target test only.
### Limitations
- Only one source scenario and one fixed answer were tested. Different sources may behave differently.
- Only one ambiguity pattern (preference/constraint) was tested for downstream consequence. Other patterns not assessed.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
- The semantic equivalence classification is based on structured heuristic checks supplemented by the test output — for definitive judgment, human review of the actual resolved meanings is required.
- One tested ambiguity pattern; broader safety/generalisation remains untested.
### Experiment Conclusion
**The broader target changed the clarification question but not the resolved meaning for the tested explicit answer.**
The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
The key finding is: **Does the distinction we lost actually matter? — In this tested scenario, it did not.** Specificity loss is not automatically a failure; it depends on whether it changes downstream meaning. Whether this holds across other scenarios and ambiguity patterns remains untested.
### Focused Test Result
Both variants produced `targetResolved: true` with zero remaining uncertainty and zero unsupported inferences. Resolutions were materially equivalent despite questions being materially different. 4/4 live inference calls completed successfully (all tests passed).
### Historical Comparison Result
Variant A's resolution ("Avoiding additional risk is established as a hard constraint") matches the expected outcome from Experiment 54V Case 1 and the handoff summary. Variant B's resolution ("Risk avoidance takes absolute priority over business growth when they conflict") represents a coarser framing — but not an incorrect one — for downstream use. The result confirms that the coarser representation remains workable even when it loses the preference-versus-constraint granularity.
### Documentation Updated
- `docs/design-evolution-log.md` — added full Experiment 54Y entry; applied 54X wording corrections
- `docs/current-handoff.md` — updated with Experiment 54Y summary and new Return-to-Work note
### Confirmation Host and Model Remained Unchanged
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54X.
### Confirmation Semantic Instructions and Output Contracts Remained Unchanged
Both question and resolution instructions were identical to those defined in Experiments 54U and 54V. Output contracts unchanged from 54U (`{ "question": "..." }`) and 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`).
### Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
### Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
### Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
### Confirmation No Specificity-Consequence Logic Entered Active Runtime
This experiment created one new test file only. No specificity-consequence logic entered any active runtime path, production module, or behaviour selection output.
---
### Return-to-Work Note (Experiment 54Y)
Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
## Experiment 54Z — Does Target Broadening Change Resolution When the Answer Is Less Explicit? (2026-08-08)
### Objective
Experiment 54Y showed that with a very explicit hard-constraint answer ("It's a hard constraint. I don't want any increase in risk."), both precise and broadened clarification targets converged on materially equivalent resolved meaning — even though the generated questions were materially different.
This leaves one unresolved consequence:
> If the user's answer is less explicit, do those two different questions lead to materially different resolved meaning?
54Z tests that only. Passive and test-only. No redesign of target generation. No integration with Behaviour Selection, graph, or UI. No production code changes.
### Hypothesis
The precise and broadened questions may behave differently when the answer does not explicitly name the missing distinction. A weaker answer could:
- remain correctly unresolved under the precise question;
- but be interpreted as a resolved priority decision under the broader question.
If that happens, target broadening has a real downstream consequence. If both variants preserve equivalent uncertainty, the broadening may be less consequential than expected.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54R–54Y)
Model: `qwen-claude:latest` (same as 54R–54Y)
### Number of Live Inference Calls
Exactly **4** live Ollama calls — one answer-resolution per variant × two weaker answers. No question-generation calls (questions are fixed from Experiment 54Y).
### Context Used
- `docs/current-handoff.md`
- Experiment 54Y in `docs/design-evolution-log.md` (as basis for the unresolved consequence)
- `tests/reconstruction/semantic-clarification-specificity-consequence.test.js` (structural reference: question generation and answer resolution helpers)
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract)
### Fixed Source
> I want the business to grow, but I don't want to take on more risk.
### Variant A — Precise Target
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
**Fixed question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
### Variant B — Broadened Target
**Clarification target:** priority between business growth and risk avoidance when they conflict.
**Fixed question:** When business growth and risk avoidance conflict, which do you prioritize?
---
### Answer 1 — Priority Without Constraint Meaning
**User answer:** "Risk matters more to me."
**Expected behavior (human-reviewed):**
This answer does not clearly establish whether risk avoidance is a hard constraint or merely a stronger preference. For Variant A, the precise target should therefore remain unresolved. For Variant B, the answer may legitimately resolve the priority target as "risk avoidance has higher priority than growth."
### Answer 1 Results
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | The user treats avoiding additional risk as a strong priority or preference rather than an absolute, non-negotiable constraint. | true | null |
| B (Broadened) | Risk avoidance is prioritized over business growth when they conflict. | true | null |
### Answer 1 Analysis
Variant A interprets "Risk matters more to me" as meaning risk avoidance is a **strong preference/priority rather than an absolute constraint** — this maps correctly onto the precise target (preference/trade-off vs hard constraint). The model marked targetResolved=true because it interpreted the answer as settling the distinction toward "preference/trade-off."
Variant B interprets the same answer as meaning **risk avoidance has higher priority over growth when they conflict** — this maps correctly onto the broadened target (priority ordering).
**Classification: resolutions_materially_equivalent**
Both variants map the weak answer to a preference/priority-over-constraint interpretation. Neither resolves to "hard constraint." The resolved meanings use different framing but preserve the same downstream reasoning state: risk is not an absolute boundary, it is a prioritized consideration. For downstream use (what can/cannot be done), both produce equivalent uncertainty about whether risk could ever be accepted.
Both variants were flagged as `potential_erasal_of_uncertainty` because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases. **The precise target may have erased uncertainty by inferring that "Risk matters more to me" means risk is not a hard constraint; that conclusion was not explicitly supplied by the user.**
---
### Answer 2 — Conditional Trade-Off
**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
**Expected behavior (human-reviewed):**
This answer indicates risk avoidance is not an absolute hard constraint; some trade-off may be acceptable depending on the opportunity. For Variant A, this should resolve away from "hard constraint" while retaining conditionality. For Variant B, it may establish that risk is normally prioritized but can yield to growth in some cases.
### Answer 2 Results
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|---|---|---|---|
| A (Precise) | Avoiding additional risk is a preference or trade-off rather than a hard constraint. | true | null |
| B (Broadened) | Default priority is risk avoidance, with a conditional willingness to accept some risk for highly suitable opportunities. | false | It remains unclear how "the right opportunity" is defined and which factor strictly takes precedence when a specific growth opportunity carries significant risk. |
### Answer 2 Analysis
**This is the critical divergence.** Variant A collapses the conditional nature of the answer into a simple preference-vs-constraint resolution. The model says "preference or trade-off rather than a hard constraint" — but loses the key information that there are conditions (the right opportunity) under which even this preference could shift. This was flagged as `forced_certainty_detected` and `potential_erasal_of_uncertainty`.
Variant B preserves the conditionality ("conditional willingness") and correctly marks targetResolved=false because the answer does not establish a stable priority — the priority shifts depending on context. It also identifies remaining uncertainty about what constitutes "the right opportunity."
**Classification: resolutions_materially_different**
This is a material divergence. Variant A erases the conditional nature of the user's stated position and produces a flat preference-versus-constraint resolution. Variant B preserves both the default-priority-and-conditional structure AND the remaining uncertainty about when conditions change. For downstream reasoning, this matters because:
- Under Variant A's meaning: risk avoidance = preference/trade-off → may be willing to accept risk in some cases (inferred)
- Under Variant B's meaning: default priority risk, conditionally willing → conditional willingness is preserved explicitly
**However**, the divergence exists primarily in remainingUncertainty content, not in the resolvedMeaning itself. Both agree that risk avoidance is not a hard constraint. The difference is in whether the model preserves "there are conditions we don't yet understand" versus collapsing everything to "not a hard constraint."
---
### Questions Answered
1. **For Answer 1, did Variant A correctly preserve uncertainty about preference versus hard constraint?** Partially. Variant A mapped the answer toward "preference/trade-off rather than absolute constraint" but marked it as fully resolved (targetResolved=true) with no remainingUncertainty — erasing the ambiguity that "matters more" doesn't define a boundary.
2. **For Answer 1, did Variant B resolve the broader priority target?** Yes. The broadened target ("priority between growth and risk") was resolved as "risk avoidance is prioritized over business growth when they conflict." This is a correct mapping from the weak answer to the priority frame.
3. **Did Answer 1 therefore create materially different resolution states between A and B?** No. Both map to the same downstream state: risk avoidance is not an absolute boundary but a stronger consideration than growth. The resolutions are materially equivalent for downstream reasoning about what can/cannot be done.
4. **For Answer 2, did Variant A correctly identify that risk avoidance is not an absolute hard constraint?** Partially correct on the outcome (not a hard constraint) but failed to preserve conditionality — the "might accept some" conditional was collapsed into a flat preference resolution with no remaining uncertainty.
5. **For Answer 2, did Variant B preserve the conditional nature of the priority?** Yes. Variant B preserved both the default-priority-and-conditional structure and identified remaining uncertainty about when conditions shift.
6. **Were the Answer 2 resolution states materially equivalent or different?** Different. Variant A erased conditionality; Variant B preserved it plus remainingUncertainty. This is a material divergence for downstream reasoning state.
7. **Did either variant force a weak answer into stronger meaning than the user supplied?** Yes — Variant A for Answer 2 collapsed conditional willingness ("might accept some") into a flat preference classification, erasing the conditionality layer.
8. **Did either variant erase uncertainty that should remain?** Yes — Variant A for both answers showed `potential_erasal_of_uncertainty`. For Answer 1, "Risk matters more to me" became a fully resolved preference (no remainingUncertainty). For Answer 2, conditionality was erased.
9. **Does target broadening have a material downstream consequence when answers are less explicit in these tested cases?** Yes — specifically for Answer 2 (conditional trade-off). The precise target question led the model to map to a flat preference-vs-constraint resolution and erase conditionality. The broadened target preserved conditional structure. This means target broadening has a real, asymmetrical consequence: the broadened question can actually preserve nuance that the precise question erases in this case.
10. **Does this establish that precise targets are always required?** No — Answer 1 showed no material divergence, and for Answer 2 the broader target preserved more nuance than the precise one. Neither is universally better.
11. **Does this establish how target-generation logic should be changed?** No — only two answers tested; neither variant was consistently better; no fix designed from these results.
12. **Does this establish UI behaviour?** No — this is a clarification-target test only.
### Limitations
- Only one source scenario and two user answers were tested. Different sources may behave differently.
- Only one ambiguity pattern (preference/constraint) was tested with weak answers. Other patterns not assessed.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
- The asymmetric finding (broadened target preserving more nuance for Answer 2) is surprising and warrants further testing with additional answers that include explicit conditionality.
- Two tested cases; broader generalisation remains untested.
### Experiment Conclusion
**Target broadening changed wording but not material resolution under weaker answers — except when the answer contained explicit conditionality.**
For Answer 1 ("Risk matters more to me."), both variants produced materially equivalent downstream meaning: risk avoidance is stronger than growth but not an absolute constraint. The broader target did not create a material divergence here.
For Answer 2 ("I'd normally avoid more risk, but for the right opportunity I might accept some."), the variants diverged. Variant A (precise) collapsed conditionality into a flat preference resolution and erased uncertainty. Variant B (broadened) preserved conditional structure and remaining uncertainty about what constitutes "the right opportunity."
**Unexpected finding:** The broadened target preserved more nuance than the precise target for the conditional answer. Neither framing was consistently superior across the two tested weaker answers.
### Focused Test Result
All 4 live inference calls completed successfully (all tests passed). Answer 1: materially equivalent resolutions from both variants. Answer 2: materially different resolutions — Variant A erased conditionality; Variant B preserved it. One forced certainty detection (Variant A on Answer 2).
### Historical Comparison Result
Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. Both tested weak answers confirmed the hypothesis: weak answers expose differences between precise and broadened targets, but only under specific content conditions. Neither variant was consistently superior across the two answers tested.
### Documentation Updated
- `docs/design-evolution-log.md` — added full Experiment 54Z entry; applied 54Y wording corrections
- `docs/current-handoff.md` — updated with Experiment 54Z summary and new Return-to-Work note
### Confirmation Host and Model Remained Unchanged
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54Y.
### Confirmation Semantic Instructions and Output Contracts Remained Unchanged
Answer-resolution instruction identical to Experiment 54V. Output contract unchanged from 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`).
### Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
### Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
### Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
### Confirmation No Weak-Answer Consequence Logic Entered Active Runtime
This experiment created one new test file only. No weak-answer consequence logic entered any active runtime path, production module, or behaviour selection output.
---