31 KiB
Experiment 54S — Can the Model Identify Exactly What the User Needs to Clarify? (2026-08-07)
Objective
When clarification genuinely requires user input, can the model identify the specific missing user-owned distinction without yet generating the clarification question? This is passive and test-only. Do not generate a question. Do not choose a winning interpretation. Do not change Behaviour Selection. Do not change production behaviour.
Hypothesis
When clarification genuinely belongs to the user, the model may be able to identify the smallest unresolved user-owned distinction. For example, for "I want the business to grow, but I don't want to take on more risk," the missing distinction is not "what are the risks?" but rather "whether avoiding additional risk is a preference or a hard constraint."
Configuration
Host: http://192.168.1.111:11434 (same as 54R)
Model: qwen-claude:latest (same as 54R)
No localhost fallback. No experiment-specific model variable.
Number of Live Inference Calls
Exactly 3 live Ollama calls — one per case.
Input Contract
Each call receives: { source, disagreement, requiresUserClarification }. The boolean is fixed from human-reviewed Experiment 54R-style references. The model does not re-decide whether clarification is required.
Output Contract
Return exactly: { "clarificationTarget": "short statement" | null }.
- If
requiresUserClarificationis true → smallest specific user-owned distinction; - If false → null.
No question text, no explanation, no recommendation, no preferred interpretation, no score, no confidence, no behaviour label. Test-only, not a production schema.
Semantic Instruction Used
Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
No examples. No expected answers. Same instruction for all three cases.
Case 1 — Growth Versus Risk Priority
Source: "I want the business to grow, but I don't want to take on more risk."
Disagreement: growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
Fixed clarification status: requiresUserClarification: true
Fixed human target: whether avoiding additional risk is a preference/trade-off or a hard constraint.
Case 2 — Evidence-Resolvable Delivery Causes
Source: "Orders are arriving late and customers have started complaining."
Disagreement: delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
Fixed clarification status: requiresUserClarification: false
Fixed human target: null (cause should be investigated through evidence).
Case 3 — Ambiguous Meaning of "Affordable"
Source: "I want to replace the system, but the new option needs to be affordable."
Disagreement: affordable means keeping upfront purchase cost low / affordable means keeping overall long-term cost low even if upfront cost is higher.
Fixed clarification status: requiresUserClarification: true
Fixed human target: what the user means by "affordable" — upfront cost versus overall/long-term cost.
Results
Case 1 — Growth Versus Risk Priority
Model output:
{
"clarificationTarget": "whether avoiding additional risk is a preference/trade-off or a hard constraint"
}
Classification: clarification_target_correct (14 bidirectional concept matches)
The model correctly identified the constraint versus preference distinction — the exact human target. No evidence confusion, no question text, no winner selection.
Case 2 — Evidence-Resolvable Delivery Causes
Model output:
{
"clarificationTarget": null
}
Classification: clarification_target_correct (correctly returned null when clarification is not required)
The model correctly respected the false signal and returned no target for an evidence-resolvable case. Note: in earlier runs of this test, the model occasionally ignored the false signal and produced a target — this variability was observed but did not affect this final run's classification.
Case 3 — Ambiguous Meaning of "Affordable"
Model output:
{
"clarificationTarget": "whether affordability prioritizes low upfront cost or low long-term total cost"
}
Classification: clarification_target_correct (7 bidirectional concept matches)
The model correctly identified the definition ambiguity — upfront cost versus long-term total cost. No vendor comparison, no budget range confusion, no question text.
Clarification-Target-Correct Count
3 of 3 cases classified correctly.
Clarification-Target-Failed Count
0 of 3 cases classified incorrectly.
Required Questions — Answers
- Did Case 1 identify preference/trade-off versus hard constraint? Yes
- Did Case 1 avoid asking about external risk evidence instead? Yes (no evidence keywords present)
- Did Case 2 correctly return null? Yes (in the final run)
- Did Case 3 identify the meaning of "affordable" as upfront versus long-term cost? Yes
- Did the model ever generate a full question? No
- Did it confuse clarification target with evidence needed? No
- Did it choose a winner? No
Inference Timing
- Total time: 55,511ms (55.5s)
- Average: 18,503.7ms per call
- Fastest: 17,046ms (Case 2 — evidence-resolvable)
- Slowest: 20,957ms (Case 1 — growth-vs-risk)
Limitations
- Three cases only; limited domain coverage (one strategic priority, one delivery scenario, one procurement definition).
- Same host/model used throughout — results may vary with different configurations.
- Does not establish generalisation beyond these specific inputs.
- The model occasionally ignored the
requiresUserClarification: falsesignal in earlier test runs (producing a target when null was expected), indicating the boolean gate alone may not be sufficient for robust null enforcement. - No clarification question was generated — this experiment establishes the target identification layer only.
Conclusion
The final three-case run was correct, but earlier repetitions showed instability when clarification was explicitly not required. Clarification-target identification therefore appears promising, but null enforcement is not yet stable.
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour (the model occasionally produced a target despite requiresUserClarification: false in earlier runs), not merely a test warning. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
Status
Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-target.test.js.
Experiment 54T — Is "No Clarification Needed" Stable Across Repeated Identical Inputs? (2026-08-08)
Objective
First, tighten Experiment 54S so its conclusion reflects the instability observed during earlier runs.
Then test one narrow question:
When
requiresUserClarificationis explicitly false, does the model consistently return no clarification target across repeated identical calls?
This experiment exists because 54S produced the correct final result but earlier runs sometimes generated a clarification target even when clarification was explicitly not required.
This is passive and test-only. Do not change the clarification-target instruction yet. Do not generate questions. Do not change production behaviour.
Hypothesis
Two possibilities are plausible.
Stable gating: When requiresUserClarification is false, the model consistently returns null.
Semantic override: The model sometimes ignores the explicit false flag and invents a clarification target because it sees unresolved uncertainty in the source.
Either result is useful. Do not try to correct the behaviour in this experiment.
Configuration
Host: http://192.168.1.111:11434 (same as 54S)
Model: qwen-claude:latest (same as 54S)
No localhost fallback. No experiment-specific model variable.
Number of Live Inference Calls
Exactly 6 live Ollama calls — three per case, repeated identical input each time.
Semantic Instruction Used
Exact Experiment 54S instruction unchanged:
Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
No examples. No expected answers. Same instruction for all six cases.
Output Contract
Unchanged from 54S:
{
"clarificationTarget": "short statement" | null
}
Case A — Evidence-Resolvable / False (the unstable case from 54S)
Source: "Orders are arriving late and customers have started complaining."
Disagreement: delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
Fixed clarification status: requiresUserClarification: false
Expected result: clarificationTarget: null
Run this exact case 3 times without changing wording. This is the unstable Case 2 from 54S.
Case B — User-Owned Ambiguity / True Control
Source: "I want the business to grow, but I don't want to take on more risk."
Disagreement: growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
Fixed clarification status: requiresUserClarification: true
Expected semantic target: whether avoiding additional risk is a preference/trade-off or a hard constraint.
Run this exact case 3 times. Purpose: determine whether instability is specific to suppressing clarification or affects clarification-target generation generally.
Results
Case A — Evidence-Resolvable / False
| Run | Result | Classification |
|---|---|---|
| 1 | null (15,566ms) |
null ✓ |
| 2 | null (13,800ms) |
null ✓ |
| 3 | null (19,087ms) |
null ✓ |
Null count: 3/3 Non-null count: 0/3
No clarification targets were invented. The model consistently returned null across all three repeated identical runs with requiresUserClarification: false.
Case B — User-Owned Ambiguity / True Control
| Run | Target | Classification |
|---|---|---|
| 1 | "The relative priority between business growth and strict risk avoidance when they conflict" (18,595ms) | target_correct |
| 2 | "Preferred priority between accelerating business growth and strictly avoiding additional risk" (19,458ms) | target_correct |
| 3 | "Your maximum acceptable level of additional risk relative to desired business growth." (18,976ms) | target_correct |
Correct-target count: 3/3 Incorrect-target count: 0/3 Null count: 0/3
All three runs produced semantically correct targets aligned with the human reference. No null responses observed when clarification was required.
Timing
- Total time: 105,470ms (105.5s)
- Average: 17,578.3ms per call
- Fastest: 13,798ms (Case A run 2)
- Slowest: 19,457ms (Case B run 3)
Required Questions — Answers
- How many Case A runs returned
null? 3 - How many Case A runs returned a non-null clarification target? 0
- If Case A produced a target, what distinction did it invent? N/A — none produced
- How many Case B runs produced the correct clarification target? 3
- Did Case B ever incorrectly return
null? No - Is false/null behaviour materially stable across the three repeated runs? Yes — all 3 returned null with zero variance
- Is true/target behaviour materially stable across the three repeated runs? Yes — all 3 produced semantically correct targets with zero variance
- Is any observed instability asymmetric: mainly false/null / mainly true/target / both / none observed? None observed in this experiment. Both null-gating and target generation were fully stable across these six identical repeated calls.
- Does this experiment establish why instability occurs? No — this is an observation experiment, not a diagnostic one.
- Does this establish how to enforce null behaviour? No — the current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested here; broader stability remains unproven.
- Does this establish how Behaviour Selection should work? No.
Limitations
- Only two cases tested (one false, one true); limited domain coverage.
- Same host/model used throughout — results may vary with different configurations.
- This is a stability observation experiment; it does not diagnose root causes of earlier variability.
- Does not establish generalisation beyond these specific inputs.
- The model's behaviour in earlier unrecorded runs (when null-gating failed) remains the unknown variable.
Evaluation Conclusion
Clarification null-gating was stable across all tested repeats
Case A returned null in 3 of 3 runs. Case B produced correct targets in 3 of 3 runs. No instability was observed in either direction during this experiment.
Status
Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-null-stability.test.js.
Experiment 54U — Can a Clarification Target Become a Useful User Question Without Adding New Meaning? (2026-08-08)
Objective
Given a fixed clarification target, can the model turn that target into one concise, neutral clarification question without adding assumptions, choosing a side, or broadening the issue?
This is test-only. Do not integrate anything into the UI. Do not change Behaviour Selection. Do not change production prompts.
Hypothesis
Given a specific clarification target, the model may be able to produce a single question that:
- asks only about the unresolved distinction;
- remains neutral between the interpretations;
- does not introduce new assumptions;
- does not ask for evidence instead;
- does not become a multi-part interview.
If it broadens the question or adds new meaning, record that failure.
Configuration
Host: http://192.168.1.111:11434 (same as 54S / 54T)
Model: qwen-claude:latest (same as 54S / 54T)
No localhost fallback. No experiment-specific model variable.
Number of Live Inference Calls
Exactly 3 live Ollama calls — one per case.
Semantic Instruction Used
Write one concise clarification question that asks only about the supplied clarification target. Keep it neutral between the possible meanings. Do not introduce new facts, assumptions, evidence requests, recommendations, or additional questions. Do not explain why you are asking.
No examples. No expected wording. Same instruction for all three cases.
Input Contract
Each call receives:
{ "source": "...", "clarificationTarget": "..." }
The target is fixed human-reviewed input. The model must not decide whether clarification is needed.
Output Contract
Return exactly:
{ "question": "one clarification question" }
No explanation, score, confidence, answer options, recommendation, preferred interpretation, or second question.
Case 1 — Preference Versus Hard Constraint
Source: "I want the business to grow, but I don't want to take on more risk." Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint. Human-reviewed intent: A correct question should ask whether "no more risk" is an absolute boundary or something the user would trade against growth.
Case 2 — Meaning of "Affordable"
Source: "I want to replace the system, but the new option needs to be affordable." Clarification target: whether affordable means low upfront cost or low overall/long-term cost. Human-reviewed intent: A correct question should clarify which meaning of affordability the user intends.
Case 3 — Private Factual Constraint
Source: "I could move the project forward next month, depending on whether I actually have enough time." Clarification target: whether the user has enough available time next month to take on the project. Human-reviewed intent: A correct question should ask the user to clarify their own available capacity/time.
Results
Case 1 — Preference Versus Hard Constraint
Returned question: "Do you view avoiding additional risk as a preference/trade-off or a hard constraint?" Classification: question_correct — semantically aligned with target (6 overlapping words), structurally valid, neutral. Did it stay within the target? Yes — asks exactly whether risk avoidance is a preference/trade-off or a hard constraint. Did it add assumptions? No. Did it ask for evidence? No. Multiple questions? No — one substantive question. Implied preferred interpretation? No.
Case 2 — Meaning of "Affordable"
Returned question: 'Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?' Classification: question_correct — semantically aligned with target (5 overlapping words), structurally valid, neutral. Did it stay within the target? Yes — asks which meaning of affordability the user intends. Did it add assumptions? No. Did it ask for evidence? No. Multiple questions? No. Implied preferred interpretation? No.
Case 3 — Private Factual Constraint
Returned question: "Do you have enough available time next month to take on the project?" Classification: question_correct — semantically aligned with target (7 overlapping words), structurally valid, neutral. Did it stay within the target? Yes — asks for the user's own capacity clarification. Did it add assumptions? No — did not estimate hours or assume availability. Did it ask for evidence? No. Multiple questions? No. Implied preferred interpretation? No.
Timing
- Total time: 71,758ms (71.8s)
- Average: 23,919.3ms per call
- Fastest: 13,573ms (Case 3)
- Slowest: 33,314ms (Case 2)
Required Questions — Answers
- Did Case 1 ask only about preference/trade-off versus hard constraint? Yes
- Did Case 2 ask only what "affordable" means? Yes
- Did Case 3 correctly ask for the user's private factual capacity? Yes
- Did any question introduce assumptions not present in the source/target? No
- Did any question ask for evidence instead of clarification? No
- Did any question contain more than one substantive question? No
- Did any question choose or imply a preferred interpretation? No
- How many cases were question_correct / question_failed? 3 correct, 0 failed.
- Does this establish when the question should be asked? No.
- Does this establish how the answer should update reasoning state? No.
- Does this establish that the question will feel good in the UI? No.
Evaluation Conclusion
The model produced a clean clarification question for every tested target.
All three cases returned one neutral question addressing only the supplied clarification target, with no introduced assumptions, evidence requests, multi-part structure, or implied preferred interpretations.
Corrected conclusion: The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Word-overlap metrics are diagnostic only; manual semantic review is the stronger evidence.
Limitations
- Only three cases tested; limited domain coverage.
- Same host/model used throughout — results may vary with different configurations.
- Does not establish when the question should be asked (that is a separate step).
- Does not establish how answers should update reasoning state.
- Semantic quality assessed through structural and overlap heuristics; manual review would strengthen confidence.
- Does not establish that the question will feel good in the UI.
Status
Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-question.test.js.
Experiment 54V — Can a Clarification Answer Resolve Only the Target Without Rewriting Everything Else? (2026-08-08)
Objective
Given the original source, a fixed clarification target, the clarification question, and the user's answer, can the model identify what has now been resolved without adding new meaning or rewriting unrelated reasoning?
This is test-only. Do not integrate with the graph, Behaviour Selection, or UI.
Hypothesis
A clarification answer should be able to resolve one narrow user-owned ambiguity without causing the model to:
- reinterpret the whole source;
- add unsupported consequences;
- reopen unrelated uncertainty.
If the model cannot preserve that boundary, clarification answers may create as much ambiguity as they remove.
Configuration
Host: http://192.168.1.111:11434 (same as 54S / 54T / 54U)
Model: qwen-claude:latest (same as 54S / 54T / 54U)
No localhost fallback. No experiment-specific model variable.
Number of Live Inference Calls
Exactly 3 live Ollama calls — one per case.
Input Contract
Each call receives:
{ "source": "...", "clarificationTarget": "...", "clarificationQuestion": "...", "userAnswer": "..." }
Output Contract
Return exactly:
{ "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
No next question, recommendation, confidence score, graph update, extra interpretation, or explanation.
Semantic Instruction Used
Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
No examples. No expected answers. Same instruction for all three cases.
Case 1 — Hard Constraint Resolved
Source: "I want the business to grow, but I don't want to take on more risk." Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint. Clarification question: Do you view avoiding additional risk as a preference/trade-off or a hard constraint? User answer: It's a hard constraint. I don't want any increase in risk.
Human reference:
- resolvedMeaning: "avoiding additional risk is a hard constraint"
- targetResolved: true
- remainingUncertainty: null
Must not infer: growth is impossible; which growth option should be chosen; how much risk currently exists.
Case 2 — Definition Resolved
Source: "I want to replace the system, but the new option needs to be affordable." Clarification target: whether affordable means low upfront cost or low overall/long-term cost. Clarification question: Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost? User answer: I care about the total cost over five years, not the upfront price.
Human reference:
- resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost"
- targetResolved: true
- remainingUncertainty: null
Do not invent a budget or preferred system.
Case 3 — Answer Does Not Fully Resolve Target
Source: "I want the business to grow, but I don't want to take on more risk." Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint. Clarification question: Do you view avoiding additional risk as a preference/trade-off or a hard constraint? User answer: It depends on the opportunity.
Human reference:
- The target is not fully resolved.
- targetResolved: false
- remainingUncertainty should preserve that the acceptable trade-off still depends on circumstances.
- resolvedMeaning may state only that the user's risk position is conditional on the opportunity.
Do not force the answer into either preference or hard-constraint category.
Results
Case 1 — Hard Constraint Resolved
Returned result:
- resolvedMeaning: "Avoiding additional risk is established as a hard constraint, meaning no increase in risk is acceptable."
- targetResolved: true
- remainingUncertainty: null
Manual semantic review:
- Used only the user answer to resolve the target? Yes.
- Preserved unresolved ambiguity where present? N/A (target was resolved).
- Avoided wider consequences? Yes — did not state growth is impossible, did not choose a growth option, did not estimate current risk levels. The phrase "meaning no increase in risk is acceptable" is a direct restatement of the user answer ("I don't want any increase in risk"), not an addition from the source.
- Avoided adding unsupported facts? Yes.
- Avoided generating another question? Yes.
Classification: resolution_correct
Case 2 — Definition Resolved
Returned result:
- resolvedMeaning: "Affordable refers to a low overall or long-term cost rather than a low upfront price."
- targetResolved: true
- remainingUncertainty: null
Manual semantic review:
- Used only the user answer to resolve the target? Yes.
- Preserved unresolved ambiguity where present? N/A (target was resolved).
- Avoided wider consequences? Yes — did not invent a budget amount, did not recommend a system.
- Avoided adding unsupported facts? Yes.
- Avoided generating another question? Yes.
Classification: resolution_correct
Case 3 — Answer Does Not Fully Resolve Target
Returned result:
- resolvedMeaning: "Avoiding additional risk is conditional rather than fixed, varying based on the specific opportunity."
- targetResolved: false
- remainingUncertainty: "It remains unclear whether avoiding additional risk is strictly a preference/trade-off or a hard constraint, as the stance depends on the opportunity."
Manual semantic review:
- Used only the user answer to resolve the target? Yes. The model preserved the conditionality present in "It depends on the opportunity" without forcing into either category.
- Preserved unresolved ambiguity where present? Yes. Correctly kept the target unresolved and described the remaining uncertainty specifically about that target.
- Avoided wider consequences? Yes.
- Avoided adding unsupported facts? Yes.
- Avoided generating another question? Yes.
Classification: resolution_correct
Timing
- Total time: 38,052ms (38.1s)
- Average: 12,684.0ms per call
- Fastest: 8,479ms (Case 1)
- Slowest: 15,208ms (Case 3)
Required Questions — Answers
- Did Case 1 resolve the target to a hard constraint without adding wider consequences? Yes. The resolved meaning stays within the user answer's scope. No inference about growth feasibility, option selection, or current risk levels.
- Did Case 2 resolve "affordable" to long-term cost without inventing a budget? Yes. The model correctly captured the five-year perspective without adding any budget figure or system recommendation.
- Did Case 3 correctly keep the target unresolved? Yes. The model returned targetResolved=false, preserved conditionality in resolvedMeaning, and provided meaningful remainingUncertainty.
- Did any case force an ambiguous answer into a stronger meaning? No. Case 3's conditional answer was kept at its actual strength — neither strengthened to preference nor to hard constraint.
- Did any case rewrite unrelated parts of the source? No. In Cases 1 and 3 (same source), the model treated the "grow" portion identically to the original source meaning without reinterpreting it.
- Did any case generate another question? No. All resolvedMeaning fields are statements, not questions.
- How many cases were resolution_correct / resolution_failed? 3 correct, 0 failed.
- Does this establish how graph state should be updated? No. This only tests semantic recognition of what a clarification answer resolves; it does not test any graph update mechanism.
- Does this establish what question should come next? No. The experiment tested one directional step (answer → resolved meaning) and did not test the next question generation cycle.
- Does this establish how Behaviour Selection should react? No. No behaviour selection logic was tested or involved.
Evaluation Conclusion
Clarification answers resolved only the intended target across all tested cases.
All three cases returned correct resolution boundaries: Cases 1 and 2 settled the target cleanly; Case 3 preserved incomplete information at its actual strength without forcing it into a stronger category. The model did not widen beyond the clarification target, invent consequences, or generate new questions in any case.
The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.
Limitations
- Only three cases tested; limited domain coverage (risk constraint, affordability definition, conditional constraint).
- Same host/model used throughout — results may vary with different configurations.
- Does not establish how graph state should update from resolved meanings.
- Does not establish what question should come next after resolution.
- Does not establish how Behaviour Selection should react to resolved vs unresolved targets.
- Semantic quality assessed through structural checks and manual review; broader generalisation untested.
- Case 3's remainingUncertainty output is longer than the human reference — acceptable because it describes the uncertainty rather than adding meaning, but worth noting.
Status
Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-answer-resolution.test.js.