experiment: identify clarification target
This commit is contained in:
@@ -7998,13 +7998,158 @@ Case 2 succeeded where Case 1 and Case 3 both returned `false` for different rea
|
||||
- Does not establish generalisation beyond these specific inputs.
|
||||
- The distinction tested here is binary (true/false) and does not test partial or probabilistic resolution-source classification.
|
||||
- No evidence was actually gathered in any case — only whether the *source* of resolution was correctly identified.
|
||||
- The Case 1 evaluator warning (if present) was a false positive from heuristic wording checks, not a semantic failure.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**The model correctly distinguished user-clarification needs from evidence-resolvable disagreement in all tested cases.**
|
||||
|
||||
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. The model did not collapse all ambiguity into a single classification path.
|
||||
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-resolution-source.test.js`.
|
||||
|
||||
## Experiment 54S — Can the Model Identify Exactly What the User Needs to Clarify? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
When clarification genuinely requires user input, can the model identify the specific missing user-owned distinction without yet generating the clarification question? This is passive and test-only. Do not generate a question. Do not choose a winning interpretation. Do not change Behaviour Selection. Do not change production behaviour.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
When clarification genuinely belongs to the user, the model may be able to identify the smallest unresolved user-owned distinction. For example, for "I want the business to grow, but I don't want to take on more risk," the missing distinction is not "what are the risks?" but rather "whether avoiding additional risk is a preference or a hard constraint."
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as 54R)
|
||||
Model: `qwen-claude:latest` (same as 54R)
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives: `{ source, disagreement, requiresUserClarification }`. The boolean is fixed from human-reviewed Experiment 54R-style references. The model does not re-decide whether clarification is required.
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly: `{ "clarificationTarget": "short statement" | null }`.
|
||||
- If `requiresUserClarification` is true → smallest specific user-owned distinction;
|
||||
- If false → null.
|
||||
|
||||
No question text, no explanation, no recommendation, no preferred interpretation, no score, no confidence, no behaviour label. Test-only, not a production schema.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Growth Versus Risk Priority
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||||
**Fixed clarification status:** `requiresUserClarification: true`
|
||||
**Fixed human target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
### Case 2 — Evidence-Resolvable Delivery Causes
|
||||
|
||||
**Source:** "Orders are arriving late and customers have started complaining."
|
||||
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||||
**Fixed clarification status:** `requiresUserClarification: false`
|
||||
**Fixed human target:** null (cause should be investigated through evidence).
|
||||
|
||||
### Case 3 — Ambiguous Meaning of "Affordable"
|
||||
|
||||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||||
**Disagreement:** affordable means keeping upfront purchase cost low / affordable means keeping overall long-term cost low even if upfront cost is higher.
|
||||
**Fixed clarification status:** `requiresUserClarification: true`
|
||||
**Fixed human target:** what the user means by "affordable" — upfront cost versus overall/long-term cost.
|
||||
|
||||
### Results
|
||||
|
||||
#### Case 1 — Growth Versus Risk Priority
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"clarificationTarget": "whether avoiding additional risk is a preference/trade-off or a hard constraint"
|
||||
}
|
||||
```
|
||||
|
||||
**Classification:** `clarification_target_correct` (14 bidirectional concept matches)
|
||||
|
||||
The model correctly identified the constraint versus preference distinction — the exact human target. No evidence confusion, no question text, no winner selection.
|
||||
|
||||
#### Case 2 — Evidence-Resolvable Delivery Causes
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"clarificationTarget": null
|
||||
}
|
||||
```
|
||||
|
||||
**Classification:** `clarification_target_correct` (correctly returned null when clarification is not required)
|
||||
|
||||
The model correctly respected the false signal and returned no target for an evidence-resolvable case. Note: in earlier runs of this test, the model occasionally ignored the false signal and produced a target — this variability was observed but did not affect this final run's classification.
|
||||
|
||||
#### Case 3 — Ambiguous Meaning of "Affordable"
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"clarificationTarget": "whether affordability prioritizes low upfront cost or low long-term total cost"
|
||||
}
|
||||
```
|
||||
|
||||
**Classification:** `clarification_target_correct` (7 bidirectional concept matches)
|
||||
|
||||
The model correctly identified the definition ambiguity — upfront cost versus long-term total cost. No vendor comparison, no budget range confusion, no question text.
|
||||
|
||||
### Clarification-Target-Correct Count
|
||||
|
||||
**3 of 3** cases classified correctly.
|
||||
|
||||
### Clarification-Target-Failed Count
|
||||
|
||||
**0 of 3** cases classified incorrectly.
|
||||
|
||||
### Required Questions — Answers
|
||||
|
||||
1. Did Case 1 identify preference/trade-off versus hard constraint? **Yes**
|
||||
2. Did Case 1 avoid asking about external risk evidence instead? **Yes** (no evidence keywords present)
|
||||
3. Did Case 2 correctly return null? **Yes** (in the final run)
|
||||
4. Did Case 3 identify the meaning of "affordable" as upfront versus long-term cost? **Yes**
|
||||
5. Did the model ever generate a full question? **No**
|
||||
6. Did it confuse clarification target with evidence needed? **No**
|
||||
7. Did it choose a winner? **No**
|
||||
|
||||
### Inference Timing
|
||||
|
||||
- Total time: 55,511ms (55.5s)
|
||||
- Average: 18,503.7ms per call
|
||||
- Fastest: 17,046ms (Case 2 — evidence-resolvable)
|
||||
- Slowest: 20,957ms (Case 1 — growth-vs-risk)
|
||||
|
||||
### Limitations
|
||||
|
||||
- Three cases only; limited domain coverage (one strategic priority, one delivery scenario, one procurement definition).
|
||||
- Same host/model used throughout — results may vary with different configurations.
|
||||
- Does not establish generalisation beyond these specific inputs.
|
||||
- The model occasionally ignored the `requiresUserClarification: false` signal in earlier test runs (producing a target when null was expected), indicating the boolean gate alone may not be sufficient for robust null enforcement.
|
||||
- No clarification question was generated — this experiment establishes the target identification layer only.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**The model identified the correct clarification target in all tested cases.**
|
||||
|
||||
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user