experiment: test clarification answer resolution
This commit is contained in:
@@ -8432,6 +8432,8 @@ No explanation, score, confidence, answer options, recommendation, preferred int
|
||||
|
||||
All three cases returned one neutral question addressing only the supplied clarification target, with no introduced assumptions, evidence requests, multi-part structure, or implied preferred interpretations.
|
||||
|
||||
**Corrected conclusion:** The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Word-overlap metrics are diagnostic only; manual semantic review is the stronger evidence.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three cases tested; limited domain coverage.
|
||||
@@ -8444,3 +8446,187 @@ All three cases returned one neutral question addressing only the supplied clari
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
|
||||
|
||||
## Experiment 54V — Can a Clarification Answer Resolve Only the Target Without Rewriting Everything Else? (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Given the original source, a fixed clarification target, the clarification question, and the user's answer, can the model identify what has now been resolved without adding new meaning or rewriting unrelated reasoning?
|
||||
|
||||
This is test-only.
|
||||
Do not integrate with the graph, Behaviour Selection, or UI.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A clarification answer should be able to resolve one narrow user-owned ambiguity without causing the model to:
|
||||
- reinterpret the whole source;
|
||||
- add unsupported consequences;
|
||||
- reopen unrelated uncertainty.
|
||||
|
||||
If the model cannot preserve that boundary, clarification answers may create as much ambiguity as they remove.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as 54S / 54T / 54U)
|
||||
Model: `qwen-claude:latest` (same as 54S / 54T / 54U)
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives:
|
||||
```json
|
||||
{ "source": "...", "clarificationTarget": "...", "clarificationQuestion": "...", "userAnswer": "..." }
|
||||
```
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly:
|
||||
```json
|
||||
{ "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
|
||||
```
|
||||
|
||||
No next question, recommendation, confidence score, graph update, extra interpretation, or explanation.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Hard Constraint Resolved
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||||
**User answer:** It's a hard constraint. I don't want any increase in risk.
|
||||
|
||||
**Human reference:**
|
||||
- resolvedMeaning: "avoiding additional risk is a hard constraint"
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
Must not infer: growth is impossible; which growth option should be chosen; how much risk currently exists.
|
||||
|
||||
### Case 2 — Definition Resolved
|
||||
|
||||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||||
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
|
||||
**Clarification question:** Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?
|
||||
**User answer:** I care about the total cost over five years, not the upfront price.
|
||||
|
||||
**Human reference:**
|
||||
- resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost"
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
Do not invent a budget or preferred system.
|
||||
|
||||
### Case 3 — Answer Does Not Fully Resolve Target
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||||
**User answer:** It depends on the opportunity.
|
||||
|
||||
**Human reference:**
|
||||
- The target is **not fully resolved**.
|
||||
- targetResolved: false
|
||||
- remainingUncertainty should preserve that the acceptable trade-off still depends on circumstances.
|
||||
- resolvedMeaning may state only that the user's risk position is conditional on the opportunity.
|
||||
|
||||
Do not force the answer into either preference or hard-constraint category.
|
||||
|
||||
### Results
|
||||
|
||||
#### Case 1 — Hard Constraint Resolved
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is established as a hard constraint, meaning no increase in risk is acceptable."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.**
|
||||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||||
- Avoided wider consequences? **Yes** — did not state growth is impossible, did not choose a growth option, did not estimate current risk levels. The phrase "meaning no increase in risk is acceptable" is a direct restatement of the user answer ("I don't want any increase in risk"), not an addition from the source.
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
#### Case 2 — Definition Resolved
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Affordable refers to a low overall or long-term cost rather than a low upfront price."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.**
|
||||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||||
- Avoided wider consequences? **Yes** — did not invent a budget amount, did not recommend a system.
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
#### Case 3 — Answer Does Not Fully Resolve Target
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is conditional rather than fixed, varying based on the specific opportunity."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "It remains unclear whether avoiding additional risk is strictly a preference/trade-off or a hard constraint, as the stance depends on the opportunity."
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.** The model preserved the conditionality present in "It depends on the opportunity" without forcing into either category.
|
||||
- Preserved unresolved ambiguity where present? **Yes.** Correctly kept the target unresolved and described the remaining uncertainty specifically about that target.
|
||||
- Avoided wider consequences? **Yes.**
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
### Timing
|
||||
|
||||
- Total time: 38,052ms (38.1s)
|
||||
- Average: 12,684.0ms per call
|
||||
- Fastest: 8,479ms (Case 1)
|
||||
- Slowest: 15,208ms (Case 3)
|
||||
|
||||
### Required Questions — Answers
|
||||
|
||||
1. Did Case 1 resolve the target to a hard constraint without adding wider consequences? **Yes.** The resolved meaning stays within the user answer's scope. No inference about growth feasibility, option selection, or current risk levels.
|
||||
2. Did Case 2 resolve "affordable" to long-term cost without inventing a budget? **Yes.** The model correctly captured the five-year perspective without adding any budget figure or system recommendation.
|
||||
3. Did Case 3 correctly keep the target unresolved? **Yes.** The model returned targetResolved=false, preserved conditionality in resolvedMeaning, and provided meaningful remainingUncertainty.
|
||||
4. Did any case force an ambiguous answer into a stronger meaning? **No.** Case 3's conditional answer was kept at its actual strength — neither strengthened to preference nor to hard constraint.
|
||||
5. Did any case rewrite unrelated parts of the source? **No.** In Cases 1 and 3 (same source), the model treated the "grow" portion identically to the original source meaning without reinterpreting it.
|
||||
6. Did any case generate another question? **No.** All resolvedMeaning fields are statements, not questions.
|
||||
7. How many cases were resolution_correct / resolution_failed? **3 correct, 0 failed.**
|
||||
8. Does this establish how graph state should be updated? **No.** This only tests semantic recognition of what a clarification answer resolves; it does not test any graph update mechanism.
|
||||
9. Does this establish what question should come next? **No.** The experiment tested one directional step (answer → resolved meaning) and did not test the next question generation cycle.
|
||||
10. Does this establish how Behaviour Selection should react? **No.** No behaviour selection logic was tested or involved.
|
||||
|
||||
### Evaluation Conclusion
|
||||
|
||||
**Clarification answers resolved only the intended target across all tested cases.**
|
||||
|
||||
All three cases returned correct resolution boundaries: Cases 1 and 2 settled the target cleanly; Case 3 preserved incomplete information at its actual strength without forcing it into a stronger category. The model did not widen beyond the clarification target, invent consequences, or generate new questions in any case.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three cases tested; limited domain coverage (risk constraint, affordability definition, conditional constraint).
|
||||
- Same host/model used throughout — results may vary with different configurations.
|
||||
- Does not establish how graph state should update from resolved meanings.
|
||||
- Does not establish what question should come next after resolution.
|
||||
- Does not establish how Behaviour Selection should react to resolved vs unresolved targets.
|
||||
- Semantic quality assessed through structural checks and manual review; broader generalisation untested.
|
||||
- Case 3's remainingUncertainty output is longer than the human reference — acceptable because it describes the uncertainty rather than adding meaning, but worth noting.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user