experiment: probe user-owned ambiguity boundary
This commit is contained in:
+13
-1
@@ -170,7 +170,7 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
|
||||
|
||||
### Return-to-Work Note (Experiment 55F)
|
||||
|
||||
@@ -223,3 +223,15 @@ One live Ollama call (18,580 ms) returned `uncertaintyType: "evidence_needed"` w
|
||||
**Classification: PASS.** The model correctly identified the disagreement as requiring evidence rather than asking the user to settle an externally knowable question by clarification. It specified concrete, relevant evidence — demonstrating understanding of the causal structure rather than producing a generic classification. This confirms the model can preserve the distinction between "evidence needed to determine what is true" and "clarification needed because only the user can establish meaning/preference/intent/constraint" for this tested case.
|
||||
|
||||
This experiment does not prove fidelity for Regression F (user-owned ambiguity), consistency across domains/phrasings, downstream reasoning preservation, or end-to-end production flow. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-e-evidence-vs-clarification.test.js` and `docs/experiment-56k.md`. Status pending Rob's review.
|
||||
|
||||
### Experiment 56L Summary — User-owned ambiguity requires clarification, not evidence
|
||||
|
||||
Tested whether the configured live Ollama model (`qwen-claude:latest` at `http://192.168.1.111:11434`) recognises that a preference-vs-constraint distinction belongs to the user's own meaning and requires clarification rather than external evidence — Regression F from `docs/reasoning-refinement-requirements.md`.
|
||||
|
||||
Fixed case: "I want the business to grow, but I don't want to take on more risk." — user has not specified whether avoiding additional risk is a hard constraint or a strong preference/trade-off.
|
||||
|
||||
One live Ollama call (14,032 ms) returned `uncertaintyType: "user_clarification_needed"` with `evidenceNeeded: null` and specific `userClarificationNeeded` describing the non-negotiable-versus-trade-off distinction only the user can establish. Matches pre-written human reference exactly at category level.
|
||||
|
||||
**Classification: PASS.** The model correctly identified the ambiguity as user-owned, did not introduce spurious evidence gathering, and preserved the evidence-vs-user-meaning distinction cleanly.
|
||||
|
||||
This experiment does not prove consistency across repeated runs, fidelity for other regression cases (A–E, G+), behavior in production reasoning paths, or downstream integration with Behaviour Selection or the SituationGraph. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-f-user-owned-ambiguity.test.js` and `docs/experiment-56l.md`. Status pending Rob's review.
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# Experiment 56L — User-owned ambiguity boundary probe
|
||||
|
||||
## Objective
|
||||
|
||||
Can the configured model recognise that an unresolved distinction belongs to the user's own meaning and therefore requires clarification rather than external evidence gathering?
|
||||
|
||||
## Config
|
||||
|
||||
- **Ollama base URL:** `http://192.168.1.111:11434`
|
||||
- **Model:** `qwen-claude:latest`
|
||||
- **Branch:** `feature/reasoning-fidelity-v0.8`
|
||||
- **Start HEAD:** [current HEAD at time of experiment]
|
||||
|
||||
## Fixed Case (Regression F)
|
||||
|
||||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
- **Answer context:** (ambiguous statement about both growth and risk — user has not specified preference/trade-off versus hard constraint)
|
||||
- **Expected preserved meaning:** User has not specified whether avoiding additional risk is a hard constraint or a strong preference/trade-off.
|
||||
- **Expected uncertainty:** Preference vs constraint distinction is user-owned and requires clarification.
|
||||
- **Must not happen:** Engine-generated classification of the ambiguity as "not requiring clarification" or resolution through evidence gathering alone.
|
||||
|
||||
## Pre-written Human Reference (written before model output)
|
||||
|
||||
```json
|
||||
{
|
||||
"uncertaintyType": "user_clarification_needed",
|
||||
"reason": "The unresolved distinction (whether avoiding additional risk is a hard constraint or a strong preference/trade-off) belongs to the user's own meaning. External evidence cannot establish what the user means, prefers, intends, defines, or constrains on their own behalf.",
|
||||
"evidenceNeeded": null,
|
||||
"userClarificationNeeded": "Whether the user treats avoiding additional risk as a hard constraint (non-negotiable boundary) versus a strong preference/trade-off (weighted but potentially overrideable)."
|
||||
}
|
||||
```
|
||||
|
||||
## Live-call Result
|
||||
|
||||
- **Call duration:** 14,032 ms
|
||||
- **Call count:** 1
|
||||
|
||||
### Raw Structured Response
|
||||
|
||||
```json
|
||||
{
|
||||
"uncertaintyType": "user_clarification_needed",
|
||||
"reason": "The ambiguity concerns whether avoiding additional risk is a strict hard constraint or a flexible preference, which reflects the user's private intent and priorities that external data cannot determine.",
|
||||
"evidenceNeeded": null,
|
||||
"userClarificationNeeded": "Whether avoiding additional risk is a non-negotiable hard constraint or a trade-off preference that can be balanced against business growth goals."
|
||||
}
|
||||
```
|
||||
|
||||
## Human Semantic Classification: PASS
|
||||
|
||||
**Rationale:** The model clearly recognises the ambiguity belongs to the user's own meaning and requires user clarification rather than external evidence. It preserved the distinction cleanly: `uncertaintyType` is `user_clarification_needed`, `evidenceNeeded` is null (no spurious evidence target introduced), and `userClarificationNeeded` specifically describes the preference/trade-off versus hard-constraint distinction that only the user can establish.
|
||||
|
||||
## Comparison with Pre-written Human Reference
|
||||
|
||||
- **Expected:** `user_clarification_needed`
|
||||
- **Actual:** `user_clarification_needed`
|
||||
- **Matches:** YES
|
||||
|
||||
The model's answer matches the human reference at the category level and substantively agrees on both what is unclear and why (the distinction is private to user meaning, not externally determinable).
|
||||
|
||||
## What This Experiment Established
|
||||
|
||||
1. The configured model (`qwen-claude:latest`) can distinguish user-owned ambiguity from evidence-resolvable uncertainty for Regression F's canonical case.
|
||||
2. It correctly identified that the preference-vs-constraint distinction is user-owned and requires clarification, not evidence gathering.
|
||||
3. It did not introduce unnecessary evidence targets where none apply.
|
||||
|
||||
## What This Experiment Does NOT Prove
|
||||
|
||||
1. Consistency across repeated runs with this or other models.
|
||||
2. Fidelity for other regression cases (A–E, G+).
|
||||
3. Behavior in production reasoning paths or graph-update contexts.
|
||||
4. Downstream integration with Behaviour Selection, UI, or the SituationGraph.
|
||||
5. Whether clarification targeting is precise enough to generate a useful user-facing question (that was explicitly excluded from this experiment's scope per output contract).
|
||||
|
||||
## Files
|
||||
|
||||
- Test: `tests/reconstruction/semantic-regression-f-user-owned-ambiguity.test.js`
|
||||
- Document: `docs/experiment-56l.md`
|
||||
Reference in New Issue
Block a user