experiment: test clarification uncertainty preservation
This commit is contained in:
@@ -152,8 +152,12 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54Z)
|
||||
|
||||
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening has material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
|
||||
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. The two tested weaker answers behaved differently: one converged across target variants and one diverged. Neither framing was consistently superior across the two tested weaker answers. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
### Experiment 55A Summary — Clarification Uncertainty Preservation
|
||||
|
||||
Isolated the answer-resolution step using one fixed target (preference/trade-off or hard constraint) and four answers of different strength: fully explicit, weak priority, conditional trade-off, non-answer. Four live Ollama calls completed at http://192.168.1.111:11434 with qwen-claude:latest (~62s total). Case 1 (explicit hard constraint) resolved correctly. Case 2 (weak priority — "Risk matters more to me.") over-resolved: the model set targetResolved=true and inferred "not a rigid, non-negotiable constraint" — meaning stronger than the user supplied. Case 3 (conditional trade-off) resolved correctly on the target but flattened conditionality into flat "preference or trade-off" language without preserving the conditional qualification ("might accept some"). Case 4 (non-answer) correctly remained unresolved with appropriate remaining uncertainty. One over-resolution across four cases (Case 2). The answer-resolution step appears biased toward resolution for weak priority statements when using qwen-claude:latest with the current semantic instruction. Broader generalisation across other models and answers remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
@@ -9176,7 +9176,7 @@ Variant B interprets the same answer as meaning **risk avoidance has higher prio
|
||||
|
||||
Both variants map the weak answer to a preference/priority-over-constraint interpretation. Neither resolves to "hard constraint." The resolved meanings use different framing but preserve the same downstream reasoning state: risk is not an absolute boundary, it is a prioritized consideration. For downstream use (what can/cannot be done), both produce equivalent uncertainty about whether risk could ever be accepted.
|
||||
|
||||
Both variants were flagged as `potential_erasal_of_uncertainty` because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases.
|
||||
Both variants were flagged as `potential_erasal_of_uncertainty` because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases. **The precise target may have erased uncertainty by inferring that "Risk matters more to me" means risk is not a hard constraint; that conclusion was not explicitly supplied by the user.**
|
||||
|
||||
---
|
||||
|
||||
@@ -9252,9 +9252,7 @@ For Answer 1 ("Risk matters more to me."), both variants produced materially equ
|
||||
|
||||
For Answer 2 ("I'd normally avoid more risk, but for the right opportunity I might accept some."), the variants diverged. Variant A (precise) collapsed conditionality into a flat preference resolution and erased uncertainty. Variant B (broadened) preserved conditional structure and remaining uncertainty about what constitutes "the right opportunity."
|
||||
|
||||
**Unexpected finding:** The broadened target preserved more nuance than the precise target for the conditional answer. This contradicts the intuition that precision always preserves information — sometimes broader framing lets the model retain conditional language it would otherwise collapse under a narrower frame.
|
||||
|
||||
The key finding: target broadening's consequence depends on the answer's content. For pure priority statements, broadening is inconsequential. For conditional trade-off statements, broadening can preserve (or erase) meaning differently than precise targeting — sometimes better, potentially worse in other cases.
|
||||
**Unexpected finding:** The broadened target preserved more nuance than the precise target for the conditional answer. Neither framing was consistently superior across the two tested weaker answers.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
@@ -9262,7 +9260,7 @@ All 4 live inference calls completed successfully (all tests passed). Answer 1:
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. The explicitness of the user's answer matters more than the question framing for determining downstream meaning equivalence. This confirms the hypothesis that weak answers expose differences between precise and broadened targets — but the direction of difference is not predictable in advance (sometimes broader is better, sometimes it may be worse).
|
||||
Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. Both tested weak answers confirmed the hypothesis: weak answers expose differences between precise and broadened targets, but only under specific content conditions. Neither variant was consistently superior across the two answers tested.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
@@ -9295,6 +9293,234 @@ This experiment created one new test file only. No weak-answer consequence logic
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 54Z)
|
||||
## Experiment 55A — Can the Model Preserve Uncertainty From Weak Clarification Answers? (2026-08-08)
|
||||
|
||||
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer, the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening does have material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
|
||||
### Objective
|
||||
|
||||
First, tighten Experiment 54Z so its conclusion does not over-generalise from two weaker answers.
|
||||
|
||||
Then isolate the strongest unresolved issue from 54Z:
|
||||
|
||||
> When a clarification answer is weak or conditional, can the model preserve uncertainty instead of forcing the answer into a stronger resolved meaning?
|
||||
|
||||
This experiment is about answer interpretation only. Do not compare precise versus broadened targets. Do not test question wording. Do not integrate with Behaviour Selection, graph, or UI.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may be too eager to convert weak clarification answers into resolved meaning. A better behaviour would preserve uncertainty when the answer does not actually settle the supplied target. If the model consistently keeps weak answers unresolved, the 54Z over-resolution may have been incidental. If it repeatedly forces resolution, that becomes a clearer limitation of the answer-resolution step itself.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **4** live Ollama calls — one answer-resolution per case. Exactly 4 cases of varying strength against the same source/target/question.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md`
|
||||
- Experiment 54Z in `docs/design-evolution-log.md` (as basis for isolating the unresolved issue)
|
||||
- `tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js` (structural reference)
|
||||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract)
|
||||
|
||||
### Semantic Instruction Unchanged
|
||||
|
||||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
No examples added. No stronger coaching attempted. Same instruction as Experiment 54V.
|
||||
|
||||
### Output Contract Unchanged
|
||||
|
||||
```json
|
||||
{
|
||||
"resolvedMeaning": "short statement",
|
||||
"targetResolved": true,
|
||||
"remainingUncertainty": "short statement or null"
|
||||
}
|
||||
```
|
||||
|
||||
### Fixed Source
|
||||
|
||||
> I want the business to grow, but I don't want to take on more risk.
|
||||
|
||||
### Clarification Target
|
||||
|
||||
> whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
### Clarification Question
|
||||
|
||||
> Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||||
|
||||
---
|
||||
|
||||
### Case 1 — Fully Explicit Hard Constraint
|
||||
|
||||
**User answer:** "It's a hard constraint. I don't want any increase in risk."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is a hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: resolution_correct**
|
||||
|
||||
The model correctly resolved the explicit hard constraint with appropriate strength and no spurious uncertainty. This is the positive control — it behaved as expected.
|
||||
|
||||
---
|
||||
|
||||
### Case 2 — Weak Priority Statement
|
||||
|
||||
**User answer:** "Risk matters more to me."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is prioritized as a strong preference or guiding principle rather than a rigid, non-negotiable constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: uncertainty_overresolved**
|
||||
|
||||
The model set `targetResolved=true` and produced a meaning stronger than the user supplied. "Risk matters more to me" establishes relative importance but does not determine whether avoiding additional risk is a hard constraint or a strong preference/trade-off. The model inferred "not a rigid, non-negotiable constraint" from a statement that only said risk was more important — which could be consistent with a strong preference that still functions as a de facto hard boundary in practice. This over-resolution matches the concern flagged in Experiment 54Z but isolated it here under a single fixed target to confirm the behaviour is real.
|
||||
|
||||
The model produced stronger meaning than the user supplied by declaring what risk avoidance is *not* (a rigid constraint) when the user only stated what it *is* relative to (more important than growth). This establishes that uncertainty should have been preserved with remainingUncertainty about whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
---
|
||||
|
||||
### Case 3 — Conditional Trade-Off
|
||||
|
||||
**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: resolution_correct (with conditionality loss)**
|
||||
|
||||
The model correctly identified that the answer settles the target (risk avoidance is not a hard constraint). However, the resolved meaning flattened the conditional qualification — "I might accept some for the right opportunity" became "preference or trade-off rather than a hard constraint." The conditional nature ("might accept some") was lost in favour of a flat classification. This is not a target resolution failure (the target itself was settled), but it demonstrates that conditionality can be erased even when the model correctly resolves to `targetResolved=true`.
|
||||
|
||||
---
|
||||
|
||||
### Case 4 — Non-Answer / Insufficient Clarification
|
||||
|
||||
**User answer:** "I'm not really sure."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "The user has not yet determined if avoiding additional risk is a hard constraint or a preference/trade-off."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "It remains unclear whether the user considers avoiding additional risk a hard constraint or a preference/trade-off."
|
||||
|
||||
**Classification: uncertainty_preserved**
|
||||
|
||||
The model correctly remained unresolved without inventing any position. The resolvedMeaning accurately describes what was supplied (nothing) and remainingUncertainty states that the distinction between hard constraint and preference/trade-off remains unresolved. This is honest behaviour.
|
||||
|
||||
---
|
||||
|
||||
### Evaluation Summary
|
||||
|
||||
| Case | Answer | Resolution Status | Classification |
|
||||
|---|---|---|---|
|
||||
| 1 | Explicit hard constraint | Correctly resolved | resolution_correct |
|
||||
| 2 | Weak priority ("Risk matters more to me.") | Over-resolved to targetResolved=true | uncertainty_overresolved |
|
||||
| 3 | Conditional trade-off | Resolved, conditionality lost | resolution_correct (with note) |
|
||||
| 4 | Non-answer ("I'm not really sure.") | Correctly unresolved | uncertainty_preserved |
|
||||
|
||||
**Classification counts:**
|
||||
- resolution_correct: 2
|
||||
- uncertainty_preserved: 1
|
||||
- uncertainty_overresolved: 1
|
||||
- resolution_failed: 0
|
||||
|
||||
---
|
||||
|
||||
### Questions Answered
|
||||
|
||||
1. **Did Case 1 correctly resolve the explicit hard constraint?** Yes — targetResolved=true, appropriate meaning strength, no spurious uncertainty.
|
||||
|
||||
2. **Did Case 2 preserve uncertainty rather than infer that risk is not a hard constraint?** No — the model over-resolved, setting targetResolved=true and inferring "not a rigid, non-negotiable constraint" from a weak priority statement. This is the key finding: one tested weak answer received stronger meaning than the user supplied.
|
||||
|
||||
3. **Did Case 3 preserve the conditional nature of the trade-off?** Partially — the model correctly resolved the target (risk avoidance is not a hard constraint) but flattened the conditional qualification ("might accept some for the right opportunity") into flat "preference or trade-off" language. The conditional layer was lost even though target resolution was correct.
|
||||
|
||||
4. **Did Case 4 correctly remain unresolved?** Yes — no invented position, appropriate remainingUncertainty stating the distinction remains unresolved.
|
||||
|
||||
5. **How many cases were:**
|
||||
- resolution_correct: 2
|
||||
- uncertainty_preserved: 1
|
||||
- uncertainty_overresolved: 1
|
||||
- resolution_failed: 0
|
||||
|
||||
6. **Did any answer get stronger meaning than the user supplied?** Yes — Case 2 ("Risk matters more to me.") received a stronger interpretation than it justified. The model inferred "not a rigid, non-negotiable constraint" from a statement that only established relative priority.
|
||||
|
||||
7. **Did any unresolved answer incorrectly return targetResolved true?** No — Case 4 (the only truly unresolved case) correctly returned false. Case 2 over-resolved but did not remain unresolved.
|
||||
|
||||
8. **Did any resolved answer incorrectly retain uncertainty?** No — both resolved cases (1 and 3) correctly returned null for remainingUncertainty.
|
||||
|
||||
9. **Does the answer-resolution step appear biased toward resolution in these tested cases?** Yes — Case 2 demonstrates that a weak priority statement can be over-resolved to a definitive classification ("not a constraint") when it should have remained unresolved. One out of four cases showed this bias, but it appeared on the weakest-answer type where uncertainty preservation matters most.
|
||||
|
||||
10. **Does this establish how answer-resolution logic should be changed?** No — one weak-priority case over-resolved; this does not justify a broad change without broader testing.
|
||||
11. **Does this establish how the graph should represent unresolved clarification?** No — the unresolved representation question is separate from whether the model *should* remain unresolved.
|
||||
12. **Does this establish how Behaviour Selection should react?** No — this experiment did not integrate with Behaviour Selection.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only one source scenario and four answer types were tested. Different sources may behave differently.
|
||||
- Only one weak-answer pattern (priority without constraint meaning) over-resolved; other patterns not assessed.
|
||||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
|
||||
- Case 3 showed conditionality loss that is subtler than over-resolution — it resolved correctly but flattened nuance. This pattern warrants further testing with additional conditional answers.
|
||||
- Four cases tested; broader generalisation remains untested.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Clarification-answer resolution preserved uncertainty appropriately across some cases but over-resolved one weak answer.**
|
||||
|
||||
Case 1 (explicit hard constraint) and Case 4 (non-answer) behaved honestly — the explicit case resolved, the empty case remained unresolved. This is the expected baseline.
|
||||
|
||||
Case 2 (weak priority: "Risk matters more to me.") demonstrates the core limitation: the model over-resolved a weak answer, converting relative priority into a definitive negative ("not a rigid, non-negotiable constraint") when only relative importance was supplied. The model set targetResolved=true and erased uncertainty that should have been preserved.
|
||||
|
||||
Case 3 (conditional trade-off) resolved the target correctly but lost the conditional qualification in the process — "might accept some for the right opportunity" was flattened to "preference or trade-off rather than a hard constraint." This is a subtler form of meaning inflation: correct resolution with erasure of nuance.
|
||||
|
||||
The answer-resolution step appears biased toward resolution when the answer contains enough language to suggest a direction (priority, conditionality), even when that language does not definitively settle the supplied target. With qwen-claude:latest and the current semantic instruction, uncertainty preservation works for truly empty answers but is unreliable for weak-priority answers.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
All 4 live inference calls completed successfully. One over-resolution (Case 2), one conditionality loss (Case 3), two honest classifications (Cases 1 and 4). Total: ~62s, average: ~15.5s per call, fastest: 8.2s, slowest: 19.3s.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 54Z's finding that both variants over-resolved Case 1 ("Risk matters more to me."), Experiment 55A confirms this is a persistent issue under the same semantic instruction and model — not a side-effect of target framing. When a single fixed target was used, the weak priority answer still over-resolved (targetResolved=true with inferred "not a constraint" meaning). This confirms the issue lives in the answer-resolution step itself, not in how the question frames it.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — added full Experiment 55A entry; applied 54Z wording corrections
|
||||
- `docs/current-handoff.md` — updated with Experiment 55A summary and new Return-to-Work note
|
||||
|
||||
### Confirmation Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||||
|
||||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged
|
||||
|
||||
Answer-resolution instruction identical to Experiment 54V. No stronger coaching, no added examples. Output contract unchanged from 54V (`{ resolvedMeaning, targetResolved, remainingUncertainty }`).
|
||||
|
||||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||||
|
||||
### Confirmation Behaviour Selection Remained Unchanged
|
||||
|
||||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||||
|
||||
### Confirmation Graph and UI Remained Unchanged
|
||||
|
||||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||||
|
||||
### Confirmation No Uncertainty-Preservation Logic Entered Active Runtime
|
||||
|
||||
This experiment created one new test file only. No uncertainty-preservation logic entered any active runtime path, production module, or behaviour selection output.
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 55A)
|
||||
|
||||
54Z showed weaker answers can produce different downstream resolution states under different framings, suggesting target broadening matters; 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer) to test whether uncertainty preservation holds independently of framing. The explicit case resolved correctly, the non-answer remained honestly unresolved, but the weak-priority case over-resolved by setting targetResolved=true and inferring "not a constraint" from relative importance alone. Conditional language was also flattened even when resolution was correct. Uncertainty preservation remains unreliable for weak-priority answers. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { config } from "dotenv";
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
config({ path: path.resolve(__dirname, "../../.env.local") });
|
||||
|
||||
const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
|
||||
const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
|
||||
|
||||
if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
|
||||
throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
|
||||
}
|
||||
|
||||
/**
|
||||
* Make one live Ollama chat call: take a clarification answer
|
||||
* and return what was resolved, whether the target is resolved,
|
||||
* and any remaining uncertainty about that specific target.
|
||||
*/
|
||||
async function callClarificationAnswerResolution(source, clarificationTarget, clarificationQuestion, userAnswer) {
|
||||
const instruction = `Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
Return valid JSON only in this shape:
|
||||
{
|
||||
"resolvedMeaning": "short statement",
|
||||
"targetResolved": true,
|
||||
"remainingUncertainty": null
|
||||
}`;
|
||||
|
||||
const messages = [
|
||||
{ role: "system", content: instruction.trim() },
|
||||
{
|
||||
role: "user",
|
||||
content: `Source: ${JSON.stringify(source)}
|
||||
|
||||
Clarification target: ${clarificationTarget}
|
||||
|
||||
Clarification question: ${clarificationQuestion}
|
||||
|
||||
User's answer: ${userAnswer}`,
|
||||
},
|
||||
];
|
||||
|
||||
const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: OLLAMA_MODEL,
|
||||
messages,
|
||||
format: "json",
|
||||
stream: false,
|
||||
}),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
|
||||
}
|
||||
|
||||
const data = await res.json();
|
||||
const rawContent = data.message?.content ?? "";
|
||||
const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
|
||||
|
||||
return JSON.parse(cleaned.trim());
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Experiment 55A — Fixed source, target, question
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
const SOURCE = "I want the business to grow, but I don't want to take on more risk.";
|
||||
const CLARIFICATION_TARGET = "whether avoiding additional risk is a preference/trade-off or a hard constraint";
|
||||
const CLARIFICATION_QUESTION = "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?";
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Four fixed cases
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
id: "Case 1 — Fully Explicit Hard Constraint",
|
||||
userAnswer: "It's a hard constraint. I don't want any increase in risk.",
|
||||
humanReference: {
|
||||
resolvedMeaning: "avoiding additional risk is a hard constraint",
|
||||
targetResolved: true,
|
||||
remainingUncertainty: null,
|
||||
},
|
||||
purpose: "Positive control — explicit hard constraint should resolve cleanly.",
|
||||
},
|
||||
{
|
||||
id: "Case 2 — Weak Priority Statement",
|
||||
userAnswer: "Risk matters more to me.",
|
||||
humanReference: {
|
||||
resolvedMeaning: null, // do not accept a resolved meaning that infers risk is not a hard constraint
|
||||
targetResolved: false,
|
||||
remainingUncertainty: "whether avoiding additional risk is a preference/trade-off or a hard constraint remains unclear",
|
||||
},
|
||||
purpose: "Weak answer should preserve uncertainty rather than over-resolve.",
|
||||
},
|
||||
{
|
||||
id: "Case 3 — Conditional Trade-Off",
|
||||
userAnswer: "I'd normally avoid more risk, but for the right opportunity I might accept some.",
|
||||
humanReference: {
|
||||
resolvedMeaning: null, // should preserve conditional qualification, not flatten to "simply a preference"
|
||||
targetResolved: true,
|
||||
remainingUncertainty: null,
|
||||
},
|
||||
purpose: "Resolved answer should preserve conditionality without flattening.",
|
||||
},
|
||||
{
|
||||
id: "Case 4 — Non-Answer / Insufficient Clarification",
|
||||
userAnswer: "I'm not really sure.",
|
||||
humanReference: {
|
||||
resolvedMeaning: null, // should not invent a position
|
||||
targetResolved: false,
|
||||
remainingUncertainty: "whether avoiding additional risk is a preference/trade-off or a hard constraint remains unresolved",
|
||||
},
|
||||
purpose: "Non-answer should remain unresolved without inventing meaning.",
|
||||
},
|
||||
];
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Semantic evaluation
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Classify the model's output for each case.
|
||||
* Returns { classification, reasoning } where classification is one of:
|
||||
* resolution_correct, uncertainty_preserved, uncertainty_overresolved, resolution_failed
|
||||
*/
|
||||
function classifyCase(caseNum, modelResult) {
|
||||
const { resolvedMeaning, targetResolved, remainingUncertainty } = modelResult;
|
||||
|
||||
// Structural checks first
|
||||
if (typeof resolvedMeaning !== "string" || !resolvedMeaning.trim()) {
|
||||
return { classification: "resolution_failed", reasoning: "missing or empty resolvedMeaning" };
|
||||
}
|
||||
if (typeof targetResolved !== "boolean") {
|
||||
return { classification: "resolution_failed", reasoning: `targetResolved is ${typeof targetResolved}, expected boolean` };
|
||||
}
|
||||
|
||||
const meaning = resolvedMeaning.toLowerCase();
|
||||
|
||||
// Check whether the model produced stronger meaning than the user supplied
|
||||
const hasForcedStronger = () => {
|
||||
// Force hard constraint where user didn't supply it
|
||||
if (caseNum !== 1 && /hard.*constraint|no.*increase|must.*not|absolute.*limit|will.*never/i.test(meaning)) {
|
||||
return true;
|
||||
}
|
||||
// Flatten conditionality
|
||||
if (caseNum === 3) {
|
||||
if (/simply a preference|risk is just|i would always avoid/i.test(meaning) && !/but|condition|might|would normally/i.test(meaning)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
// Case-specific classification
|
||||
switch (caseNum) {
|
||||
case 1: {
|
||||
// Expected: targetResolved=true, no remaining uncertainty, meaning says hard constraint
|
||||
if (targetResolved !== true) {
|
||||
return { classification: "resolution_failed", reasoning: `Expected targetResolved=true, got ${targetResolved}` };
|
||||
}
|
||||
if (remainingUncertainty !== null && remainingUncertainty !== undefined) {
|
||||
return { classification: "resolution_correct", reasoning: `Correctly resolved but retained uncertainty "${remainingUncertainty}" when null was expected` };
|
||||
}
|
||||
if (/hard.*(constraint|boundary|limit)|no.*(increase|more).*risk|must.*not.*take/i.test(meaning)) {
|
||||
return { classification: "resolution_correct", reasoning: "Correctly resolved explicit hard constraint with appropriate strength." };
|
||||
}
|
||||
return { classification: "uncertainty_overresolved", reasoning: `Meaning does not clearly express hard constraint: "${resolvedMeaning}"` };
|
||||
}
|
||||
|
||||
case 2: {
|
||||
// Expected: targetResolved=false, remainingUncertainty preserves the distinction
|
||||
if (targetResolved === true) {
|
||||
return { classification: "uncertainty_overresolved", reasoning: `Over-resolved weak answer: targetResolved=true when it should be false. Meaning: "${resolvedMeaning}"` };
|
||||
}
|
||||
if (!/risk.*preference|trade.?off|constraint|preference.*(vs|or|versus)|uncertain.*risk/i.test(meaning) && meaning.includes("not")) {
|
||||
// If the model explicitly says "not a hard constraint" that's over-resolving
|
||||
if (/not.*hard.*constraint|clearly.*not.*constraint|definitely.*not/i.test(meaning)) {
|
||||
return { classification: "uncertainty_overresolved", reasoning: `Model inferred "risk is not a hard constraint" from weak answer "${resolvedMeaning}"` };
|
||||
}
|
||||
}
|
||||
if (remainingUncertainty === null || remainingUncertainty === undefined) {
|
||||
return { classification: "uncertainty_overresolved", reasoning: "targetResolved=false but no remainingUncertainty provided — uncertainty erased." };
|
||||
}
|
||||
// Check that remainingUncertainty preserves the preference vs constraint distinction
|
||||
if (/preference|trade.?off|constraint|uncertain.*distinction|preference.*(vs|or)/i.test(remainingUncertainty)) {
|
||||
return { classification: "uncertainty_preserved", reasoning: `Weak answer correctly left unresolved with appropriate remaining uncertainty.` };
|
||||
}
|
||||
return { classification: "uncertainty_preserved", reasoning: `targetResolved=false with remainingUncertainty provided (semantic match unclear but structurally correct).` };
|
||||
}
|
||||
|
||||
case 3: {
|
||||
// Expected: targetResolved=true, meaning preserves conditionality
|
||||
if (targetResolved !== true) {
|
||||
return { classification: "resolution_failed", reasoning: `Expected targetResolved=true, got ${targetResolved}` };
|
||||
}
|
||||
if (/but|condition|might|would normally|conditional/i.test(meaning)) {
|
||||
return { classification: "resolution_correct", reasoning: "Correctly resolved conditional trade-off while preserving its conditional qualification." };
|
||||
}
|
||||
// Check for over-resolution (flattening)
|
||||
if (/simply.*preference|risk is just|i would always|definitely not.*hard constraint/i.test(meaning)) {
|
||||
return { classification: "uncertainty_overresolved", reasoning: `Flattened conditional answer to flat meaning: "${resolvedMeaning}"` };
|
||||
}
|
||||
return { classification: "resolution_correct", reasoning: `Resolved with meaning "${resolvedMeaning}" — conditionality may or may not be explicit.` };
|
||||
}
|
||||
|
||||
case 4: {
|
||||
// Expected: targetResolved=false, no invented position, remainingUncertainty states unresolved
|
||||
if (targetResolved === true) {
|
||||
return { classification: "resolution_failed", reasoning: `Over-resolved non-answer: targetResolved=true when it should be false.` };
|
||||
}
|
||||
// Check for invented meaning
|
||||
const hasInventedPosition = /will.*avoid|would.*never|always avoid/i.test(meaning);
|
||||
if (hasInventedPosition) {
|
||||
return { classification: "uncertainty_overresolved", reasoning: `Model invented a position from non-answer: "${resolvedMeaning}"` };
|
||||
}
|
||||
// Check that the distinction is stated as unresolved in remainingUncertainty
|
||||
if (remainingUncertainty === null || remainingUncertainty === undefined) {
|
||||
return { classification: "uncertainty_preserved", reasoning: `Correctly left target unresolved with no remaining uncertainty — the distinction is genuinely absent.` };
|
||||
}
|
||||
if (/unresolved|remains unclear|still unknown|preference.*constraint/i.test(remainingUncertainty)) {
|
||||
return { classification: "uncertainty_preserved", reasoning: "Non-answer correctly left unresolved without inventing meaning." };
|
||||
}
|
||||
return { classification: "uncertainty_preserved", reasoning: `targetResolved=false with remainingUncertainty provided (structurally correct).` };
|
||||
}
|
||||
|
||||
default:
|
||||
return { classification: "resolution_failed", reasoning: "Unrecognized case number" };
|
||||
}
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Test suite
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("Experiment 55A - Clarification Uncertainty Preservation", () => {
|
||||
const results = [];
|
||||
const timings = [];
|
||||
|
||||
for (let i = 0; i < CASES.length; i++) {
|
||||
const c = CASES[i];
|
||||
const caseNum = i + 1;
|
||||
|
||||
it(`${c.id}`, async () => {
|
||||
const start = Date.now();
|
||||
const result = await callClarificationAnswerResolution(
|
||||
SOURCE, CLARIFICATION_TARGET, CLARIFICATION_QUESTION, c.userAnswer
|
||||
);
|
||||
const elapsed = Date.now() - start;
|
||||
timings.push({ caseId: c.id, ms: elapsed });
|
||||
|
||||
const ev = classifyCase(caseNum, result);
|
||||
|
||||
results.push({
|
||||
caseNumber: caseNum,
|
||||
case: c,
|
||||
modelResult: result,
|
||||
classification: ev.classification,
|
||||
reasoning: ev.reasoning,
|
||||
timingMs: elapsed,
|
||||
});
|
||||
|
||||
// Structural assertions for all cases
|
||||
expect(result.resolvedMeaning).toBeDefined();
|
||||
expect(typeof result.resolvedMeaning).toBe("string");
|
||||
expect(result.resolvedMeaning.trim().length).toBeGreaterThan(0);
|
||||
expect(result.targetResolved).toBeDefined();
|
||||
expect(typeof result.targetResolved).toBe("boolean");
|
||||
|
||||
// Case-specific semantic assertions from human reference
|
||||
if (caseNum === 1) {
|
||||
expect(result.targetResolved).toBe(true);
|
||||
} else if (caseNum === 2) {
|
||||
expect(result.targetResolved).toBe(false);
|
||||
} else if (caseNum === 3) {
|
||||
expect(result.targetResolved).toBe(true);
|
||||
} else if (caseNum === 4) {
|
||||
expect(result.targetResolved).toBe(false);
|
||||
}
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
it("55A: aggregate results and analysis", () => {
|
||||
const classificationCounts = {
|
||||
resolution_correct: 0,
|
||||
uncertainty_preserved: 0,
|
||||
uncertainty_overresolved: 0,
|
||||
resolution_failed: 0,
|
||||
};
|
||||
|
||||
for (const r of results) {
|
||||
classificationCounts[r.classification]++;
|
||||
}
|
||||
|
||||
const totalMs = timings.reduce((s, t) => s + t.ms, 0);
|
||||
const msArr = timings.map((t) => t.ms);
|
||||
|
||||
console.log("\n========== Experiment 55A Results ==========");
|
||||
console.log(`\nSource: ${SOURCE}`);
|
||||
console.log(`Target: ${CLARIFICATION_TARGET}`);
|
||||
console.log(`Question: ${CLARIFICATION_QUESTION}`);
|
||||
|
||||
for (const r of results) {
|
||||
console.log(`\n--- Case ${r.caseNumber}: ${r.case.purpose} ---`);
|
||||
console.log("Answer:", `"${r.case.userAnswer}"`);
|
||||
console.log("resolvedMeaning:", r.modelResult.resolvedMeaning);
|
||||
console.log("targetResolved:", r.modelResult.targetResolved);
|
||||
console.log("remainingUncertainty:", r.modelResult.remainingUncertainty ?? "null");
|
||||
console.log("Classification:", r.classification);
|
||||
console.log("Reasoning:", r.reasoning);
|
||||
}
|
||||
|
||||
console.log("\n--- Classification Counts ---");
|
||||
Object.entries(classificationCounts).forEach(([k, v]) => {
|
||||
console.log(`${k}: ${v}`);
|
||||
});
|
||||
|
||||
console.log("\n--- Timing ---");
|
||||
console.log("Calls:", timings.length);
|
||||
console.log("Total:", totalMs + "ms");
|
||||
console.log("Average:", (totalMs / timings.length).toFixed(1) + "ms per call");
|
||||
console.log("Fastest:", Math.min(...msArr) + "ms");
|
||||
console.log("Slowest:", Math.max(...msArr) + "ms");
|
||||
|
||||
// Summary questions
|
||||
const q2_preserved = results[1].classification === "uncertainty_preserved";
|
||||
const q3_conditionality = /resolution_correct/i.test(results[2].reasoning);
|
||||
const q4_unresolved = results[3].classification !== "resolution_failed" && !/invented.*position/i.test(results[3].reasoning.toLowerCase());
|
||||
|
||||
console.log("\n--- Key Questions ---");
|
||||
console.log("Q1: Case 1 correctly resolved explicit hard constraint?", results[0].classification === "resolution_correct" || results[0].classification === "uncertainty_overresolved" ? "Yes (resolved)" : "No");
|
||||
console.log("Q2: Case 2 preserved uncertainty rather than over-resolving?", q2_preserved ? "Yes" : "No — over-resolved or failed");
|
||||
console.log("Q3: Case 3 preserved conditional trade-off?", q3_conditionality ? "Yes" : "Need review");
|
||||
console.log("Q4: Case 4 remained unresolved without inventing meaning?", q4_unresolved ? "Yes" : "No");
|
||||
|
||||
const anyOverresolving = classificationCounts.uncertainty_overresolved > 0;
|
||||
const anyFailed = classificationCounts.resolution_failed > 0;
|
||||
|
||||
console.log("\n--- Summary ---");
|
||||
console.log("Any over-resolution of weak answers:", anyOverresolving);
|
||||
console.log("Any resolution failures:", anyFailed);
|
||||
console.log("Conclusion: The answer-resolution step", anyOverresolving ? "does appear to over-resolve some weak answers" : "appears honest about uncertainty levels");
|
||||
|
||||
results.summary = {
|
||||
classificationCounts,
|
||||
totalTimeMs: totalMs,
|
||||
averageMs: (totalMs / timings.length).toFixed(1),
|
||||
fastestMs: Math.min(...msArr),
|
||||
slowestMs: Math.max(...msArr),
|
||||
anyOverresolution: anyOverresolving,
|
||||
};
|
||||
|
||||
console.log("\n========== End of Experiment 55A ==========\n");
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user