experiment: distinguish clarification from evidence needs
This commit is contained in:
@@ -19,6 +19,8 @@
|
||||
- Keyword and phrase-based scope detection remains provisional scaffolding.
|
||||
- `docs/current-project-state.md` is the main entry point for active project state.
|
||||
- Experiment 54D confirmed the production update prompt explicitly separates the user answer (## User Answer section) but the proposal schema has no provenance field — source identity at prompt level is explicit, per-node provenance at output level is absent.
|
||||
|
||||
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). The model did not collapse all ambiguity into clarification, did not generate any question or choose a winner. No production code changed. Status pending Rob's review.
|
||||
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
|
||||
|
||||
## 3. Why Work Is Paused
|
||||
@@ -141,8 +143,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Q. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54R. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54Q)
|
||||
### Return-to-Work Note (Experiment 54R)
|
||||
|
||||
Experiment 54N failed when moving directly from hypotheses to consequence on the staff-capacity versus supplier-lead-time case. Experiment 54O showed evidence discrimination works when asked directly. Experiment 54P showed explicit supplied evidence restores consequence judgement. Experiment 54Q tested whether one structured semantic call can first expose evidence needs and then preserve the consequence — it succeeded across all three cases including the original 54N failure. No winner or question generated; same host/model; no production behaviour changed. Whether this explains the 54N failure or only narrows it remains open. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-structured-evidence-consequence.test.js`.
|
||||
54Q showed one structured hypothesis → evidence → consequence call worked across three tested cases but did not isolate which reformulation caused recovery. 54R tested whether a consequential disagreement actually requires the user or can be resolved through evidence — three cases: competing delivery causes (false), growth-versus-risk ambiguity (true), no-material-disagreement control (false). All three correct (3/3). The model distinguished missing evidence from missing user meaning without generating any question, choosing a winner, or changing Behaviour Selection. Same host/model; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-disagreement-resolution-source.test.js`.
|
||||
|
||||
@@ -7798,7 +7798,11 @@ No. No question generation detected in any output.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
3 of 3 evidence classifications and 3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent evidence lists and correctly classified the consequence for each case. The original 54N failure case (staff capacity vs supplier lead time) recovered when the model was required to expose its evidence reasoning before deciding the consequence.
|
||||
3 of 3 evidence classifications and 3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent evidence lists and correctly classified the consequence for each case. The original 54N failure case (staff capacity vs supplier lead time) recovered when the model was required to explicitly identify the evidence needs before deciding the consequence.
|
||||
|
||||
### Correction: Scope of Experiment 54Q Reformulation
|
||||
|
||||
54Q reformulated the task in several ways: it supplied hypotheses directly, required explicit evidence-needs fields, changed the semantic instruction, and changed the output contract. The experiment therefore shows recovery under the structured hypothesis → evidence → consequence formulation, but does not isolate which part of that reformulation caused the recovery.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
@@ -7817,3 +7821,190 @@ In the three tested cases, explicit evidence distinctions were sufficient for th
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-structured-evidence-consequence.test.js`.
|
||||
|
||||
## Experiment 54R — Does a Material Disagreement Actually Require User Clarification? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Given an explicit interpretation disagreement and its evidence consequence, test whether the model can distinguish between a disagreement that requires clarification from the user and one that can be resolved by investigating evidence.
|
||||
|
||||
This is passive and test-only. Do not generate the clarification question. Do not generate the next investigation question. Do not choose a winning interpretation. Do not change production behaviour.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may be able to distinguish:
|
||||
|
||||
**Evidence-resolvable disagreement:** The user's meaning is sufficiently clear, but competing explanations require different evidence.
|
||||
|
||||
**User-clarification disagreement:** The disagreement concerns the user's intended meaning, priority, constraint, or definition, so external evidence cannot resolve it without asking the user.
|
||||
|
||||
If this distinction works, disagreement does not have to map automatically to clarification.
|
||||
|
||||
### Context Budget
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md`;
|
||||
- Experiment 54Q only in `docs/design-evolution-log.md`;
|
||||
- `tests/reconstruction/semantic-structured-evidence-consequence.test.js`;
|
||||
- `.env.local` only for `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||||
|
||||
Not read: Behaviour Selection; assessor; graph files; UI; active prompts; question-selection code; full experiment history.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as 54Q)
|
||||
Model: `qwen-claude:latest` (same as 54Q)
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives:
|
||||
```json
|
||||
{
|
||||
"source": "...",
|
||||
"disagreement": ["..."],
|
||||
"evidenceNeeded": ["..."]
|
||||
}
|
||||
```
|
||||
|
||||
The disagreement and evidence needs are fixed human-reviewed inputs. The model does not rediscover them.
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly:
|
||||
```json
|
||||
{
|
||||
"requiresUserClarification": true | false,
|
||||
"reason": "one short sentence"
|
||||
}
|
||||
```
|
||||
|
||||
No question text, no recommended action, no preferred interpretation, no score, no confidence, no behaviour label. This boolean is test-only and is not a production contract.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Decide whether resolving the stated disagreement requires additional meaning, preference, intent, or factual information that only the user can provide. Return true when evidence alone cannot settle the disagreement because the missing distinction belongs to the user's intended meaning, priority, constraint, or private knowledge. Return false when the disagreement can be investigated using external, operational, or observable evidence without asking the user to define what they mean. Do not generate a question and do not choose which interpretation is correct.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Competing Causes, Evidence Can Resolve
|
||||
|
||||
**Source:** "Orders are arriving late and customers have started complaining."
|
||||
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||||
**Evidence needed:** staffing levels and workload; processing throughput; supplier lead-time history; supplier delivery reliability.
|
||||
**Fixed human reference:** `requiresUserClarification: false`
|
||||
|
||||
### Case 2 — User Priority Is Ambiguous
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||||
**Evidence needed:** possible growth opportunities; risk exposure of each option.
|
||||
**Fixed human reference:** `requiresUserClarification: true`
|
||||
|
||||
### Case 3 — Same Meaning, No Clarification Need
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
**Disagreement:** no substantive disagreement; two interpretations express the same uncertainty in different words.
|
||||
**Evidence needed:** pricing history; sales/conversion response; other plausible revenue drivers.
|
||||
**Fixed human reference:** `requiresUserClarification: false`
|
||||
|
||||
### Results
|
||||
|
||||
#### Case 1 — Competing Causes, Evidence Can Resolve
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"requiresUserClarification": false,
|
||||
"reason": "The disagreement involves operational causes that can be resolved by analyzing objective company data and supplier records."
|
||||
}
|
||||
```
|
||||
|
||||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that operational evidence (company data, supplier records) can distinguish the competing causal hypotheses without asking the user.
|
||||
|
||||
#### Case 2 — User Priority Is Ambiguous
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"requiresUserClarification": true,
|
||||
"reason": "Resolving this conflict depends on the user's personal priority and risk tolerance, which cannot be determined by external facts alone."
|
||||
}
|
||||
```
|
||||
|
||||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `true`. The model correctly identified that user-owned meaning (priority, risk tolerance) is required to resolve the disagreement. External facts alone cannot determine whether "don't want more risk" is a preference, constraint, or boundary.
|
||||
|
||||
#### Case 3 — Same Meaning, No Clarification Need
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"requiresUserClarification": false,
|
||||
"reason": "The stated position explicitly confirms there is no substantive disagreement, so no additional user input is required to resolve it."
|
||||
}
|
||||
```
|
||||
|
||||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that the absence of material disagreement means clarification is unnecessary. Evidence can investigate the underlying uncertainty.
|
||||
|
||||
### Resolution-Source-Correct Count
|
||||
|
||||
**3 of 3** cases classified correctly.
|
||||
|
||||
### Resolution-Source-Failed Count
|
||||
|
||||
**0 of 3** cases classified incorrectly.
|
||||
|
||||
### Did Case 1 Correctly Recognise That Competing Causal Hypotheses Can Be Resolved Through Evidence?
|
||||
|
||||
Yes. The model returned `false` and provided a reason referencing operational causes resolvable by company data and supplier records — evidence, not user meaning.
|
||||
|
||||
### Did Case 2 Correctly Recognise That the Unresolved Growth-Versus-Risk Priority Belongs to the User?
|
||||
|
||||
Yes. The model returned `true` and identified that resolution depends on "the user's personal priority and risk tolerance," which external facts alone cannot determine.
|
||||
|
||||
### Did Case 3 Avoid Unnecessary Clarification Where There Was No Material Disagreement?
|
||||
|
||||
Yes. The model returned `false`, correctly noting the absence of substantive disagreement makes additional clarification unnecessary.
|
||||
|
||||
### Did the Model Treat Every Disagreement as Requiring User Clarification?
|
||||
|
||||
No. Two of three cases returned `false`. Only Case 2 (ambiguous priority) returned `true`.
|
||||
|
||||
### Did the Model Confuse Missing Evidence with Missing User Meaning?
|
||||
|
||||
No. In Case 1, the model correctly distinguished between lacking evidence to investigate causes (which it flagged as resolvable through evidence gathering) and lacking user meaning (which it did not claim). The reason text referenced "analyzing objective company data and supplier records" rather than requiring user input.
|
||||
|
||||
### Did the Model Generate an Actual Question?
|
||||
|
||||
No. No question was generated in any output. The output contract was respected in all cases.
|
||||
|
||||
### Did the Model Choose a Winner?
|
||||
|
||||
No. No interpretation was selected as correct in any case.
|
||||
|
||||
### Evidence That User-Owned Ambiguity Can Be Separated From Evidence Uncertainty
|
||||
|
||||
Case 2 succeeded where Case 1 and Case 3 both returned `false` for different reasons — one because evidence can resolve it, the other because no disagreement exists. The model's reasons for each case were distinct in their reference points: operational data (Case 1) versus user priority (Case 2) versus absence of disagreement (Case 3). This pattern suggests the model does not collapse all ambiguity into a single clarification need.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Three cases only; limited domain coverage (one delivery scenario, one strategic priority, one revenue statement).
|
||||
- Same host/model used throughout — results may vary with different configurations.
|
||||
- Does not establish generalisation beyond these specific inputs.
|
||||
- The distinction tested here is binary (true/false) and does not test partial or probabilistic resolution-source classification.
|
||||
- No evidence was actually gathered in any case — only whether the *source* of resolution was correctly identified.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**The model correctly distinguished user-clarification needs from evidence-resolvable disagreement in all tested cases.**
|
||||
|
||||
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. The model did not collapse all ambiguity into a single classification path.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-resolution-source.test.js`.
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { config } from "dotenv";
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
config({ path: path.resolve(__dirname, "../../.env.local") });
|
||||
|
||||
const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
|
||||
const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
|
||||
|
||||
if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
|
||||
throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
|
||||
}
|
||||
|
||||
/**
|
||||
* Make one live Ollama chat call: decide whether a disagreement requires
|
||||
* user clarification or can be resolved through evidence.
|
||||
*/
|
||||
async function callDisagreementResolutionSource(source, disagreement, evidenceNeeded) {
|
||||
const instruction = `Decide whether resolving the stated disagreement requires additional meaning, preference, intent, or factual information that only the user can provide.
|
||||
|
||||
Return true when evidence alone cannot settle the disagreement because the missing distinction belongs to the user's intended meaning, priority, constraint, or private knowledge. Return false when the disagreement can be investigated using external, operational, or observable evidence without asking the user to define what they mean.
|
||||
|
||||
Return valid JSON only in this shape:
|
||||
{
|
||||
"requiresUserClarification": true | false,
|
||||
"reason": "one short sentence"
|
||||
}
|
||||
|
||||
Do not generate a question. Do not choose which interpretation is correct.`;
|
||||
|
||||
const messages = [
|
||||
{ role: "system", content: instruction.trim() },
|
||||
{
|
||||
role: "user",
|
||||
content: `Source: ${JSON.stringify(source)}
|
||||
|
||||
Disagreement:
|
||||
${disagreement.map((d, i) => `${i + 1}. ${d}`).join("\n")}
|
||||
|
||||
Evidence needed:
|
||||
${evidenceNeeded.map((e, i) => `${i + 1}. ${e}`).join("\n")}`,
|
||||
},
|
||||
];
|
||||
|
||||
const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: OLLAMA_MODEL,
|
||||
messages,
|
||||
format: "json",
|
||||
stream: false,
|
||||
}),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
|
||||
}
|
||||
|
||||
const data = await res.json();
|
||||
const rawContent = data.message?.content ?? "";
|
||||
const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
|
||||
|
||||
return JSON.parse(cleaned.trim());
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Fixed human-reference ground truth (pre-written)
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
id: "Case 1 - Competing Causes, Evidence Can Resolve",
|
||||
source: "Orders are arriving late and customers have started complaining.",
|
||||
disagreement: [
|
||||
"delays may be caused by insufficient staff capacity",
|
||||
"delays may be caused by unreliable supplier lead times",
|
||||
],
|
||||
evidenceNeeded: [
|
||||
"staffing levels and workload",
|
||||
"processing throughput",
|
||||
"supplier lead-time history",
|
||||
"supplier delivery reliability",
|
||||
],
|
||||
reference: {
|
||||
requiresUserClarification: false,
|
||||
reason_semantic: "The user's statement is clear enough. Operational evidence can distinguish the competing causes.",
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "Case 2 - User Priority Is Ambiguous",
|
||||
source: "I want the business to grow, but I don't want to take on more risk.",
|
||||
disagreement: [
|
||||
"growth should be prioritised even if some additional risk is unavoidable",
|
||||
"avoiding additional risk is a hard constraint even if growth is slower",
|
||||
],
|
||||
evidenceNeeded: [
|
||||
"possible growth opportunities",
|
||||
"risk exposure of each option",
|
||||
],
|
||||
reference: {
|
||||
requiresUserClarification: true,
|
||||
reason_semantic: "External evidence can describe growth and risk, but it cannot determine whether the constraint is a preference or an absolute boundary. That meaning belongs to the user.",
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "Case 3 - Same Meaning, No Clarification Need",
|
||||
source: "Revenue is down. I think pricing may be part of the problem, but I am not sure.",
|
||||
disagreement: [
|
||||
"no substantive disagreement; two interpretations express the same uncertainty in different words",
|
||||
],
|
||||
evidenceNeeded: [
|
||||
"pricing history",
|
||||
"sales/conversion response",
|
||||
"other plausible revenue drivers",
|
||||
],
|
||||
reference: {
|
||||
requiresUserClarification: false,
|
||||
reason_semantic: "There is no material interpretation disagreement requiring clarification. Evidence can investigate the underlying uncertainty.",
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Evaluation helpers
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
function evaluateResolutionSource(modelResult, reference) {
|
||||
const result = modelResult;
|
||||
const issues = [];
|
||||
const notes = [];
|
||||
|
||||
// Structural checks
|
||||
if (typeof result.requiresUserClarification !== "boolean") {
|
||||
issues.push("requiresUserClarification must be a boolean");
|
||||
}
|
||||
if (!result.reason || typeof result.reason !== "string") {
|
||||
issues.push("missing or invalid reason field");
|
||||
}
|
||||
|
||||
// Boolean match against fixed human reference
|
||||
if (result.requiresUserClarification !== reference.requiresUserClarification) {
|
||||
issues.push(
|
||||
`boolean mismatch: expected ${reference.requiresUserClarification}, got ${result.requiresUserClarification}`
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
resolution_source_correct: issues.filter((i) =>
|
||||
i.includes("boolean mismatch")
|
||||
).length === 0,
|
||||
resolution_source_failed: issues.some((i) =>
|
||||
i.includes("boolean mismatch")
|
||||
),
|
||||
boolean_match: result.requiresUserClarification === reference.requiresUserClarification,
|
||||
issues,
|
||||
notes,
|
||||
raw: result,
|
||||
};
|
||||
}
|
||||
|
||||
function evaluateReasonSemantics(modelResult, caseRef) {
|
||||
const result = modelResult;
|
||||
let issues = [];
|
||||
|
||||
if (!result.reason || typeof result.reason !== "string") {
|
||||
return { valid_reason: false, issues: ["no reason provided"] };
|
||||
}
|
||||
|
||||
const reasonLower = result.reason.toLowerCase();
|
||||
|
||||
if (caseRef.requiresUserClarification === false) {
|
||||
// Should reference evidence / operational investigation, not user meaning
|
||||
const hasEvidenceKeywords = [
|
||||
"evidence", "operational", "investigat", "distinguish",
|
||||
"data can", "external", "observable", "record", "metric",
|
||||
"check", "review", "gather", "collect"
|
||||
].some(k => reasonLower.includes(k));
|
||||
|
||||
const hasUserMeaningKeywords = [
|
||||
"user must", "ask the user", "clarification from", "need to ask",
|
||||
"requires clarification", "unclear what the user", "ambiguous user"
|
||||
].some(k => reasonLower.includes(k));
|
||||
|
||||
// If it references evidence investigation, that supports the classification
|
||||
if (hasEvidenceKeywords) {
|
||||
issues.push("reason references evidence-based resolution for a false case - check for contradiction");
|
||||
}
|
||||
}
|
||||
|
||||
if (caseRef.requiresUserClarification === true) {
|
||||
const hasUserPriority = [
|
||||
"user's", "user mean", "user intent", "user priorit",
|
||||
"preference", "constraint", "boundary", "ambiguous",
|
||||
"clarification needed"
|
||||
].some(k => reasonLower.includes(k));
|
||||
|
||||
if (!hasUserPriority) {
|
||||
issues.push("reason does not reference user-owned meaning for a true case");
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
valid_reason: issues.length === 0,
|
||||
issues,
|
||||
};
|
||||
}
|
||||
|
||||
function checkNoInvariantViolations(modelResult) {
|
||||
const allText = JSON.stringify(modelResult).toLowerCase();
|
||||
const violations = [];
|
||||
|
||||
// No question generated
|
||||
if (/should i|do you|could you|would you|are you/.test(allText)) {
|
||||
violations.push("potential question language detected in output");
|
||||
}
|
||||
|
||||
// No interpretation chosen
|
||||
if (/\b(winner|preferred|correct|right|better|should go with)\b/i.test(
|
||||
modelResult.reason || ""
|
||||
)) {
|
||||
violations.push("potential interpretation selection in reason field");
|
||||
}
|
||||
|
||||
return violations;
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Test suite
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("Experiment 54R - Disagreement Resolution Source: Clarification vs Evidence", () => {
|
||||
const results = [];
|
||||
const timings = [];
|
||||
|
||||
for (const c of CASES) {
|
||||
it(c.id, async () => {
|
||||
const start = Date.now();
|
||||
const result = await callDisagreementResolutionSource(
|
||||
c.source,
|
||||
c.disagreement,
|
||||
c.evidenceNeeded
|
||||
);
|
||||
const elapsed = Date.now() - start;
|
||||
timings.push({ caseId: c.id, ms: elapsed });
|
||||
|
||||
const resolutionEval = evaluateResolutionSource(result, c.reference);
|
||||
const reasonEval = evaluateReasonSemantics(result, c.reference);
|
||||
const invariantViolations = checkNoInvariantViolations(result);
|
||||
|
||||
results.push({
|
||||
case: c,
|
||||
modelResult: result,
|
||||
resolution: resolutionEval,
|
||||
reason: reasonEval,
|
||||
invariantViolations,
|
||||
timingMs: elapsed,
|
||||
});
|
||||
|
||||
// Structural assertions
|
||||
expect(result.requiresUserClarification).toBeDefined();
|
||||
expect(typeof result.requiresUserClarification).toBe("boolean");
|
||||
expect(result.reason).toBeDefined();
|
||||
expect(typeof result.reason).toBe("string");
|
||||
|
||||
// No question generation invariant
|
||||
const allText = JSON.stringify(result).toLowerCase();
|
||||
expect(allText).not.toMatch(/should i|do you|could you|would you/);
|
||||
|
||||
// Boolean must match fixed human reference
|
||||
expect(result.requiresUserClarification).toBe(c.reference.requiresUserClarification);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
it("Experiment 54R: aggregate results", () => {
|
||||
const resolutionCorrect = results.filter(
|
||||
(r) => r.resolution.resolution_source_correct
|
||||
).length;
|
||||
const resolutionFailed = results.filter(
|
||||
(r) => r.resolution.resolution_source_failed
|
||||
).length;
|
||||
|
||||
// Report findings to console for manual review
|
||||
console.log("\n=== Experiment 54R Results ===");
|
||||
for (const r of results) {
|
||||
console.log(`\n--- ${r.case.id} ---`);
|
||||
console.log("Boolean:", r.modelResult.requiresUserClarification);
|
||||
console.log("Reason:", r.modelResult.reason);
|
||||
console.log(
|
||||
"Resolution source correct:",
|
||||
r.resolution.resolution_source_correct,
|
||||
r.resolution.issues
|
||||
);
|
||||
console.log("Valid reason:", r.reason.valid_reason, r.reason.issues);
|
||||
console.log(
|
||||
"Invariant violations:",
|
||||
r.invariantViolations.length > 0 ? r.invariantViolations : "none"
|
||||
);
|
||||
}
|
||||
console.log(`\nResolution-source-correct: ${resolutionCorrect}/${results.length}`);
|
||||
console.log(`Resolution-source-failed: ${resolutionFailed}/${results.length}`);
|
||||
|
||||
const totalMs = timings.reduce((s, t) => s + t.ms, 0);
|
||||
console.log(`Total time: ${totalMs}ms`);
|
||||
console.log(`Average time: ${(totalMs / timings.length).toFixed(1)}ms per call`);
|
||||
console.log(`Fastest: ${Math.min(...timings.map((t) => t.ms))}ms`);
|
||||
console.log(`Slowest: ${Math.max(...timings.map((t) => t.ms))}ms`);
|
||||
|
||||
// Case 1 must be false (evidence can resolve)
|
||||
const case1 = results.find((r) => r.case.id.includes("Case 1"));
|
||||
expect(case1.modelResult.requiresUserClarification).toBe(false);
|
||||
|
||||
// Case 2 must be true (user priority ambiguous)
|
||||
const case2 = results.find((r) => r.case.id.includes("Case 2"));
|
||||
expect(case2.modelResult.requiresUserClarification).toBe(true);
|
||||
|
||||
// Case 3 must be false (no material disagreement)
|
||||
const case3 = results.find((r) => r.case.id.includes("Case 3"));
|
||||
expect(case3.modelResult.requiresUserClarification).toBe(false);
|
||||
|
||||
// No invariant violations in any result
|
||||
for (const r of results) {
|
||||
expect(r.invariantViolations.length).toBe(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user