experiment: test evidence needs across competing hypotheses
This commit is contained in:
@@ -141,8 +141,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54N. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54O. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54N)
|
||||
### Return-to-Work Note (Experiment 54O)
|
||||
|
||||
Experiment 54M showed two interpretations can expose their substantive disagreement without selecting a winner. Experiment 54N tested whether that disagreement materially changes what needs to be established next, using three fixed cases: pricing ambiguity (correct — model detected changed information need), paraphrase agreement (correct — model avoided false consequence), and competing causes (incorrect — model treated them as same-direction investigation). The model correctly refrained from choosing either interpretation as correct or generating an actual next question in any case. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||||
Experiment 54N exposed a candidate failure pattern where competing causes sharing one diagnostic goal were treated as needing the same information. 54O directly tested whether the model can distinguish evidence needs for such hypotheses. Results across delivery causes, paraphrased same-cause control, and website-sales causes all passed (3/3). Model distinguished staff-capacity from supplier evidence, recognised paraphrased same-cause as same evidence need, and distinguished pricing from technical checkout evidence. No winner or next question was generated. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
|
||||
|
||||
@@ -7103,7 +7103,7 @@ Case 3's failure is notable but potentially narrow — the model may succeed wit
|
||||
|
||||
**Disagreement consequence detection is promising but imperfect.**
|
||||
|
||||
The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a blind spot: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model did not recognise the divergence. This is the narrowest gap identified so far in the semantic comparison chain (54K–54N).
|
||||
The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a candidate failure pattern: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model may collapse them into the same information need. This is the specific gap observed in this experiment within the semantic comparison chain (54K–54N).
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
@@ -7137,4 +7137,265 @@ No engine components, no UI components, no configuration changes. This experimen
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||||
|
||||
## Experiment 54O — Can the Model Distinguish Same Goal From Different Evidence Needs? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Test one narrow question following from Experiment 54N's candidate failure pattern:
|
||||
|
||||
> When two interpretations share the same overall diagnostic goal, can the model still recognise that they require different evidence to investigate?
|
||||
|
||||
This is a passive test-only experiment. Do not generate a next question. Do not choose a winning interpretation. Do not change production behaviour.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may be able to distinguish evidence needs correctly when asked directly about evidence rather than about the broader "information needed next" consequence. If it still collapses different hypotheses into one evidence need, the 54N failure pattern becomes stronger evidence. If it distinguishes them cleanly, the 54N failure may have been caused by the abstraction level of the consequence question rather than inability to understand the evidence difference.
|
||||
|
||||
### Context Budget
|
||||
|
||||
Read only:
|
||||
|
||||
- `docs/current-handoff.md` (Experiment 54N findings and Return-to-Work Note);
|
||||
- Experiment 54N only in `docs/design-evolution-log.md`;
|
||||
- `tests/reconstruction/semantic-disagreement-consequence.test.js` as historical reference;
|
||||
- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||||
|
||||
Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; earlier semantic experiment files.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54N)
|
||||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54N)
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives:
|
||||
|
||||
```json
|
||||
{
|
||||
"problem": "...",
|
||||
"hypothesisA": "...",
|
||||
"hypothesisB": "..."
|
||||
}
|
||||
```
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly:
|
||||
|
||||
```json
|
||||
{
|
||||
"sameEvidenceNeeded": true,
|
||||
"evidenceForA": ["..."],
|
||||
"evidenceForB": ["..."]
|
||||
}
|
||||
```
|
||||
|
||||
or the same shape with `sameEvidenceNeeded: false`.
|
||||
|
||||
No preferred hypothesis, confidence, score, next question, or recommendation.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Compare the evidence needed to investigate the two hypotheses. Return sameEvidenceNeeded: true only when materially the same evidence would test both hypotheses. Return false when each hypothesis requires meaningfully different evidence, even if both are trying to explain the same overall problem. List the main evidence needed for each hypothesis. Do not decide which hypothesis is correct and do not generate questions.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Delivery Delay / Different Evidence
|
||||
|
||||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||||
|
||||
**Hypothesis A:** "Delivery delays are being caused by insufficient staff capacity."
|
||||
|
||||
**Hypothesis B:** "Delivery delays are being caused by unreliable supplier lead times."
|
||||
|
||||
**Fixed human reference:** `sameEvidenceNeeded: false`
|
||||
|
||||
Expected evidence distinction:
|
||||
|
||||
- A should involve staffing/capacity/workload evidence;
|
||||
- B should involve supplier lead-time/reliability evidence.
|
||||
|
||||
This is the exact failure pattern from 54N.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"sameEvidenceNeeded": false,
|
||||
"evidenceForA": [
|
||||
"Staff scheduling and shift coverage data compared to daily order volume",
|
||||
"Average order processing time per employee or team",
|
||||
"Employee absenteeism, turnover rates, and overtime logs",
|
||||
"Warehouse throughput capacity limits and task completion bottlenecks"
|
||||
],
|
||||
"evidenceForB": [
|
||||
"Supplier delivery records versus contracted lead times",
|
||||
"Inventory stockout logs for critical raw materials or components",
|
||||
"Purchase order acknowledgment and fulfillment timelines",
|
||||
"Historical reliability metrics and documented delay reasons from specific vendors"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||||
|
||||
### Case 2 — Same Cause, Paraphrased / Same Evidence
|
||||
|
||||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||||
|
||||
**Hypothesis A:** "The team may not have enough capacity to process orders on time."
|
||||
|
||||
**Hypothesis B:** "Insufficient staff capacity may be causing the order delays."
|
||||
|
||||
**Fixed human reference:** `sameEvidenceNeeded: true`
|
||||
|
||||
Both should broadly require staffing/capacity/workload evidence.
|
||||
|
||||
Purpose: ensure different wording is not mistaken for different evidence needs.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"sameEvidenceNeeded": true,
|
||||
"evidenceForA": [
|
||||
"Current order volume compared to historical processing baselines",
|
||||
"Staffing levels, shift coverage, and scheduled vs. actual hours worked",
|
||||
"Order processing time logs and bottleneck analysis",
|
||||
"Workload distribution metrics across the team"
|
||||
],
|
||||
"evidenceForB": [
|
||||
"Current order volume compared to historical processing baselines",
|
||||
"Staffing levels, shift coverage, and scheduled vs. actual hours worked",
|
||||
"Order processing time logs and bottleneck analysis",
|
||||
"Workload distribution metrics across staff"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||||
|
||||
### Case 3 — Different Causes, Different Domain
|
||||
|
||||
**Problem:** "Website sales have fallen sharply over the last month."
|
||||
|
||||
**Hypothesis A:** "The fall may be caused by a recent increase in product prices."
|
||||
|
||||
**Hypothesis B:** "The fall may be caused by a technical checkout problem."
|
||||
|
||||
**Fixed human reference:** `sameEvidenceNeeded: false`
|
||||
|
||||
Expected evidence distinction:
|
||||
|
||||
- A should involve pricing/conversion/customer-response evidence;
|
||||
- B should involve checkout errors/technical telemetry/funnel evidence.
|
||||
|
||||
Purpose: check the pattern outside delivery operations.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"sameEvidenceNeeded": false,
|
||||
"evidenceForA": [
|
||||
"Sales volume and revenue data correlated with specific price change dates",
|
||||
"Customer feedback or survey responses regarding price sensitivity",
|
||||
"Competitor pricing data for similar products",
|
||||
"Cart abandonment rates specifically at the pricing/price review stage"
|
||||
],
|
||||
"evidenceForB": [
|
||||
"Server error logs and monitoring metrics from the checkout subsystem",
|
||||
"Checkout funnel analytics identifying specific step drop-offs",
|
||||
"Categorization of customer support tickets for technical/errors reports",
|
||||
"Cross-browser/device compatibility testing results and page load latency data during checkout"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||||
|
||||
### Evidence-Need-Correct Count
|
||||
|
||||
**3 of 3** cases classified correctly.
|
||||
|
||||
### Evidence-Need-Failed Count
|
||||
|
||||
**0 of 3** cases classified incorrectly.
|
||||
|
||||
### Question Answers
|
||||
|
||||
1. Did Case 1 correctly distinguish staff-capacity evidence from supplier evidence? Yes — model returned `sameEvidenceNeeded: false` with materially different evidence lists (staffing/scheduling/throughput vs. supplier records/SLA/historical reliability).
|
||||
2. Did Case 2 correctly recognise materially the same evidence need? Yes — model returned `sameEvidenceNeeded: true`; both evidence lists share nearly identical topics (order volume baselines, staffing levels, shift coverage, processing time logs).
|
||||
3. Did Case 3 distinguish pricing evidence from technical checkout evidence? Yes — model returned `sameEvidenceNeeded: false` with clearly distinct evidence for each hypothesis.
|
||||
4. Did the model collapse same diagnostic goal into same evidence need in any case? No. In all three cases where the correct reference was `false`, the model correctly returned `false`. In the only case where the correct reference was `true`, it correctly returned `true`.
|
||||
5. Did it invent evidence unrelated to the hypotheses? No. All evidence items are materially relevant to their respective hypotheses.
|
||||
6. Did it choose a winner? No. No preferred hypothesis, scoring, or preference language in any output.
|
||||
7. Did it generate a next question? No. No question generation in any output.
|
||||
|
||||
### Comparison with Experiment 54N
|
||||
|
||||
In 54N (abstraction level: "does the disagreement change what information needs to be established next?"), Case 3 — identical problem and hypotheses to this experiment's Case 1 — failed. The model returned `changesInformationNeededNext: false`, collapsing the two competing causes into one investigation direction ("verifying order timelines").
|
||||
|
||||
In 54O (abstraction level: "what evidence is needed to investigate each hypothesis?"), the same hypotheses now correctly return `sameEvidenceNeeded: false` with distinct evidence lists.
|
||||
|
||||
This suggests the 54N failure was partly caused by the abstraction level of the consequence question rather than an inability to understand the evidence difference. When asked directly about evidence, the model distinguished the competing causes cleanly.
|
||||
|
||||
However, the 54N result still stands as a candidate failure pattern in practice: if the engine asks "does this disagreement change what information needs to be established next?" (rather than asking for evidence comparison), it may still collapse the hypotheses. The question is whether that abstraction level is what the engine actually uses downstream.
|
||||
|
||||
### Inference Timing
|
||||
|
||||
- Number of live calls: **3**
|
||||
- Total time: **62,360ms (~62s)**
|
||||
- Average: **20,786.76ms per call**
|
||||
- Fastest: **13,755.44ms (Case 3)**
|
||||
- Slowest: **24,818.00ms (Case 1)**
|
||||
|
||||
### Questionable or Unsupported Findings
|
||||
|
||||
The 3/3 pass rate is strong but comes from only three cases. The test deliberately includes the exact failure case from 54N plus a same-evidence control and a different-domain cross-check. While this pattern is promising, a broader sample of competing-causes scenarios would be needed to confirm generalisation. It is also worth testing whether paraphrased variants of Case 3 still return `sameEvidenceNeeded: false`.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**The model distinguished same-goal hypotheses by their evidence needs in all tested cases.**
|
||||
|
||||
All three classifications matched fixed human references. The model returned the correct boolean for same-evidence and different-evidence cases, identified materially appropriate evidence for each hypothesis, and avoided winner selection and question generation in every case. This result strengthens confidence that the underlying semantic understanding of evidence differences exists — the 54N failure may have been an artifact of asking at a higher abstraction level (information need change) rather than at the direct evidence level.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
3 of 3 evidence-need classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent and materially distinct evidence lists for Case 1 and Case 3, and appropriately overlapping lists for Case 2.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Experiment 54N showed that asking "does this disagreement change what information needs to be established next?" fails on competing causal explanations sharing a diagnostic purpose (Case 3). Experiment 54O shows that asking directly "do these hypotheses require the same evidence?" succeeds on the same hypotheses plus two additional cases. The progression from consequence detection (54N) → direct evidence comparison (54O) reveals that the capability may be present at one abstraction level but lost at another.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — Experiment 54N wording corrected (blind spot → candidate failure pattern, narrowest gap → specific gap observed); Experiment 54O section appended;
|
||||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54O findings.
|
||||
|
||||
### Confirmation: Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54N)
|
||||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54N)
|
||||
|
||||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ sameEvidenceNeeded, evidenceForA, evidenceForB }`) is the experiment-only shape.
|
||||
|
||||
### Confirmation: No Evidence-Need Logic Entered Active Runtime
|
||||
|
||||
All inference calls were made exclusively within test code via `callEvidenceModel()`. No evidence-need logic was integrated into any production module. No runtime code changed.
|
||||
|
||||
### Confirmation: Active Engine and UI Remained Unchanged
|
||||
|
||||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
|
||||
@@ -0,0 +1,231 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { config } from "dotenv";
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
config({ path: path.resolve(__dirname, "../../.env.local") });
|
||||
|
||||
const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
|
||||
const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
|
||||
|
||||
if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
|
||||
throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
|
||||
}
|
||||
|
||||
/**
|
||||
* Make one live Ollama chat call for evidence-need comparison.
|
||||
*/
|
||||
async function callEvidenceModel(problem, hypothesisA, hypothesisB) {
|
||||
const instruction = `Compare the evidence needed to investigate the two hypotheses. Return sameEvidenceNeeded: true only when materially the same evidence would test both hypotheses. Return false when each hypothesis requires meaningfully different evidence, even if both are trying to explain the same overall problem. List the main evidence needed for each hypothesis. Do not decide which hypothesis is correct and do not generate questions.
|
||||
|
||||
Return valid JSON only in this shape:
|
||||
{
|
||||
"sameEvidenceNeeded": true | false,
|
||||
"evidenceForA": ["..."],
|
||||
"evidenceForB": ["..."]
|
||||
}`;
|
||||
|
||||
const messages = [
|
||||
{ role: "system", content: instruction.trim() },
|
||||
{
|
||||
role: "user",
|
||||
content: `Problem: ${JSON.stringify(problem)}
|
||||
|
||||
Hypothesis A: ${JSON.stringify(hypothesisA)}
|
||||
|
||||
Hypothesis B: ${JSON.stringify(hypothesisB)}`,
|
||||
},
|
||||
];
|
||||
|
||||
const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: OLLAMA_MODEL,
|
||||
messages,
|
||||
format: "json",
|
||||
stream: false,
|
||||
}),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
|
||||
}
|
||||
|
||||
const data = await res.json();
|
||||
const rawContent = data.message?.content ?? "";
|
||||
const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
|
||||
|
||||
return JSON.parse(cleaned.trim());
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Fixed human-reference ground truth (pre-written)
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
id: "Case 1 — Delivery Delay / Different Evidence",
|
||||
problem: "Orders are arriving late and customers have started complaining.",
|
||||
hypothesisA: "Delivery delays are being caused by insufficient staff capacity.",
|
||||
hypothesisB: "Delivery delays are being caused by unreliable supplier lead times.",
|
||||
reference: { sameEvidenceNeeded: false },
|
||||
},
|
||||
{
|
||||
id: "Case 2 — Same Cause, Paraphrased / Same Evidence",
|
||||
problem: "Orders are arriving late and customers have started complaining.",
|
||||
hypothesisA: "The team may not have enough capacity to process orders on time.",
|
||||
hypothesisB: "Insufficient staff capacity may be causing the order delays.",
|
||||
reference: { sameEvidenceNeeded: true },
|
||||
},
|
||||
{
|
||||
id: "Case 3 — Different Causes, Different Domain",
|
||||
problem: "Website sales have fallen sharply over the last month.",
|
||||
hypothesisA: "The fall may be caused by a recent increase in product prices.",
|
||||
hypothesisB: "The fall may be caused by a technical checkout problem.",
|
||||
reference: { sameEvidenceNeeded: false },
|
||||
},
|
||||
];
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Semantic evaluation against fixed human references
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
function evaluateEvidenceNeed(modelResult, reference) {
|
||||
const result = modelResult;
|
||||
|
||||
let issues = [];
|
||||
let notes = [];
|
||||
|
||||
// Boolean must match reference
|
||||
if (result.sameEvidenceNeeded !== reference.sameEvidenceNeeded) {
|
||||
issues.push("boolean_mismatch: model evidence-need assessment does not match fixed human reference");
|
||||
}
|
||||
|
||||
// Must provide evidence lists
|
||||
if (!Array.isArray(result.evidenceForA) || result.evidenceForA.length === 0) {
|
||||
issues.push("missing_evidence_A: evidenceForA is empty or not an array");
|
||||
}
|
||||
if (!Array.isArray(result.evidenceForB) || result.evidenceForB.length === 0) {
|
||||
issues.push("missing_evidence_B: evidenceForB is empty or not an array");
|
||||
}
|
||||
|
||||
// Check for invariant violations (no winner, no next question)
|
||||
const allText = JSON.stringify(result).toLowerCase();
|
||||
const hasWinnerSelection = allText.includes("preferred") ||
|
||||
allText.includes("more likely") ||
|
||||
allText.includes("should go with");
|
||||
if (hasWinnerSelection) {
|
||||
issues.push("invariant_failed: model appears to have chosen a winning hypothesis");
|
||||
}
|
||||
|
||||
const hasNextQuestion = allText.includes("ask the user") ||
|
||||
allText.includes("ask next") ||
|
||||
allText.includes("question to ask");
|
||||
if (hasNextQuestion) {
|
||||
notes.push("caution: model generated a next-question suggestion alongside the evidence assessment");
|
||||
}
|
||||
|
||||
// Semantic check: if sameEvidenceNeeded=false, evidenceForA and evidenceForB should be materially different
|
||||
if (reference.sameEvidenceNeeded === false && Array.isArray(result.evidenceForA) && Array.isArray(result.evidenceForB)) {
|
||||
const aText = result.evidenceForA.join(" ").toLowerCase();
|
||||
const bText = result.evidenceForB.join(" ").toLowerCase();
|
||||
if (aText === bText) {
|
||||
notes.push("caution: evidenceForA and evidenceForB are identical despite sameEvidenceNeeded=false");
|
||||
}
|
||||
}
|
||||
|
||||
if (issues.length === 0) return "evidence_need_correct";
|
||||
return "evidence_need_failed";
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Describe the experiment as a single test suite
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("Experiment 54O — Evidence Needs Across Competing Hypotheses (test-only)", () => {
|
||||
const results = [];
|
||||
const timings = [];
|
||||
|
||||
for (const testCase of CASES) {
|
||||
it(`${testCase.id} — evidence need comparison`, async () => {
|
||||
const t0 = performance.now();
|
||||
const result = await callEvidenceModel(
|
||||
testCase.problem,
|
||||
testCase.hypothesisA,
|
||||
testCase.hypothesisB
|
||||
);
|
||||
const elapsed = performance.now() - t0;
|
||||
timings.push(elapsed);
|
||||
|
||||
expect(result).toHaveProperty("sameEvidenceNeeded");
|
||||
expect(typeof result.sameEvidenceNeeded).toBe("boolean");
|
||||
expect(Array.isArray(result.evidenceForA)).toBe(true);
|
||||
expect(Array.isArray(result.evidenceForB)).toBe(true);
|
||||
|
||||
const classification = evaluateEvidenceNeed(result, testCase.reference);
|
||||
|
||||
results.push({
|
||||
id: testCase.id,
|
||||
problem: testCase.problem,
|
||||
hypothesisA: testCase.hypothesisA,
|
||||
hypothesisB: testCase.hypothesisB,
|
||||
reference: testCase.reference,
|
||||
modelResult: result,
|
||||
classification: classification,
|
||||
timingMs: Number(elapsed.toFixed(2)),
|
||||
});
|
||||
|
||||
console.log(`\n=== ${testCase.id} ===`);
|
||||
console.log(`Model output:`);
|
||||
console.log(` sameEvidenceNeeded:`, result.sameEvidenceNeeded);
|
||||
console.log(` evidenceForA:`, result.evidenceForA);
|
||||
console.log(` evidenceForB:`, result.evidenceForB);
|
||||
console.log(`Reference:`, testCase.reference);
|
||||
console.log(`Classification: ${classification}`);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
it("54O — summary and required questions", () => {
|
||||
const correct = results.filter((r) => r.classification === "evidence_need_correct").length;
|
||||
const failed = results.filter((r) => r.classification === "evidence_need_failed").length;
|
||||
const total = timings.reduce((a, b) => a + b, 0);
|
||||
const avg = total / timings.length;
|
||||
const fastest = Math.min(...timings);
|
||||
const slowest = Math.max(...timings);
|
||||
|
||||
console.log("\n=== Experiment 54O Summary ===");
|
||||
console.log(`Cases: ${results.length}`);
|
||||
console.log(`Evidence-need-correct: ${correct}, Evidence-need-failed: ${failed}`);
|
||||
|
||||
results.forEach((r) => {
|
||||
const expectedLabel = r.reference.sameEvidenceNeeded ? "same" : "different";
|
||||
const resultLabel = r.modelResult.sameEvidenceNeeded ? "same" : "different";
|
||||
const match = r.classification === "evidence_need_correct" ? "✓" : "✗";
|
||||
console.log(` ${match} ${r.id}: expected ${expectedLabel}, model said ${resultLabel}`);
|
||||
});
|
||||
|
||||
// Check for invariant violations across all results
|
||||
let winnerChosen = false;
|
||||
let questionGenerated = false;
|
||||
for (const r of results) {
|
||||
const text = JSON.stringify(r.modelResult).toLowerCase();
|
||||
if (text.includes("preferred") || text.includes("more likely")) {
|
||||
winnerChosen = true;
|
||||
}
|
||||
if (text.includes("ask the user") || text.includes("question to ask")) {
|
||||
questionGenerated = true;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Total inference time: ${total.toFixed(2)}ms`);
|
||||
console.log(`Average: ${avg.toFixed(2)}ms, Fastest: ${fastest.toFixed(2)}ms, Slowest: ${slowest.toFixed(2)}ms`);
|
||||
console.log(`Winner chosen by model: ${winnerChosen ? "yes" : "no"}`);
|
||||
console.log(`Next question generated: ${questionGenerated ? "yes" : "no"}`);
|
||||
|
||||
expect(false).toBe(false); // Q1-Q10 addressed in report
|
||||
expect(results.length).toBe(3);
|
||||
});
|
||||
}, 600000);
|
||||
Reference in New Issue
Block a user