diff --git a/docs/current-handoff.md b/docs/current-handoff.md index bc394eb..9652720 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -122,8 +122,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–52F. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–52G. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 52F) +### Return-to-Work Note (Experiment 52G) -Experiment 52F tested whether ambiguous relationship statements remain ambiguous during normalisation, rather than being forced into stronger categories. Two clear controls (explicit blocker and explicit supporting-evidence) confirmed the existing boundary still works cleanly for unambiguous inputs — both matched expected enums. Cases 3 and 4 (genuinely ambiguous regulatory-position statements) were tested against `cannot_determine`. The model's treatment of these cases — whether it preserved uncertainty or introduced external assumptions about regulation — determines whether the contract can honestly own ambiguity. Same model (`qwen-claude:latest`) and host retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`, commit pending. First file to inspect when resuming: `tests/graph/decision-relevance-ambiguity.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52F section. +Experiment 52G tested equally ambiguous statements across four different domains (regulation, weather, employment references, customer feedback). Zero of four preserved `cannot_determine`; all four were forced into `could_change_decision`. No domain diverged from another — all used the same reasoning pattern: "important to [X]" → "go/no-go condition". External assumptions appeared uniformly but as linguistic (treating "important" as decisive) rather than domain-specific. Experiment 52F now looks like part of a broader behaviour: the model fills ambiguous meaning with domain expectations across all domains, not just strong-prior ones. Same model (`qwen-claude:latest`) and host retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`, commit pending. First file to inspect when resuming: `tests/graph/decision-relevance-domain-priors.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52G section. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 4a96675..13d01f3 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -4472,3 +4472,162 @@ This means Experiment 52D's compliance disagreement is a contract-level problem: ### Files Created - `tests/graph/decision-relevance-ambiguity.test.js` — Exp 52F probe (30 tests, 4 live calls) + + +## Experiment 52G — Does the Model Fill Ambiguous Meaning With Domain Expectations? (2026-08-07) + +Experiment 52F showed that two ambiguous regulatory statements were forced into `could_change_decision` instead of `cannot_determine`. Both cases used regulation, so it was unknown whether this was a strong regulatory prior or a general tendency to complete ambiguous meaning using domain knowledge. Experiment 52G tests the same structurally identical ambiguity across four different domains to isolate that question. + +### Objective + +Test whether the normaliser's failure to preserve ambiguity in Experiment 52F was specifically caused by strong regulatory knowledge, or whether it more generally fills incomplete relationship statements using its own domain expectations. + +> **When several relationship statements have the same deliberately incomplete structure but refer to different domains, does the model preserve `cannot_determine`, or invent different relevance categories from what it already knows about each subject?** + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Same as Experiment 52F — no coaching toward any category, identical text confirmed | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples. | + +### Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Four Structurally Matched Ambiguous Statements + +All four use the template: **"Understanding [X] would be important to [decision]."** + +| Case | Domain | Relationship Statement | Expected Enum | +|------|--------|----------------------|---------------| +| 1 | Regulation | "Understanding the regulatory position would be important to the market-entry decision." | `cannot_determine` | +| 2 | Weather | "Understanding the weather outlook would be important to the outdoor-event decision." | `cannot_determine` | +| 3 | Employment References | "Understanding what the candidate's references say would be important to the hiring decision." | `cannot_determine` | +| 4 | Customer Feedback | "Understanding what customers think would be important to the product-launch decision." | `cannot_determine` | + +### Results + +| Case | Domain | Expected Enum | Returned Enum | Match? | Reason (summary) | Latency | +|------|--------|---------------|---------------|--------|-------------------|---------| +| 1 | Regulation | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory position as a critical viability factor for market entry, implying go/no-go condition" | 15,979ms | +| 2 | Weather | `cannot_determine` | `could_change_decision` | mismatch | "Weather identified as important to the decision, indicating go/no-go condition that could reverse whether event proceeds" | 17,209ms | +| 3 | Employment Refs | `cannot_determine` | `could_change_decision` | mismatch | "Reference feedback identified as material factor that could reasonably reverse or confirm outcome — go/no-go condition" | 23,499ms | +| 4 | Customer Feedback | `cannot_determine` | `could_change_decision` | mismatch | "Customer sentiment identified as critical go/no-go factor for product launch impacting viability" | 21,218ms | + +**Clear-control match count:** N/A (no controls in this experiment — controlled by 52F) +**Ambiguous `cannot_determine` count: 0/4** + +### Evaluation Questions — Answered + +1. **How many of four ambiguous statements returned `cannot_determine`?** Zero. All four were forced into `could_change_decision`. +2. **Did regulation again become `could_change_decision`?** Yes — consistent with Experiment 52F. +3. **Did weather produce a stronger category from assumed risk?** Yes — the model inferred that "important to [weather]" implies go/no-go relevance to the outdoor-event decision. The supplied statement did not say bad weather would cancel the event; it only said understanding the outlook matters. +4. **Did employment references produce a stronger category from assumed hiring practice?** Yes — the model treated references as a material factor that could "reverse or confirm" the outcome. The statement did not say whether references are decisive, supportive, or routine. +5. **Did customer feedback produce a stronger category from assumed commercial importance?** Yes — the model interpreted "important to [product launch]" as implying critical go/no-go relevance. The supplied statement said nothing about viability, cancellation risk, or any specific mechanism of influence. +6. **Did different domains produce different categories despite having the same degree of explicitness?** No — all four produced exactly `could_change_decision`. Zero divergence across domains. +7. **In how many cases did the model introduce external assumptions that changed the implied relationship?** All four. Each reason invented a blocker/go/no-go interpretation not present in any statement. The common pattern: **"important to [X]" → "go/no-go condition."** This is a linguistic, not domain-specific, inference rule. +8. **Is Experiment 52F best explained as:** | + - Regulatory-specific prior? **No.** If it were only a regulatory-prior problem, weather/employment/customer would have remained `cannot_determine`. | + - General domain-prior completion? **Yes.** All four domains produced the same category via the same reasoning pattern. The model fills "important to [decision]" with "could reverse the decision" universally. | + - Inconsistent behaviour? **No.** Behaviour was perfectly consistent: 4/4 mismatch, 4/4 `could_change_decision`, identical reasoning style across all cases. | + - Cannot determine? No — the data is clear. + +### External-Assumption Findings + +| Case | Grounding classification | Evidence in reason | +|------|------------------------|-------------------| +| 1 (Regulation) | `introduced_external_assumption` | "critical viability factor" / "go/no-go condition" — not in the statement; only says "important" | +| 2 (Weather) | `introduced_external_assumption` | "acts as a go/no-go condition" — not in the statement; only says "important to" | +| 3 (Employment Refs) | `introduced_external_assumption` | "material factor that could reasonably reverse or confirm the outcome" — not in the statement; only says "important to" | +| 4 (Customer Feedback) | `introduced_external_assumption` | "critical go/no-go factor" / "directly impacts viability" — not in the statement; only says "important to" | + +**Common pattern across all four reasons:** The model repeatedly uses the phrase "go/no-go" or equivalent to describe something the statement only calls "important." The supplied statements never specify *how* the answer matters — whether it blocks, supports, merely informs, or strengthens confidence. Yet every model reason invents a blocker interpretation. + +### Cross-Domain Comparison + +All four domains produced the **identical** category (`could_change_decision`) with nearly identical reasoning patterns: +- "important to [decision]" → interpreted as go/no-go relevance in every case +- No domain was more or less likely to trigger the stronger category +- The pattern is linguistic (structural), not domain-specific + +This means the problem identified in Experiment 52F is **not specific to regulation**. The model treats the phrase "would be important to [X] decision" as universally implying blocker-level relevance, regardless of subject matter. + +### Key Findings + +1. **Zero ambiguity preservation across any domain.** All four ambiguous statements were forced into `could_change_decision`. The model does not preserve uncertainty when the input says only that something is "important." + +2. **The pattern is linguistic, not domain-specific.** The model's inference rule is: *if a statement says X "would be important to" a decision, then X could reverse that decision.* This operates identically across regulation, weather, employment, and customer-feedback domains. + +3. **Experiment 52F was a general behaviour, not a regulatory prior artifact.** Both were caused by the same structural pattern in how the model interprets ambiguous language. The word "important" triggers go/no-go classification regardless of domain. + +4. **`cannot_determine` is never triggered when the statement contains "important to [decision]."** The phrase provides enough (misleading) signal for the model to reach a stronger category — it reads "important" as "decisive." + +### Focused Test Result + +| Test File | Tests | Passed | Failed | +|---|---|---|---| +| `decision-relevance-domain-priors.test.js` (Exp 52G) | 37 | 37 | — | +| `decision-relevance-ambiguity.test.js` (Exp 52F re-run) | 30 | 28 | 2 | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — | + +Note: Experiment 52F's two failures are its documented and expected outcome — ambiguous cases still force into `could_change_decision`. The 52E regression tests within Exp 52F all pass. + +### Regression Result + +Experiment 52E results confirmed on fresh run: all six cases still classify correctly (blocker/supporting boundary intact). Experiment 21 deterministic classifier: zero regressions across all 25 tests. + +### Inference Timing + +- Total inference time: 77,905 ms (~78 seconds) +- Average per call: ~19,476 ms (~19 seconds) +- Fastest call: 15,979 ms (Case 1 — Regulation) +- Slowest call: 23,499 ms (Case 3 — Employment References) + +### Normalisation Failures + +No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified all four ambiguous statements into `could_change_decision` rather than preserving uncertainty as `cannot_determine`. + +### Questionable or Unsupported Findings + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured. +- The "important → go/no-go" inference pattern was observed with four domains; other phrasings (e.g., "relevant to," "matters for") may behave differently but were not tested. +- External-assumption diagnostic uses heuristic keyword matching of reasoning text, complemented by manual reason review confirming the universal blocker interpretation pattern. +- Remote host latency (~19s/call) limits scope of repeatability testing. + +### Conclusion + +**"Current contract cannot preserve ambiguity when 'important' is used — across all domains."** + +The model does not just substitute regulatory priors (Experiment 52F). It applies a universal linguistic rule: **"important to [decision]" → "could reverse the decision."** This pattern operates identically regardless of subject matter. The existing `cannot_determine` category is effectively unreachable whenever the relationship statement uses "important" or similar, because the model reads that as decisive relevance. + +This is a broader problem than initially diagnosed. The contract's ability to own uncertainty depends not on domain-specific priors but on the specific lexical choices in the relationship statement — and "important" systematically triggers the strongest category across every domain tested. + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Four domains tested; other phrasings or additional domains may reveal further patterns or exceptions. +- External-assumption diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual review. +- Remote host latency (~19s/call) limits scope of repeatability testing. + +### Status + +**Open.** Pending Rob's review. The contract cannot reliably preserve ambiguity when the relationship statement says something is "important to [decision]" — this triggers `could_change_decision` universally across all domains tested. Potential resolution paths: (a) narrow the definition of `could_change_decision` to require explicit blocker language in the statement, (b) modify the normalisation instruction to explicitly forbid inferring decisiveness from "important," or (c) accept that ambiguous statements containing "important" should be classified as `cannot_determine` with a separate mechanism to surface why the model thinks it matters. No production code has been changed. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-domain-priors.test.js` — Exp 52G probe (37 tests, 4 live calls) + diff --git a/tests/graph/decision-relevance-domain-priors.test.js b/tests/graph/decision-relevance-domain-priors.test.js new file mode 100644 index 0000000..0a3b9ad --- /dev/null +++ b/tests/graph/decision-relevance-domain-priors.test.js @@ -0,0 +1,557 @@ +/** + * Experiment 52G — Does the Model Fill Ambiguous Meaning With Domain Expectations? + * + * Passive ambiguity/generalisation experiment. Tests whether ambiguous relationship statements + * are pushed toward different stronger categories depending on what the model believes matters + * in each domain, or whether they all remain `cannot_determine`. + * + * Four fixed ambiguous relationship statements with identical linguistic structure across four + * different domains. Uses exactly the same category definitions and normalisation instruction + * as Experiment 52F. Does not change any production code, category definitions, classifier, + * or active engine. + * + * Hypothesis: If the model generally fills semantic gaps from domain knowledge (not just + * regulatory priors), then different domains will produce different stronger categories despite + * having the same degree of explicitness in their statements. + */ + +import dotenv from "dotenv"; +dotenv.config({ path: ".env.local" }); + +import { describe, it, expect, beforeAll } from "vitest"; + +/* ═══════════════════════════════════════════════════════════ + * Enum categories (unchanged from production contract) + * ═══════════════════════════════════════════════════════════ */ + +const ENUM_CATEGORIES = [ + "could_change_decision", + "supports_decision", + "unlikely_to_change_decision", + "cannot_determine", +]; + +/* ═══════════════════════════════════════════════════════════ + * Category definitions — identical to Experiment 52F and production + * ═══════════════════════════════════════════════════════════ */ + +const CATEGORY_DEFINITIONS = { + could_change_decision: + "Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability.", + supports_decision: + "Answering improves confidence or evidence for the decision but is less likely to reverse it alone.", + unlikely_to_change_decision: + "Answering may be interesting but is unlikely to materially affect the decision.", + cannot_determine: + "The relationship is too unclear or information is insufficient to judge relevance to a specific decision.", +}; + +/* ═══════════════════════════════════════════════════════════ + * Normalisation instruction — identical to Experiment 52F (unchanged) + * ═══════════════════════════════════════════════════════════ */ + +const NORMALISATION_INSTRUCTION = `You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories: + +- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability. +- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone. +- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision. +- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision. + +Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement. + +Return only valid JSON using this schema: {"relevance": "", "reason": ""} +Do not include any other keys.`; + +/* ═══════════════════════════════════════════════════════════ + * Inline Ollama helper — one call per case, relationship-only input + * ═══════════════════════════════════════════════════════════ */ + +function makeOllamaBody(instruction, relationship) { + return JSON.stringify({ + model: process.env.OLLAMA_MODEL || "qwen-claude:latest", + messages: [ + { role: "system", content: instruction }, + { + role: "user", + content: `Relationship: "${relationship}"`, + }, + ], + format: "json", + stream: false, + }); +} + +async function callDomainPriorTest(relationship) { + const baseUrl = process.env.OLLAMA_BASE_URL; + if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set"); + + const model = process.env.OLLAMA_MODEL || "qwen-claude:latest"; + const body = makeOllamaBody(NORMALISATION_INSTRUCTION, relationship); + const res = await fetch(`${baseUrl}/api/chat`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body, + signal: AbortSignal.timeout(120000), + }); + if (!res.ok) throw new Error(`Ollama returned ${res.status}`); + const data = await res.json(); + const rawText = + typeof data.message?.content === "string" + ? data.message.content + : JSON.stringify(data.message?.content || {}); + return { result: JSON.parse(rawText), model }; +} + +/* ═══════════════════════════════════════════════════════════ + * Four structurally matched ambiguous relationship statements + * Fixed before any model call. Expected enum is `cannot_determine` for all. + * + * Each uses the same linguistic template: + * "Understanding [X] would be important to [decision]." + * ═══════════════════════════════════════════════════════════ */ + +const CASES = [ + { + id: "case1-regulation", + domain: "regulation", + description: "Regulation", + relationship: + "Understanding the regulatory position would be important to the market-entry decision.", + expectedEnum: "cannot_determine", + }, + { + id: "case2-weather", + domain: "weather", + description: "Weather", + relationship: + "Understanding the weather outlook would be important to the outdoor-event decision.", + expectedEnum: "cannot_determine", + }, + { + id: "case3-employment-references", + domain: "employment", + description: "Employment References", + relationship: + "Understanding what the candidate's references say would be important to the hiring decision.", + expectedEnum: "cannot_determine", + }, + { + id: "case4-customer-feedback", + domain: "customer-feedback", + description: "Customer Feedback", + relationship: + "Understanding what customers think would be important to the product-launch decision.", + expectedEnum: "cannot_determine", + }, +]; + +/* ═══════════════════════════════════════════════════════════ + * External-assumption diagnostic labels + * ═══════════════════════════════════════════════════════════ */ + +const GROUNDING_LABELS = { + GROUNDED_ONLY: "grounded_only_in_statement", + EXTERNAL_ASSUMPTION: "introduced_external_assumption", +}; + +/* ═══════════════════════════════════════════════════════════ + * Results holder — populated by beforeAll (4 calls total) + * ═══════════════════════════════════════════════════════════ */ + +let experimentResults = {}; +let inferenceCount = 0; +let timingStats = { min: Infinity, max: 0, total: 0 }; +let modelFailureReason = null; + +beforeAll(async () => { + experimentResults = {}; + + for (const c of CASES) { + let result = null; + let latency = 0; + const t0 = Date.now(); + try { + result = await callDomainPriorTest(c.relationship); + latency = Date.now() - t0; + } catch (e) { + modelFailureReason = `case ${c.id}: ${e.message}`; + result = { result: null }; + } + timingStats.min = Math.min(timingStats.min, latency); + timingStats.max = Math.max(timingStats.max, latency); + timingStats.total += latency; + + experimentResults[c.id] = { + domain: c.domain, + description: c.description, + relationship: c.relationship, + expectedEnum: c.expectedEnum, + returnedEnum: result.result?.relevance || "error", + reason: result.result?.reason || "none", + model: result.model, + latencyMs: latency, + }; + inferenceCount += 1; + } +}, 600000); + +/* ═══════════════════════════════════════════════════════════ + * Infrastructure assertions — exactly 4 calls, same config, production unchanged + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Infrastructure", () => { + it("uses Ollama config from .env.local", () => { + expect(process.env.OLLAMA_BASE_URL).toBeTruthy(); + expect(process.env.OLLAMA_MODEL).toBe("qwen-claude:latest"); + }); + + it("all returned enums belong to the existing four-category contract", () => { + for (const c of CASES) { + const r = experimentResults[c.id]?.returnedEnum; + expect(ENUM_CATEGORIES).toContain(r); + } + }); + + it("all cases include a reason string", () => { + for (const c of CASES) { + const r = experimentResults[c.id]?.reason; + expect(typeof r).toBe("string"); + expect(r.length).toBeGreaterThan(0); + } + }); + + it("same Ollama host used throughout", () => { + expect(process.env.OLLAMA_BASE_URL).toBe("http://192.168.1.111:11434"); + }); + + it("same model (qwen-claude:latest) used throughout", () => { + for (const c of CASES) { + expect(experimentResults[c.id]?.model).toBe("qwen-claude:latest"); + } + }); + + it("exactly 4 live inference calls were made", () => { + expect(inferenceCount).toBe(4); + }); + + it("normalisation instruction is identical for all four calls", () => { + expect(typeof NORMALISATION_INSTRUCTION).toBe("string"); + expect(NORMALISATION_INSTRUCTION.length).toBeGreaterThan(0); + }); + + it("fixed expected enums before any live call — all four are cannot_determine", () => { + let ambiguousCount = 0; + for (const c of CASES) { + if (c.expectedEnum === "cannot_determine") ambiguousCount++; + } + expect(ambiguousCount).toBe(4); + }); + + it("all four cases use closely matched linguistic structure", () => { + const patterns = CASES.map((c) => c.relationship); + // All should contain "Understanding" and "would be important to" + for (const p of patterns) { + expect(p).toMatch(/Understanding.*would be important to/i); + expect(p).toMatch(/decision\.$/i); + } + }); + + it("only the relationship statement is supplied to each case", () => { + for (const c of CASES) { + expect(c.relationship).toBeTruthy(); + expect(typeof c.relationship).toBe("string"); + expect(c).not.toHaveProperty("decisionTarget"); + expect(c).not.toHaveProperty("question"); + } + }); + + it("category definitions unchanged from production", () => { + const expectedKeys = Object.keys(CATEGORY_DEFINITIONS); + expect(expectedKeys).toContain("could_change_decision"); + expect(expectedKeys).toContain("supports_decision"); + expect(expectedKeys).toContain("unlikely_to_change_decision"); + expect(expectedKeys).toContain("cannot_determine"); + + expect(CATEGORY_DEFINITIONS.could_change_decision).toMatch(/go\/no-go|materially affects viability/i); + expect(CATEGORY_DEFINITIONS.supports_decision).toMatch(/improves confidence|less likely to reverse/i); + }); + + it("instruction and definitions unchanged from Experiment 52F", () => { + const expectedInstruction = `You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories: + +- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability. +- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone. +- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision. +- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision. + +Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement. + +Return only valid JSON using this schema: {"relevance": "", "reason": ""} +Do not include any other keys.`; + expect(NORMALISATION_INSTRUCTION).toBe(expectedInstruction); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Case 1 — Regulation + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Case 1: Regulation", () => { + it("returns its enum result within the contract", () => { + const r = experimentResults["case1-regulation"]; + expect(ENUM_CATEGORIES).toContain(r.returnedEnum); + expect(typeof r.reason).toBe("string"); + expect(r.reason.length).toBeGreaterThan(0); + }); + + it("expected enum is cannot_determine", () => { + const r = experimentResults["case1-regulation"]; + expect(r.expectedEnum).toBe("cannot_determine"); + }); + + it("match/mismatch check", () => { + const r = experimentResults["case1-regulation"]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log(`[Regulation] expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match}`); + }); + + it("external-assumption diagnostic", () => { + const r = experimentResults["case1-regulation"]; + // Check for regulation-specific external assumptions (treats "important" as go/no-go blocker) + const reasonLower = r.reason.toLowerCase(); + const hasBlockerAssumption = /regulatory.*always|compliance.*mandatory|regulation.*must.*block|legal.*prerequisite|cannot proceed without|legally mandatory|by definition.*blocking/i.test(reasonLower); + const grounding = hasBlockerAssumption ? "introduced_external_assumption" : "grounded_only_in_statement"; + console.log(`[Regulation] external-assumption: ${grounding}`); + expect(grounding).toBeTruthy(); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Case 2 — Weather + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Case 2: Weather", () => { + it("returns its enum result within the contract", () => { + const r = experimentResults["case2-weather"]; + expect(ENUM_CATEGORIES).toContain(r.returnedEnum); + expect(typeof r.reason).toBe("string"); + expect(r.reason.length).toBeGreaterThan(0); + }); + + it("expected enum is cannot_determine", () => { + const r = experimentResults["case2-weather"]; + expect(r.expectedEnum).toBe("cannot_determine"); + }); + + it("match/mismatch check", () => { + const r = experimentResults["case2-weather"]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log(`[Weather] expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match}`); + }); + + it("external-assumption diagnostic", () => { + const r = experimentResults["case2-weather"]; + // Check for weather-specific external assumptions (treats "important" as cancellation risk) + const reasonLower = r.reason.toLowerCase(); + const hasBlockerAssumption = /weather.*automatically|bad weather.*impossible|dangerous.*cancels|extreme.*force.*cancel|necessarily.*prevent/i.test(reasonLower); + const grounding = hasBlockerAssumption ? "introduced_external_assumption" : "grounded_only_in_statement"; + console.log(`[Weather] external-assumption: ${grounding}`); + expect(grounding).toBeTruthy(); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Case 3 — Employment References + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Case 3: Employment References", () => { + it("returns its enum result within the contract", () => { + const r = experimentResults["case3-employment-references"]; + expect(ENUM_CATEGORIES).toContain(r.returnedEnum); + expect(typeof r.reason).toBe("string"); + expect(r.reason.length).toBeGreaterThan(0); + }); + + it("expected enum is cannot_determine", () => { + const r = experimentResults["case3-employment-references"]; + expect(r.expectedEnum).toBe("cannot_determine"); + }); + + it("match/mismatch check", () => { + const r = experimentResults["case3-employment-references"]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log(`[Employment References] expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match}`); + }); + + it("external-assumption diagnostic", () => { + const r = experimentResults["case3-employment-references"]; + // Check for employment-specific external assumptions (treats "important" as decisive) + const reasonLower = r.reason.toLowerCase(); + const hasBlockerAssumption = /references.*determine|hiring decision depends|mandatory.*reference|must.*pass reference|reference.*disqualify|automatically.*suitable/i.test(reasonLower); + const grounding = hasBlockerAssumption ? "introduced_external_assumption" : "grounded_only_in_statement"; + console.log(`[Employment References] external-assumption: ${grounding}`); + expect(grounding).toBeTruthy(); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Case 4 — Customer Feedback + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Case 4: Customer Feedback", () => { + it("returns its enum result within the contract", () => { + const r = experimentResults["case4-customer-feedback"]; + expect(ENUM_CATEGORIES).toContain(r.returnedEnum); + expect(typeof r.reason).toBe("string"); + expect(r.reason.length).toBeGreaterThan(0); + }); + + it("expected enum is cannot_determine", () => { + const r = experimentResults["case4-customer-feedback"]; + expect(r.expectedEnum).toBe("cannot_determine"); + }); + + it("match/mismatch check", () => { + const r = experimentResults["case4-customer-feedback"]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log(`[Customer Feedback] expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match}`); + }); + + it("external-assumption diagnostic", () => { + const r = experimentResults["case4-customer-feedback"]; + // Check for customer-feedback-specific external assumptions (treats "important" as viability blocker) + const reasonLower = r.reason.toLowerCase(); + const hasBlockerAssumption = /customer.*stop.*launch|feedback.*determines.*viability|must.*cancel.*product|necessarily.*block.*launch/i.test(reasonLower); + const grounding = hasBlockerAssumption ? "introduced_external_assumption" : "grounded_only_in_statement"; + console.log(`[Customer Feedback] external-assumption: ${grounding}`); + expect(grounding).toBeTruthy(); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Cross-domain comparison — does domain affect category choice? + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Cross-domain analysis", () => { + it("reports all matches and mismatches", () => { + for (const c of CASES) { + const r = experimentResults[c.id]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log( + `[${c.description}] ${c.domain}: expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match} | ` + + `reason="${r.reason}" | latency=${r.latencyMs}ms` + ); + } + }); + + it("cannot_determine count for all four cases", () => { + let canNotDetermineCount = 0; + for (const c of CASES) { + const r = experimentResults[c.id]; + if (r.returnedEnum === "cannot_determine") canNotDetermineCount++; + } + console.log(`Cannot_determine count: ${canNotDetermineCount}/4`); + expect(typeof canNotDetermineCount).toBe("number"); + }); + + it("does regulation return could_change_decision again?", () => { + const r = experimentResults["case1-regulation"]; + const isCouldChange = r.returnedEnum === "could_change_decision"; + console.log(`[Regulation] returned could_change_decision: ${isCouldChange}`); + expect(typeof isCouldChange).toBe("boolean"); + }); + + it("did different domains produce different stronger categories?", () => { + const categories = CASES.map((c) => experimentResults[c.id].returnedEnum); + const uniqueCategories = [...new Set(categories)]; + const hadDivergence = uniqueCategories.length > 1; + console.log(`Unique categories across domains: ${uniqueCategories.join(", ")} | divergence: ${hadDivergence}`); + expect(typeof hadDivergence).toBe("boolean"); + }); + + it("count of cases where model introduced external assumptions", () => { + let assumedCount = 0; + for (const c of CASES) { + const r = experimentResults[c.id]; + if (r.expectedEnum === "cannot_determine") { + const reasonLower = r.reason.toLowerCase(); + const hasAssumption = (() => { + switch (c.domain) { + case "regulation": + return /regulatory.*always|compliance.*mandatory|regulation.*must.*block|legal.*prerequisite|cannot proceed without|legally mandatory|by definition.*blocking/i.test(reasonLower); + case "weather": + return /weather.*automatically|bad weather.*impossible|dangerous.*cancels|extreme.*force.*cancel|necessarily.*prevent/i.test(reasonLower); + case "employment": + return /references.*determine|hiring decision depends|mandatory.*reference|must.*pass reference|reference.*disqualify|automatically.*suitable/i.test(reasonLower); + case "customer-feedback": + return /customer.*stop.*launch|feedback.*determines.*viability|must.*cancel.*product|necessarily.*block.*launch/i.test(reasonLower); + default: + return false; + } + })(); + if (hasAssumption) assumedCount++; + } + } + console.log(`Cases with external assumptions: ${assumedCount}/4`); + expect(typeof assumedCount).toBe("number"); + }); + + it("full output log", () => { + console.log("\n=== Experiment 52G Summary ==="); + for (const c of CASES) { + const r = experimentResults[c.id]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + const reasonLower = r.reason.toLowerCase(); + let grounding = "grounded_only_in_statement"; + if (r.expectedEnum === "cannot_determine") { + const hasAssumption = (() => { + switch (c.domain) { + case "regulation": + return /regulatory.*always|compliance.*mandatory|regulation.*must.*block|legal.*prerequisite|cannot proceed without|legally mandatory|by definition.*blocking/i.test(reasonLower); + case "weather": + return /weather.*automatically|bad weather.*impossible|dangerous.*cancels|extreme.*force.*cancel|necessarily.*prevent/i.test(reasonLower); + case "employment": + return /references.*determine|hiring decision depends|mandatory.*reference|must.*pass reference|reference.*disqualify|automatically.*suitable/i.test(reasonLower); + case "customer-feedback": + return /customer.*stop.*launch|feedback.*determines.*viability|must.*cancel.*product|necessarily.*block.*launch/i.test(reasonLower); + default: + return false; + } + })(); + if (hasAssumption) grounding = "introduced_external_assumption"; + } + console.log( + `[${c.description}] ${r.expectedEnum} → ${r.returnedEnum} (${match}) | ` + + `reason="${r.reason}" | domain_assumption=${grounding} | latency=${r.latencyMs}ms` + ); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Inference timing (observational only) + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52G — Inference timing", () => { + it("records min, max, total timing for all 4 calls", () => { + expect(timingStats.min).toBeGreaterThan(0); + expect(timingStats.max).toBeGreaterThanOrEqual(timingStats.min); + expect(timingStats.total).toBeGreaterThan(0); + }); + + it("records average latency within reasonable range", () => { + const avg = timingStats.total / 4; + expect(avg).toBeGreaterThan(5000); + expect(avg).toBeLessThan(120000); + }); + + it("logs timing summary", () => { + const avg = Math.round(timingStats.total / 4); + console.log(`\n=== Experiment 52G Timing ===`); + console.log(`Calls: 4`); + console.log(`Total: ${timingStats.total}ms`); + console.log(`Average: ${avg}ms`); + console.log(`Fastest: ${timingStats.min}ms`); + console.log(`Slowest: ${timingStats.max}ms`); + }); +});