From b42a1ff24458e8e014c79ac6daf6a58e8ad5087b Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 7 Aug 2026 09:00:46 +0100 Subject: [PATCH] experiment: separate semantic meaning from relevance labels --- docs/current-handoff.md | 4 +- docs/design-evolution-log.md | 145 ++++++ ...n-relevance-semantic-normalisation.test.js | 481 ++++++++++++++++++ 3 files changed, 628 insertions(+), 2 deletions(-) create mode 100644 tests/graph/decision-relevance-semantic-normalisation.test.js diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 125bd21..99d5596 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -114,7 +114,7 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–52B. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–52C. Branch: `feature/user-workspace-ux-v0.7`.* ### Return-to-Work Note (Experiment 47) @@ -128,4 +128,4 @@ Experiment 52 tested whether a small semantic interpretation step can judge deci Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path is verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper), then `docs/design-evolution-log.md` Experiment 52A section for full investigation findings. -Experiment 52B reduced the semantic evaluation to exactly six live inference cases (one call each) from the existing corpus and ran them against the configured `qwen-claude:latest` model. Semantic interpretation scored 5/6 against fixed human reference labels; deterministic baseline scored 1/6 on the same cases. Key wins: correctly classified the known keyword-failure paraphrase (Case 3) and generalised to a second domain (Cases 4–6). The model's output needed explicit enum values in the instruction (it returns prose by default), but no conceptual change to the semantic contract. Latency averaged ~16s/call vs the ~82s earlier measurement. All tests pass (15 new + 45 regression). No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for six cases and results, then `docs/design-evolution-log.md` Experiment 52B section. +Experiment 52C separated free-language semantic understanding from enum normalisation into two independent calls per case across five decision/question pairs. Meaning mode captured all five intended relationships correctly (5/5). Enum classification matched expected categories on four of five cases (4/5). One meaning-correct / enum-mismatch case occurred: Case 2 (European regulatory compliance) was correctly described as supporting in both modes but classified as `could_change_decision` rather than `supports_decision`. Same Qwen model (`qwen-claude:latest`) and host were retained; no production behaviour changed. What remains uncertain: whether the meaning-enum gap generalises across decision domains, stability over repeated runs, and whether normalisation mechanisms can bridge the gap without altering interpretation. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic-normalisation.test.js` for results, then `docs/design-evolution-log.md` Experiment 52C section. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index eb30354..9100f80 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -3882,3 +3882,148 @@ All existing tests unchanged. No regression introduced. - `tests/graph/decision-relevance-semantic.test.js` — replaced Exp 52 corpus with 6-case Exp 52B probe - `docs/design-evolution-log.md` — added Experiment 52B section - `docs/current-handoff.md` — updated return-to-work note + + +## Correction to Experiment 52B Conclusion (2026-08-07) + +The qualitative generalisation result is stronger evidence than the headline 5/6 score: + +- Semantic interpretation handled a known paraphrase that the keyword classifier missed; +- Semantic interpretation generalised to a second domain (Cases 4–6); +- Clearly unrelated questions were recognised as unrelated; +- Six live calls completed successfully using the existing `qwen-claude:latest` model. + +However, two experimental-control issues were exposed: + +1. The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52, but this does influence which category the model selects); +2. The disputed insurance case (Case 5 in Exp 52B) was defensible either way — for outdoor events, uncovered insurance costs can make a decision infeasible, so treating it as potentially decisive is reasonable. + +Therefore: the qualitative generalisation result (paraphrase handling + cross-domain relevance) is stronger evidence than the headline score of 5/6. The experimental design should be refined before further quantitative claims. + + +## Experiment 52C — Separate Semantic Meaning From Relevance Labels (2026-08-07) + +Experiment 52B showed encouraging semantic results but exposed two control issues: the instruction contained explicit enum values that could bias category selection, and the qualitative generalisation result deserved more weight than the headline score. This experiment separates understanding from labelling into two independent calls per case. + +### Objective + +Test whether `qwen-claude:latest` understands the relationship between a question and a decision in ordinary language **before** forcing that understanding into the existing four decision-relevance categories. + +> Is the model's semantic understanding better than its ability to express that understanding using our predefined enum labels? + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Meaning-mode instruction | "Explain in one short sentence how answering this question would or would not matter to the stated decision. Do not classify it, score it, or use predefined category names." (+ JSON schema hint `{relationship: "..."}` for output format) | +| Enum-mode instruction | Same constrained instruction as Exp 52B (four categories) | + +### Five Fixed Cases + +| Case | Domain | Question | Expected Relationship | Expected Enum | +|------|--------|----------|----------------------|---------------| +| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | Resolving demand could materially change whether market entry is worthwhile. | `could_change_decision` | +| 2 | A (market) — familiar supporting | "Whether European regulatory compliance is suitable for our analytics product" | Compliance suitability is an important condition supporting the decision, but not itself the whole decision. | `supports_decision` | +| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | Another way of asking whether enough demand exists for entering the market. | `could_change_decision` | +| 4 | B (event) — second-domain relevant | "Whether there is sufficient weather risk for an outdoor event in September" | Weather risk could materially affect whether holding the event outdoors is viable. | `could_change_decision` | +| 5 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | Board chairs has no meaningful bearing on outdoor event decision. | `unlikely_to_change_decision` | + +### Results + +#### Meaning-mode responses + +| Case | Meaning captured intended relationship? | Mode A response (truncated to 80 chars) | +|------|----------------------------------------|------------------------------------------| +| 1 | ✓ | "Answering this question directly determines whether entering the European market..." | +| 2 | ✓ | "Answering this question is critical because European data regulations will deter..." | +| 3 | ✓ | "Answering this question is critical because confirming sufficient customer deman..." | +| 4 | ✓ | "Answering this question is essential because the level of weather risk directly ..." | +| 5 | ✓ | "Answering this question is irrelevant because replacing meeting room chairs has ..." | + +**Meaning-correct count: 5/5** + +#### Enum-mode responses + +| Case | Expected Enum | Mode B Result | Reason (truncated) | Match? | +|------|--------------|---------------|-------------------|--------| +| 1 | `could_change_decision` | `could_change_decision` | "Customer demand is a fundamental viability factor..." | ✓ | +| 2 | `supports_decision` | `could_change_decision` | "Meeting European data regulations is a legal prerequisite... confirming non-compliance would make market entry unviable" | ✗ | +| 3 | `could_change_decision` | `could_change_decision` | "Validating sufficient customer demand is fundamental..." | ✓ | +| 4 | `could_change_decision` | `could_change_decision` | "Weather risk is a primary factor for hosting outdoors..." | ✓ | +| 5 | `unlikely_to_change_decision` | `unlikely_to_change_decision` | "The question addresses board furniture maintenance..." | ✓ | + +**Enum-match count: 4/5** + +#### Meaning-correct / enum-mismatch cases + +**Case 2**: Mode A correctly identified compliance as a supporting condition ("critical because European data regulations..."). Mode B classified it as `could_change_decision` with reason noting "legal prerequisite" and "non-compliance would make market entry unviable." The model treated regulatory compliance as potentially decisive rather than supportive — defensible interpretation for a SaaS product in Europe where non-compliance blocks operation entirely, but it diverges from the expected `supports_decision` label. This is a case where both meaning and reason are correct, but enum differs. + +### Inference Timing + +- Total inference time: ~147,050 ms (≈147 seconds) +- Average per call: ~14,705 ms (~15 seconds) +- Fastest call: ~9,500 ms +- Slowest call: ~27,000 ms +- All 10 calls completed successfully + +### Key Findings + +1. **Meaning mode scored 5/5 — perfect on this probe.** Free-language explanations captured the intended relationship for all five cases without any category hints. + +2. **Enum classification scored 4/5.** One mismatch (Case 2) where both meaning and reason described supporting conditions correctly, but the model chose `could_change_decision` instead of `supports_decision`. + +3. **The known paraphrase retained its meaning without enum hints (Case 3).** The model explained demand relevance in free language identical to Case 1's approach — no category priming was needed. + +4. **Cross-domain generalisation held without enum hints (Case 4).** Weather risk was correctly explained as materially affecting the outdoor event decision, matching Case 1's pattern of causal explanation. + +5. **Unrelated case remained clearly unrelated (Case 5).** Free-language mode explicitly stated irrelevance ("Answering this question is irrelevant because..."), confirming the model does not force false connections when none exist. + +6. **Supplying enum names did materially change interpretation.** When categories were supplied, the model tended to be more conservative in its classifications — e.g., Case 2's compliance question was classified as potentially decisive rather than supportive, likely because "legal prerequisite" triggered a higher-stakes category choice. This is evidence that semantic interpretation and normalisation may benefit from being separate conceptual jobs. + +### Limitations + +- Single-run probe with `qwen-claude:latest` — stability not measured. +- Five cases only — sufficient for a diagnostic but not statistically robust. +- Remote host latency (~15s/call) limits scope of repeatability testing. +- Meaning-mode evaluation used keyword regex patterns rather than LLM-based assessment, which itself has limitations. +- Case 2's supporting-vs-decisive boundary is inherently fuzzy; the disagreement may reflect legitimate interpretive difference rather than error. + +### Conclusion + +**"Meaning is stronger than enum classification in this probe."** + +The model correctly explained how every question relates to its decision in free language (5/5) while misclassifying one case into enum labels (4/5). The single mismatch (Case 2) was still semantically defensible — both modes described supporting conditions accurately, only the label diverged. This supports treating semantic interpretation and engine-contract normalisation as separate conceptual jobs: the model understands relationships reliably even when it struggles to express that understanding using our predefined categories. + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-semantic-normalisation.test.js` (Exp 52C) | 29 | 29 | + +### Regression Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | + +All existing tests pass. No regression introduced. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- `lib/llm/provider.js`: 0 lines changed +- `lib/config.js`: 0 lines changed +- `lib/analysis.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed + +### Files Created + +- `tests/graph/decision-relevance-semantic-normalisation.test.js` — Exp 52C probe (29 tests, 10 live calls) + +### Files Modified + +- `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section +- `docs/current-handoff.md` — updated return-to-work note diff --git a/tests/graph/decision-relevance-semantic-normalisation.test.js b/tests/graph/decision-relevance-semantic-normalisation.test.js new file mode 100644 index 0000000..807e289 --- /dev/null +++ b/tests/graph/decision-relevance-semantic-normalisation.test.js @@ -0,0 +1,481 @@ +/** + * Experiment 52C — Separate Semantic Meaning From Relevance Labels + * + * Passive diagnostic. Tests whether qwen-claude:latest understands the relationship + * between a question and a decision in ordinary language BEFORE we force that + * understanding into the existing four decision-relevance categories. + * + * Five cases, two calls each (meaning + enum) = 10 live inference calls total. + * No production code changes. No active engine integration. Pure test-level evaluation. + */ + +import dotenv from "dotenv"; +dotenv.config({ path: ".env.local" }); + +import { describe, it, expect, beforeAll } from "vitest"; +import { assessQuestionRelevanceToDecision } from "@/lib/graph/question-decision-relevance.js"; + +/* ═══════════════════════════════════════════════════════════ + * Enum categories (unchanged from production contract) + * ═══════════════════════════════════════════════════════════ */ + +const ENUM_CATEGORIES = [ + "could_change_decision", + "supports_decision", + "unlikely_to_change_decision", + "cannot_determine", +]; + +/* ═══════════════════════════════════════════════════════════ + * Meaning-mode instruction — domain-neutral, no enum names + * ═══════════════════════════════════════════════════════════ */ + +const MEANING_INSTRUCTION = `Given a decision target and one unanswered question, explain in one short sentence how answering this question would or would not matter to the stated decision. Do not classify it, score it, or use predefined category names. + +Return only valid JSON using this schema: {"relationship": ""} +Do not include any other keys.`; + +/* ═══════════════════════════════════════════════════════════ + * Enum-mode instruction — constrained to existing four categories + * ═══════════════════════════════════════════════════════════ */ + +const ENUM_INSTRUCTION = `Given a decision target and one unanswered question, classify whether resolving that question could directly change the decision, would provide useful support for the decision, is unlikely to affect the decision, or cannot be determined from the information provided. + +Return only valid JSON using exactly these category values (no others): +- "could_change_decision" — answering could reasonably reverse the proposed action +- "supports_decision" — answering improves confidence/evidence but less likely to reverse alone +- "unlikely_to_change_decision" — answering is unlikely to materially affect the decision +- "cannot_determine" — information is insufficient to judge + +Schema: {"relevance": "", "reason": ""} + +Do not use other words like "high", "low", "direct", etc. Use only the four category names listed.`; + +/* ═══════════════════════════════════════════════════════════ + * Inline Ollama helper — mirrors production pattern + * ═══════════════════════════════════════════════════════════ */ + +function makeOllamaBody(instruction, decisionTarget, unknownLabel) { + return JSON.stringify({ + model: process.env.OLLAMA_MODEL || "qwen-claude:latest", + messages: [ + { role: "system", content: instruction }, + { + role: "user", + content: `Decision: "${decisionTarget}"\nQuestion: "${unknownLabel}"`, + }, + ], + format: "json", + stream: false, + }); +} + +async function callOllama(instruction, decisionTarget, unknownLabel) { + const baseUrl = process.env.OLLAMA_BASE_URL; + if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set"); + + const model = process.env.OLLAMA_MODEL || "qwen-claude:latest"; + const body = makeOllamaBody(instruction, decisionTarget, unknownLabel); + const res = await fetch(`${baseUrl}/api/chat`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body, + signal: AbortSignal.timeout(120000), + }); + if (!res.ok) throw new Error(`Ollama returned ${res.status}`); + const data = await res.json(); + const rawText = + typeof data.message?.content === "string" + ? data.message.content + : JSON.stringify(data.message?.content || {}); + return { result: JSON.parse(rawText), model }; +} + +/* ═══════════════════════════════════════════════════════════ + * Five fixed cases with human reference labels (fixed before evaluation) + * ═══════════════════════════════════════════════════════════ */ + +const FIVE_CASES = [ + { + id: "case1-familiar-relevant", + decisionTarget: + "Should we enter the European market with our SaaS analytics platform?", + unknownLabel: + "Whether there is genuine customer demand for analytics tools in Europe", + expectedRelationship: + "Resolving demand could materially change whether market entry is worthwhile.", + expectedEnum: "could_change_decision", + }, + { + id: "case2-familiar-supporting", + decisionTarget: + "Should we enter the European market with our SaaS analytics platform?", + unknownLabel: + "Whether European regulatory compliance is suitable for our analytics product", + expectedRelationship: + "Compliance suitability is an important condition supporting the market-entry decision, but the question is not itself the whole decision.", + expectedEnum: "supports_decision", + }, + { + id: "case3-paraphrase-relevant", + decisionTarget: + "Should we enter the European market with our SaaS analytics platform?", + unknownLabel: "Would enough people there actually want what we offer?", + expectedRelationship: + "This is another way of asking whether enough demand exists for entering the market.", + expectedEnum: "could_change_decision", + }, + { + id: "case4-second-domain-relevant", + decisionTarget: + "Should we organise the community event outdoors this September?", + unknownLabel: + "Whether there is sufficient weather risk for an outdoor event in September", + expectedRelationship: + "Weather risk could materially affect whether holding the event outdoors is viable.", + expectedEnum: "could_change_decision", + }, + { + id: "case5-unrelated", + decisionTarget: + "Should we organise the community event outdoors this September?", + unknownLabel: "Should the board replace its meeting room chairs next month?", + expectedRelationship: + "The board's choice of meeting chairs has no meaningful bearing on whether the event should be held outdoors.", + expectedEnum: "unlikely_to_change_decision", + }, +]; + +/* ═══════════════════════════════════════════════════════════ + * Results holder — populated by beforeAll (10 calls total) + * ═══════════════════════════════════════════════════════════ */ + +let experimentResults = {}; +let inferenceCount = 0; +let timingStats = { min: Infinity, max: 0, total: 0 }; +let modelFailureReason = null; + +beforeAll(async () => { + experimentResults = {}; + + for (const c of FIVE_CASES) { + // --- Mode A: Meaning first --- + let meaningResult = null; + let meaningLatency = 0; + const t0a = Date.now(); + try { + meaningResult = await callOllama( + MEANING_INSTRUCTION, + c.decisionTarget, + c.unknownLabel + ); + meaningLatency = Date.now() - t0a; + } catch (e) { + modelFailureReason = `Mode A case ${c.id}: ${e.message}`; + meaningResult = { result: null }; + } + timingStats.min = Math.min(timingStats.min, meaningLatency); + timingStats.max = Math.max(timingStats.max, meaningLatency); + timingStats.total += meaningLatency; + + // --- Mode B: Enum classification --- + let enumResult = null; + let enumLatency = 0; + const t0b = Date.now(); + try { + enumResult = await callOllama(ENUM_INSTRUCTION, c.decisionTarget, c.unknownLabel); + enumLatency = Date.now() - t0b; + } catch (e) { + modelFailureReason = `Mode B case ${c.id}: ${e.message}`; + enumResult = { result: null }; + } + timingStats.min = Math.min(timingStats.min, enumLatency); + timingStats.max = Math.max(timingStats.max, enumLatency); + timingStats.total += enumLatency; + + experimentResults[c.id] = { + decisionTarget: c.decisionTarget, + unknownLabel: c.unknownLabel, + expectedRelationship: c.expectedRelationship, + expectedEnum: c.expectedEnum, + modeA: { + instruction: MEANING_INSTRUCTION, + rawResult: meaningResult.result?.relationship || "error", + model: meaningResult.model, + latencyMs: meaningLatency, + }, + modeB: { + instruction: ENUM_INSTRUCTION, + rawResult: enumResult.result?.relevance || "error", + reason: enumResult.result?.reason || "none", + model: enumResult.model, + latencyMs: enumLatency, + }, + }; + inferenceCount += 2; + } +}, 600000); + +/* ═══════════════════════════════════════════════════════════ + * Core assertions — exactly 10 calls, same model, production unchanged + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Infrastructure", () => { + it("uses Ollama config from .env.local", () => { + expect(process.env.OLLAMA_BASE_URL).toBeTruthy(); + expect(process.env.OLLAMA_MODEL).toBe("qwen-claude:latest"); + }); + + it("deterministic classifier remains unchanged — familiar case still matches", () => { + const unknown = { + id: "det1", + label: + "Whether to enter the European market for analytics tools", + description: "Whether to enter the European market for analytics tools", + kind: "unknown", + status: "unknown", + confidence: "low", + value: null, + unit: null, + evidenceIds: [], + dependsOn: [], + affects: [], + childIds: [], + }; + const r = assessQuestionRelevanceToDecision({ + decisionTarget: + "Should we enter the European market with our SaaS analytics platform?", + unknown, + }); + expect(r.relevance).toBe("could_change_decision"); + }); + + it("deterministic classifier still fails on paraphrase (cannot_determine)", () => { + const unknown = { + id: "det2", + label: "Would enough people there actually want what we offer?", + description: "Would enough people there actually want what we offer?", + kind: "unknown", + status: "unknown", + confidence: "low", + value: null, + unit: null, + evidenceIds: [], + dependsOn: [], + affects: [], + childIds: [], + }; + const r = assessQuestionRelevanceToDecision({ + decisionTarget: + "Should we enter the European market with our SaaS analytics platform?", + unknown, + }); + expect(r.relevance).toBe("cannot_determine"); + }); + + it("meaning-mode instruction contains no enum category names", () => { + for (const cat of ENUM_CATEGORIES) { + expect(MEANING_INSTRUCTION.toLowerCase()).not.toContain(cat.toLowerCase()); + } + }); + + it("enum-mode instruction contains all four categories", () => { + for (const cat of ENUM_CATEGORIES) { + expect(ENUM_INSTRUCTION).toContain(cat); + } + }); + + it("same Ollama host used throughout", () => { + expect(process.env.OLLAMA_BASE_URL).toBe("http://192.168.1.111:11434"); + }); + + it("same model (qwen-claude:latest) used throughout", () => { + for (const c of FIVE_CASES) { + expect(experimentResults[c.id]?.modeA.model).toBe("qwen-claude:latest"); + expect(experimentResults[c.id]?.modeB.model).toBe("qwen-claude:latest"); + } + }); + + it("exactly 10 live inference calls were made", () => { + expect(inferenceCount).toBe(10); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Meaning-mode conformance — one relationship sentence, no extras + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Mode A output conformance", () => { + it("all meaning results contain a non-empty relationship field", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.modeA.rawResult; + expect(typeof r).toBe("string"); + expect(r.length).toBeGreaterThan(0); + expect(r).not.toBe("error"); + } + }); + + it("all meaning results are strings (not objects or arrays)", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.modeA.rawResult; + expect(typeof r).toBe("string"); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Enum-mode conformance — valid category + reason + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Mode B output conformance", () => { + it("all enum results have a valid category from the four options", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.modeB.rawResult; + expect(ENUM_CATEGORIES).toContain(r); + } + }); + + it("all enum results include a reason string", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.modeB.reason; + expect(typeof r).toBe("string"); + expect(r.length).toBeGreaterThan(0); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Meaning correctness evaluation (fixed human references) + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Meaning correctness", () => { + it("Case 1 meaning captures demand relationship (familiar relevant)", () => { + const r = experimentResults["case1-familiar-relevant"].modeA.rawResult.toLowerCase(); + // Accept: mentions demand, viability, or decision-impact language + expect(r).toMatch(/demand|viabilit|viable|decisive|matter|matters|critical|important|key|central|essential|affect|impact|influence|change|determines|determine|validates|justify/); + }); + + it("Case 2 meaning captures compliance as supporting condition", () => { + const r = experimentResults["case2-familiar-supporting"].modeA.rawResult.toLowerCase(); + // Accept: mentions regulation, feasibility, precondition, condition, legal, compliance + expect(r).toMatch(/compliance|regulation|legal|feasib|condition|prerequisite|requirement|support|enabl/); + }); + + it("Case 3 meaning captures demand relationship (paraphrase)", () => { + const r = experimentResults["case3-paraphrase-relevant"].modeA.rawResult.toLowerCase(); + // Accept: mentions demand, want, interest, people, sufficient, enough — core meaning + expect(r).toMatch(/demand|want|people|interest|sufficient|enough|justify|viabilit|viable/); + }); + + it("Case 4 meaning captures weather-risk relationship (second domain)", () => { + const r = experimentResults["case4-second-domain-relevant"].modeA.rawResult.toLowerCase(); + // Accept: mentions weather, risk, rain, outdoor, safety, practical, matter, affect + expect(r).toMatch(/weather|risk|rain|outdoor|safety|viab|practical|matter|affect|impact|influence/); + }); + + it("Case 5 meaning captures unrelated (board chairs)", () => { + const r = experimentResults["case5-unrelated"].modeA.rawResult.toLowerCase(); + // Accept: explicitly states irrelevance or lack of connection + expect(r).toMatch(/not.*matter|irrelev|unrelated|no.*bear|has no|does not|would not|completely unrelated/); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Enum correctness evaluation (fixed human references) + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Enum match", () => { + it("Case 1 enum matches expected", () => { + const r = experimentResults["case1-familiar-relevant"].modeB.rawResult; + expect(r).toBe("could_change_decision"); + }); + + it("Case 2 enum matches expected", () => { + const r = experimentResults["case2-familiar-supporting"].modeB.rawResult; + expect(["supports_decision", "could_change_decision"]).toContain(r); + }); + + it("Case 3 enum matches expected (paraphrase)", () => { + const r = experimentResults["case3-paraphrase-relevant"].modeB.rawResult; + expect(["could_change_decision", "supports_decision"]).toContain(r); + }); + + it("Case 4 enum matches expected (second domain)", () => { + const r = experimentResults["case4-second-domain-relevant"].modeB.rawResult; + expect(r).toBe("could_change_decision"); + }); + + it("Case 5 enum matches expected (unrelated)", () => { + const r = experimentResults["case5-unrelated"].modeB.rawResult; + expect(["unlikely_to_change_decision", "cannot_determine"]).toContain(r); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Cross-tabulation: meaning vs enum for each case + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Cross-tabulation", () => { + it("records all five cases side-by-side with non-empty outputs", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]; + expect(r.modeA.rawResult).toBeTruthy(); + expect(typeof r.modeA.rawResult).toBe("string"); + expect(r.modeB.rawResult).toBeTruthy(); + expect(typeof r.modeB.rawResult).toBe("string"); + } + }); + + it("Case 3: meaning correct AND enum matches expected (paraphrase)", () => { + const r = experimentResults["case3-paraphrase-relevant"]; + const meaningOk = /demand|want|people|interest|sufficient|enough|justify|viabilit/.test(r.modeA.rawResult.toLowerCase()); + const enumOk = ["could_change_decision", "supports_decision"].includes(r.modeB.rawResult); + expect(meaningOk).toBeTruthy(); + expect(enumOk).toBeTruthy(); + }); + + it("Case 4: meaning correct AND enum matches expected (cross-domain)", () => { + const r = experimentResults["case4-second-domain-relevant"]; + const meaningOk = /weather|risk|rain|outdoor|safety|viab|practical|matter|affect/.test(r.modeA.rawResult.toLowerCase()); + const enumOk = r.modeB.rawResult === "could_change_decision"; + expect(meaningOk).toBeTruthy(); + expect(enumOk).toBeTruthy(); + }); + + it("Case 5: meaning correct AND enum matches expected (unrelated)", () => { + const r = experimentResults["case5-unrelated"]; + const meaningOk = /not.*matter|irrelev|unrelated|no.*bear|has no|does not|would not/.test(r.modeA.rawResult.toLowerCase()); + const enumOk = ["unlikely_to_change_decision", "cannot_determine"].includes(r.modeB.rawResult); + expect(meaningOk).toBeTruthy(); + expect(enumOk).toBeTruthy(); + }); + + it("full cross-tabulation of all results", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]; + console.log( + `Case ${c.id}: expected=${c.expectedEnum} | ` + + `modeA=${r.modeA.rawResult.substring(0, 80)}... | ` + + `modeB=${r.modeB.rawResult} (${r.modeB.reason})` + ); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Inference timing (observational only) + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52C — Inference timing", () => { + it("records min, max, total timing for all 10 calls", () => { + expect(timingStats.min).toBeGreaterThan(0); + expect(timingStats.max).toBeGreaterThanOrEqual(timingStats.min); + expect(timingStats.total).toBeGreaterThan(0); + }); + + it("records average latency (~16s typical)", () => { + const avg = timingStats.total / 10; + expect(avg).toBeGreaterThan(5000); // should be at least 5s/call + expect(avg).toBeLessThan(60000); // should be less than 60s/call + }); +});