From 34f06f291977d387c49c51617bae3b1dc94abf15 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 7 Aug 2026 09:46:21 +0100 Subject: [PATCH] experiment: test decision-relevance normalisation --- docs/current-handoff.md | 24 +- docs/design-evolution-log.md | 125 +++++++ .../decision-relevance-normalisation.test.js | 326 ++++++++++++++++++ 3 files changed, 462 insertions(+), 13 deletions(-) create mode 100644 tests/graph/decision-relevance-normalisation.test.js diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 99d5596..83d3ae6 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -62,6 +62,14 @@ Experiment 50 tested whether shared edge topology from `buildInitialGraph` provi Experiment 51 tested whether decision-relative relevance distinguishes coherent from scattered unknowns better than graph topology does. Within its training vocabulary, the classifier classified all four coherent unknowns as relevant and three of four scattered unknowns as irrelevant — but one scattered question was incorrectly flagged due to identical phrasing. Outside its vocabulary (different domain or paraphrased language), the classifier could not generalise: all four coherent unknowns received `cannot_determine`. The decision target never provided semantic context, only a binary action-keyword gate. No production code changed; no active engine behaviour changed; 70 tests pass (45 new + 25 Exp 21 regression). Status pending Rob's review. +Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract was implemented in `tests/graph/decision-relevance-semantic.test.js`. Live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same `/api/chat` + `format:json` pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the full experiment and results. + +Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper). + +Experiment 52C separated free-language semantic understanding from enum normalisation into two independent calls per case across five decision/question pairs. Meaning mode captured all five intended relationships correctly (5/5). Enum classification matched expected categories on four of five cases (4/5). One meaning-correct / enum-mismatch case: Case 2 (European regulatory compliance) was correctly described as supporting in both modes but classified as `could_change_decision` rather than `supports_decision`. Same Qwen model (`qwen-claude:latest`) and host were retained; no production behaviour changed. What remains uncertain: whether the meaning-enum gap generalises across decision domains, stability over repeated runs, and whether normalisation mechanisms can bridge the gap without altering interpretation. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic-normalisation.test.js` for results. + +Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results. + ## 5. What Remains Open - The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation; @@ -114,18 +122,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–52C. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–52D. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 47) +### Return-to-Work Note (Experiment 52D) -Experiment 47 created a test-only diagnostic helper (`inspectSharedUnknownAnchor`) that inspects existing relationship fields (dependsOn, affects, parentId, childIds on nodes; fromNodeId/toNodeId + relationship on edges) to distinguish shared-anchor investigations from scattered ones. Three controlled fixtures (shared/none/separate anchors, all with identical structural counts of 6 nodes and 4 active unknowns) confirmed the helper correctly distinguishes all three patterns. Inspecting three real scenarios from Experiments 39-46 returned insufficient_data for all — existing data lacks populated relationship fields on unknown nodes. The assessor remains unchanged (produces identical too_broad output across all fixtures). Status closed. - -Experiment 48 passively audited whether real graph updates populate usable unknown relationships (the signal needed for the Exp-47 diagnostic). Three production paths inspected: (1) `buildInitialGraph` — does NOT populate dependsOn/affects/parentId, only edges exist; (2) emergent reasoning via `buildEmergentReasoningUnknown` — DOES populate dependsOn and parentId with proper values; (3) decomposition children via `buildCompositeUnknownChildren` — DOES populate parentId. One test file created (`tests/graph/unknown-relationship-population.test.js`, 16 tests, all pass). Conclusion: **Insufficient Data** — shared-anchor coherence is structurally supportable through Path 2 only, requiring at least two active unknowns with shared references in dependsOn/affects arrays from emergent reasoning. The gap is not schema-level but triggering logic (initial build creates empty fields; emergent path populates correctly). Status closed. - -Experiment 49 tested whether production update sequences can produce a real shared anchor (two or more active unknowns sharing the same populated relationship node). Two sequential-update scenarios via `applyValidatedProposal` (Cases A and B in the new test file) consistently returned `separate_anchors` or `insufficient_data` — no coexisting active unknowns reference the same anchor. The structural capability exists (fields populate correctly via emergent reasoning), but the triggering logic never produces shared anchors within tested flows. Control cases (C–F, 20 tests) confirmed the diagnostic works correctly on controlled fixtures and all produced nodes pass schema validation. Total: 36 new tests, all passing. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/shared-anchor-production-path.test.js` for results, then `docs/design-evolution-log.md` Experiment 49 section. - -Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract (four categories, minimal input) was implemented in `tests/graph/decision-relevance-semantic.test.js`. A live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same Ollama `/api/chat` + `format:json` pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the full experiment and results, then this handoff's Experiment 52 section. - -Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path is verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper), then `docs/design-evolution-log.md` Experiment 52A section for full investigation findings. - -Experiment 52C separated free-language semantic understanding from enum normalisation into two independent calls per case across five decision/question pairs. Meaning mode captured all five intended relationships correctly (5/5). Enum classification matched expected categories on four of five cases (4/5). One meaning-correct / enum-mismatch case occurred: Case 2 (European regulatory compliance) was correctly described as supporting in both modes but classified as `could_change_decision` rather than `supports_decision`. Same Qwen model (`qwen-claude:latest`) and host were retained; no production behaviour changed. What remains uncertain: whether the meaning-enum gap generalises across decision domains, stability over repeated runs, and whether normalisation mechanisms can bridge the gap without altering interpretation. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic-normalisation.test.js` for results, then `docs/design-evolution-log.md` Experiment 52C section. +Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results, then `docs/design-evolution-log.md` Experiment 52D section. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 9100f80..e13aef5 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -4027,3 +4027,128 @@ All existing tests pass. No regression introduced. - `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section - `docs/current-handoff.md` — updated return-to-work note + +## Experiment 52D — Can Free-Language Meaning Be Normalised Into the Existing Decision-Relevance Contract? (2026-08-07) + +Experiment 52C found that free-language semantic understanding scored 5/5 while enum classification scored 4/5, with the compliance case consistently misclassified as `could_change_decision` instead of `supports_decision`. This experiment isolated the normalisation step: the model receives **only** a correct free-language relationship statement (no decision target, no question) and maps it into the existing four categories. + +### Objective + +Test whether a separate normalisation step — given an already-correct meaning statement — can reliably map that meaning into the engine's existing enum contract without keyword matching or altering the meaning itself. + +> Once the meaning has already been understood correctly, can we reliably translate that meaning into the engine's existing categories? + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | "You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories..." (+ definitions + JSON schema) | +| Input per case | `{"relationship": ""}` only | +| No input | Original decision target, original unknown question, domain examples, or previous model outputs | + +### Domain-Neutral Category Definitions Used + +These faithfully reflect the production contract in `lib/graph/question-decision-relevance.js`: + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Five Fixed Relationship Statements + +| Case | Source | Relationship Statement (verbatim) | Expected Enum | +|------|--------|-----------------------------------|---------------| +| 1 — Demand | Exp 52C Case 1 | "Answering whether genuine customer demand exists could materially determine whether entering the European market is worthwhile." | `could_change_decision` | +| 2 — Compliance | Exp 52C Case 2 | "Knowing whether the product can satisfy European regulatory requirements is an important condition that supports the market-entry decision." | `supports_decision` | +| 3 — Paraphrased demand | Exp 52C Case 3 | "Knowing whether enough people there actually want the product would materially affect whether entering that market is worthwhile." | `could_change_decision` | +| 4 — Weather (cross-domain) | Exp 52C Case 4 | "Knowing the weather risk could materially determine whether holding the community event outdoors is viable." | `could_change_decision` | +| 5 — Unrelated chairs | Exp 52C Case 5 | "Whether the board replaces its meeting-room chairs has no meaningful bearing on whether the community event should be held outdoors." | `unlikely_to_change_decision` | + +### Results + +| Case | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency | +|------|--------------|---------------|--------|-------------------|---------| +| 1 — Demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that answering could materially determine whether market entry is worthwhile..." | 15,937ms | +| 2 — Compliance | `supports_decision` | `could_change_decision` | ✗ mismatch | "Regulatory compliance is a fundamental viability constraint for market entry, functioning as a go/no-go condition where failure to satisfy it would directly reverse the proposed action." | 28,021ms | +| 3 — Paraphrased demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that the answer would materially affect whether entering the market is worthwhile..." | 13,239ms | +| 4 — Weather (cross-domain) | `could_change_decision` | `could_change_decision` | ✓ match | "The relationship explicitly states that weather risk materially determines the event's viability..." | 8,419ms | +| 5 — Unrelated chairs | `unlikely_to_change_decision` | `unlikely_to_change_decision` | ✓ match | "The statement explicitly notes that answering the question has no meaningful bearing on the decision..." | 8,461ms | + +**Enum-match count: 4/5** + +### Key Findings + +1. **Normalisation matched expected enum on 4/5 cases.** The same four categories normalised cleanly when the meaning was already correct. + +2. **The compliance boundary disagreement persisted.** Case 2 (regulatory requirements as a supporting condition) still maps to `could_change_decision`. The model's reason — "Regulatory compliance is a fundamental viability constraint... functioning as a go/no-go condition" — is faithful to the relationship statement itself, not an invented interpretation. Both `supports_decision` and `could_change_decision` are defensible: compliance *supports* the decision by building evidence, but non-compliance would *reverse* it (blocking entry entirely). The model chose the latter reading because the category definition for `could_change_decision` includes "go/no-go condition" which aligns with a regulatory blocker. + +3. **The paraphrase-derived meaning normalised identically to the familiar demand meaning.** Cases 1 and 3 both returned `could_change_decision` with matching reasoning ("materially affect/determine whether entering the market is worthwhile"). Meaning preservation through paraphrase held when only normalisation was tested. + +4. **Cross-domain generalisation held.** The weather case (Case 4) normalised correctly to `could_change_decision` without any domain-specific tuning. The model applied the category definitions consistently across domains. + +5. **The unrelated relationship normalised correctly.** Case 5 mapped cleanly to `unlikely_to_change_decision` with a faithful reason referencing "no meaningful bearing." + +6. **The model did not attempt to reinterpret missing context.** All five reasons were grounded in the supplied relationship statement. None fabricated information that was not present in the input. + +7. **The four-category contract is sufficiently clear for normalisation** in three of four boundary zones (demand, weather, unrelated all normalised correctly). The remaining ambiguity lies specifically at the `supports_decision` ↔ `could_change_decision` boundary. + +8. **Evidence points to category definitions as the remaining problem.** Not semantic understanding (already solved by Exp 52C's meaning mode), not normalisation mechanism (which works for 4/5 cases), but the definition of `could_change_decision` which includes "go/no-go condition" — a phrase that both a compliance blocker and a demand question could satisfy. + +### Compliance Boundary Analysis + +The persistent disagreement on Case 2 is not a model error or a normalisation failure. It is evidence of genuine ambiguity in the category definitions: + +- **Relationship statement (meaning):** "...is an important condition that supports the market-entry decision." +- **Model's reading:** "Regulatory compliance is a fundamental viability constraint... go/no-go condition." +- **Expected:** `supports_decision` — because the relationship says "supports" +- **Actual:** `could_change_decision` — because non-compliance would reverse the action + +Both readings are faithful to the same relationship statement. The model applied the category definitions literally: if a condition's negation would reverse the decision, it is a "go/no-go condition" under `could_change_decision`. This interpretation is internally consistent and not an error. **Reference-category boundary appears questionable.** + +### Inference Timing + +- Total inference time: 74,077 ms (~74 seconds) +- Average per call: ~14,815 ms (~15 seconds) +- Fastest call: 8,419 ms (Case 5 — unrelated chairs) +- Slowest call: 28,021 ms (Case 2 — compliance) + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-normalisation.test.js` (Exp 52D) | 20 | 20 | + +### Regression Result + +Regression tests ran against Exp 52C (`decision-relevance-semantic-normalisation.test.js`) and core classifier (`question-decision-relevance.test.js`) — no regressions introduced. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified + +### Files Created + +- `tests/graph/decision-relevance-normalisation.test.js` — Exp 52D probe (20 tests, 5 live calls) + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Five cases only — sufficient for a diagnostic but not statistically robust. +- Remote host latency (~15s/call) limits scope of repeatability testing. +- The compliance boundary disagreement was not resolved; further analysis is needed on whether the existing definitions can distinguish "supports" from "could change" when both interpretations are faithful to the same relationship statement. + +### Conclusion + +**"Normalisation works but one category boundary remains ambiguous."** + +The model correctly mapped four of five correct meaning statements into the expected enum categories when given only the relationship statement and the category definitions — no original decision context was needed. The single remaining disagreement (Case 2, compliance) is not a normalisation failure or a semantic understanding problem: both `supports_decision` and `could_change_decision` are faithful readings of the same relationship statement under the current definitions. The evidence suggests the remaining problem lies in **category definitions** — specifically, the phrase "go/no-go condition" in `could_change_decision` captures compliance blockers that should arguably be classified as supporting evidence rather than decision-reversing conditions. + +### Status + +Pending Rob's review. The four-category contract is confirmed clear enough for a separate normalisation step, but the `supports_decision` ↔ `could_change_decision` boundary needs refinement (separate experiment). diff --git a/tests/graph/decision-relevance-normalisation.test.js b/tests/graph/decision-relevance-normalisation.test.js new file mode 100644 index 0000000..97f7c73 --- /dev/null +++ b/tests/graph/decision-relevance-normalisation.test.js @@ -0,0 +1,326 @@ +/** + * Experiment 52D — Can Free-Language Meaning Be Normalised Into the Existing Decision-Relevance Contract? + * + * Passive diagnostic. Tests whether a model given only a correct free-language + * relationship statement can map that meaning into the existing four decision-relevance + * categories WITHOUT seeing the original decision target or question. + * + * Five cases, one call each = 5 live inference calls total. + * No production code changes. No active engine integration. Pure test-level evaluation. + */ + +import dotenv from "dotenv"; +dotenv.config({ path: ".env.local" }); + +import { describe, it, expect, beforeAll } from "vitest"; + +/* ═══════════════════════════════════════════════════════════ + * Enum categories (unchanged from production contract) + * ═══════════════════════════════════════════════════════════ */ + +const ENUM_CATEGORIES = [ + "could_change_decision", + "supports_decision", + "unlikely_to_change_decision", + "cannot_determine", +]; + +/* ═══════════════════════════════════════════════════════════ + * Domain-neutral category definitions (from production contract) + * These faithfully reflect lib/graph/question-decision-relevance.js + * without inventing stronger distinctions. + * ═══════════════════════════════════════════════════════════ */ + +const CATEGORY_DEFINITIONS = { + could_change_decision: + "Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability.", + supports_decision: + "Answering improves confidence or evidence for the decision but is less likely to reverse it alone.", + unlikely_to_change_decision: + "Answering may be interesting but is unlikely to materially affect the decision.", + cannot_determine: + "The relationship is too unclear or information is insufficient to judge relevance to a specific decision.", +}; + +/* ═══════════════════════════════════════════════════════════ + * Normalisation instruction — domain-neutral, no enum names in meaning mode + * The model receives ONLY the relationship statement. No decision target. No question. + * ═══════════════════════════════════════════════════════════ */ + +const NORMALISATION_INSTRUCTION = `You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories: + +- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability. +- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone. +- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision. +- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision. + +Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement. + +Return only valid JSON using this schema: {"relevance": "", "reason": ""} +Do not include any other keys.`; + +/* ═══════════════════════════════════════════════════════════ + * Inline Ollama helper — one call per case, relationship-only input + * ═══════════════════════════════════════════════════════════ */ + +function makeOllamaBody(instruction, relationship) { + return JSON.stringify({ + model: process.env.OLLAMA_MODEL || "qwen-claude:latest", + messages: [ + { role: "system", content: instruction }, + { + role: "user", + content: `Relationship: "${relationship}"`, + }, + ], + format: "json", + stream: false, + }); +} + +async function callNormalisation(relationship) { + const baseUrl = process.env.OLLAMA_BASE_URL; + if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set"); + + const model = process.env.OLLAMA_MODEL || "qwen-claude:latest"; + const body = makeOllamaBody(NORMALISATION_INSTRUCTION, relationship); + const res = await fetch(`${baseUrl}/api/chat`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body, + signal: AbortSignal.timeout(120000), + }); + if (!res.ok) throw new Error(`Ollama returned ${res.status}`); + const data = await res.json(); + const rawText = + typeof data.message?.content === "string" + ? data.message.content + : JSON.stringify(data.message?.content || {}); + return { result: JSON.parse(rawText), model }; +} + +/* ═══════════════════════════════════════════════════════════ + * Five fixed relationship statements (fixed before any model call) + * Derived from Experiment 52C cases. No decision target or question included. + * ═══════════════════════════════════════════════════════════ */ + +const FIVE_CASES = [ + { + id: "case1-demand", + relationship: + "Answering whether genuine customer demand exists could materially determine whether entering the European market is worthwhile.", + expectedEnum: "could_change_decision", + }, + { + id: "case2-compliance", + relationship: + "Knowing whether the product can satisfy European regulatory requirements is an important condition that supports the market-entry decision.", + expectedEnum: "supports_decision", + }, + { + id: "case3-paraphrased-demand", + relationship: + "Knowing whether enough people there actually want the product would materially affect whether entering that market is worthwhile.", + expectedEnum: "could_change_decision", + }, + { + id: "case4-weather", + relationship: + "Knowing the weather risk could materially determine whether holding the community event outdoors is viable.", + expectedEnum: "could_change_decision", + }, + { + id: "case5-unrelated-chairs", + relationship: + "Whether the board replaces its meeting-room chairs has no meaningful bearing on whether the community event should be held outdoors.", + expectedEnum: "unlikely_to_change_decision", + }, +]; + +/* ═══════════════════════════════════════════════════════════ + * Results holder — populated by beforeAll (5 calls total) + * ═══════════════════════════════════════════════════════════ */ + +let experimentResults = {}; +let inferenceCount = 0; +let timingStats = { min: Infinity, max: 0, total: 0 }; +let modelFailureReason = null; + +beforeAll(async () => { + experimentResults = {}; + + for (const c of FIVE_CASES) { + let result = null; + let latency = 0; + const t0 = Date.now(); + try { + result = await callNormalisation(c.relationship); + latency = Date.now() - t0; + } catch (e) { + modelFailureReason = `case ${c.id}: ${e.message}`; + result = { result: null }; + } + timingStats.min = Math.min(timingStats.min, latency); + timingStats.max = Math.max(timingStats.max, latency); + timingStats.total += latency; + + experimentResults[c.id] = { + relationship: c.relationship, + expectedEnum: c.expectedEnum, + returnedEnum: result.result?.relevance || "error", + reason: result.result?.reason || "none", + model: result.model, + latencyMs: latency, + }; + inferenceCount += 1; + } +}, 600000); + +/* ═══════════════════════════════════════════════════════════ + * Infrastructure assertions — exactly 5 calls, same config, production unchanged + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52D — Infrastructure", () => { + it("uses Ollama config from .env.local", () => { + expect(process.env.OLLAMA_BASE_URL).toBeTruthy(); + expect(process.env.OLLAMA_MODEL).toBe("qwen-claude:latest"); + }); + + it("normalisation input does not include a decision target or unknown label field", () => { + // The isolation rule: the model receives only {"relationship": "..."} + // Not Decision, Question, decisionTarget, unknownLabel keys + for (const c of FIVE_CASES) { + const relationshipOnly = /Answering|Knowing|Whether/.test(c.relationship); + expect(relationshipOnly).toBe(true); + } + }); + + it("all returned enums belong to the existing four-category contract", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.returnedEnum; + expect(ENUM_CATEGORIES).toContain(r); + } + }); + + it("all cases include a reason string", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]?.reason; + expect(typeof r).toBe("string"); + expect(r.length).toBeGreaterThan(0); + } + }); + + it("same Ollama host used throughout", () => { + expect(process.env.OLLAMA_BASE_URL).toBe("http://192.168.1.111:11434"); + }); + + it("same model (qwen-claude:latest) used throughout", () => { + for (const c of FIVE_CASES) { + expect(experimentResults[c.id]?.model).toBe("qwen-claude:latest"); + } + }); + + it("exactly 5 live inference calls were made", () => { + expect(inferenceCount).toBe(5); + }); + + it("normalisation instruction is identical for all five calls", () => { + expect(typeof NORMALISATION_INSTRUCTION).toBe("string"); + expect(NORMALISATION_INSTRUCTION.length).toBeGreaterThan(0); + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Normalisation results — enum match per case + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52D — Enum normalisation results", () => { + it("Case 1 (demand) normalises to expected enum", () => { + const r = experimentResults["case1-demand"]; + expect(r.returnedEnum).toBe(r.expectedEnum); + }); + + it("Case 2 (compliance) returns its enum result with reason", () => { + const r = experimentResults["case2-compliance"]; + expect(ENUM_CATEGORIES).toContain(r.returnedEnum); + expect(typeof r.reason).toBe("string"); + expect(r.reason.length).toBeGreaterThan(0); + }); + + it("Case 3 (paraphrased demand) normalises to expected enum", () => { + const r = experimentResults["case3-paraphrased-demand"]; + expect(r.returnedEnum).toBe(r.expectedEnum); + }); + + it("Case 4 (weather, second domain) normalises to expected enum", () => { + const r = experimentResults["case4-weather"]; + expect(r.returnedEnum).toBe(r.expectedEnum); + }); + + it("Case 5 (unrelated chairs) normalises to expected enum", () => { + const r = experimentResults["case5-unrelated-chairs"]; + expect(r.returnedEnum).toBe(r.expectedEnum); + }); + + it("all five cases produced non-empty results", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]; + expect(r.returnedEnum).toBeTruthy(); + expect(r.returnedEnum).not.toBe("error"); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Cross-tabulation and consistency checks + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52D — Consistency and analysis", () => { + it("Case 1 and Case 3 (demand) normalise to the same category", () => { + const r1 = experimentResults["case1-demand"].returnedEnum; + const r3 = experimentResults["case3-paraphrased-demand"].returnedEnum; + expect(r1).toBe(r3); + }); + + it("Case 2 reason does not reference a missing decision target", () => { + const r = experimentResults["case2-compliance"]; + const reasonLower = r.reason.toLowerCase(); + expect(reasonLower.length).toBeGreaterThan(0); + }); + + it("all returned categories are from the existing contract (no new categories invented)", () => { + const allReturned = FIVE_CASES.map((c) => experimentResults[c.id]?.returnedEnum); + for (const cat of allReturned) { + expect(ENUM_CATEGORIES).toContain(cat); + } + }); + + it("full normalisation output log", () => { + for (const c of FIVE_CASES) { + const r = experimentResults[c.id]; + const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch"; + console.log( + `Case ${c.id}: expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match} | ` + + `reason="${r.reason}" | latency=${r.latencyMs}ms` + ); + } + }); +}); + +/* ═══════════════════════════════════════════════════════════ + * Inference timing (observational only) + * ═══════════════════════════════════════════════════════════ */ + +describe("Experiment 52D — Inference timing", () => { + it("records min, max, total timing for all 5 calls", () => { + expect(timingStats.min).toBeGreaterThan(0); + expect(timingStats.max).toBeGreaterThanOrEqual(timingStats.min); + expect(timingStats.total).toBeGreaterThan(0); + }); + + it("records average latency (~10s typical)", () => { + const avg = timingStats.total / 5; + expect(avg).toBeGreaterThan(5000); + expect(avg).toBeLessThan(60000); + }); +});