diff --git a/docs/archive/experiments/vol-1-chapters/README.md b/docs/archive/experiments/vol-1-chapters/README.md index 376d9ee..dbfb768 100644 --- a/docs/archive/experiments/vol-1-chapters/README.md +++ b/docs/archive/experiments/vol-1-chapters/README.md @@ -93,7 +93,7 @@ Experiments 35–41 and Experiment 41 Conclusion — handoff validation, behavio Fidelity: Exact contiguous copy. -Tranches 1, 2, and 3 have now been extracted. +Tranches 1, 2, 3, and 4 have now been extracted. The original monolithic log remains intact and authoritative while extraction is incomplete. ## Tranche 3 @@ -124,6 +124,21 @@ Experiments 47–51 — shared-anchor coherence diagnostic, unknown relationship Fidelity: Exact contiguous copy. +## Tranche 4 + +### Chapter 9 +Path: +docs/archive/experiments/vol-1-chapters/ch9/experiments-52-to-52i.md + +Original source: +docs/design-evolution-log.md lines 3535–4980 + +Contents: +Experiments 52–52I and Correction to Experiment 52B Conclusion — semantic interpretation generalisation, evaluation using existing Qwen model, separate meaning from relevance labels, normalisation into decision-relevance contract, supports_decision/could_change_decision boundary coherence, ambiguity normalisation failures, ambiguous meaning filled with domain expectations, ambiguity wording resistance, grounding rule for relationship strength invention, and provenance boundary analysis. + +Fidelity: +Exact contiguous copy. + ## Refactor status Only tranches 1 and 2 have been extracted. diff --git a/docs/archive/experiments/vol-1-chapters/ch9/experiments-52-to-52i.md b/docs/archive/experiments/vol-1-chapters/ch9/experiments-52-to-52i.md new file mode 100644 index 0000000..036c0bd --- /dev/null +++ b/docs/archive/experiments/vol-1-chapters/ch9/experiments-52-to-52i.md @@ -0,0 +1,1446 @@ +## Experiment 52 — Can Semantic Interpretation Generalise Decision Relevance Beyond Keywords? (2026-08-07) + +### Objective + +Test whether a small, passive semantic interpretation step can judge whether an unknown matters to a stated decision more reliably than the existing keyword-based decision-relevance classifier. Specifically: can the same decision-relevance contract work across paraphrases and different domains when the language is interpreted for meaning rather than matched against known phrases? + +### Hypothesis + +A semantic interpreter given only `{ decisionTarget, unknown }` may classify decision relevance more consistently across different wording and domains than the current deterministic keyword rules. The experiment may also show that semantic interpretation is inconsistent, overconfident, or difficult to constrain. Either result would be useful. + +### Semantic Contract + +The semantic interpreter receives: +```js +{ decisionTarget, unknown } +``` + +And returns exactly one of the existing four categories: +```js +{ relevance: "could_change_decision" | "supports_decision" | "unlikely_to_change_decision" | "cannot_determine", reason: "short factual explanation" } +``` + +No new categories. No chain-of-thought. The `reason` is a single short explanation of the relationship between the unknown and the decision. + +### Interpretation Instruction (domain-neutral, identical for all domains) + +> Given a decision and one unanswered question, classify whether resolving that question could directly change the decision, would provide useful support for the decision, is unlikely to affect the decision, or cannot be determined from the information provided. + +No domain-specific examples, no keyword mentions. The same instruction was used for both Domain A (market entry) and Domain B (community event). + +### Context Pack Used + +Engine Experiment Work pack from `docs/task-context-packs.md`. + +### Additional Documents Loaded and Why + +- `lib/graph/question-decision-relevance.js` — to understand the deterministic baseline classifier being tested +- `tests/graph/decision-relative-coherence.test.js` (Exp 51) — to reuse the test cases and confirm regression stability +- Experiment 51 entry in `docs/design-evolution-log.md` — to establish what Exp 51 found (language-sensitive keyword matching) and provide test cases for comparison + +### Semantic Infrastructure Used + +The repository has `lib/llm/provider.js` which calls Ollama `/api/chat` with `format: "json"`. For the experiment, a minimal inline helper (10 lines in the test file) was created — it mirrors the same fetch-to-Ollama pattern without introducing production infrastructure. No new module was created. + +### Evaluation Result: Infrastructure Limitation + +**Ollama is not running on this machine.** `OLLAMA_BASE_URL` is unset and no process listens on port 11434. The semantic interpretation cases (18 test cases × 3 runs each) could not be executed against a live model. + +Per the experiment constraint: +> "If no existing helper can make this small request without substantial architecture work: document that dependency as the experiment result." + +The helper was created inline in the test file using the same Ollama `/api/chat` + `format: json` pattern as the production provider. The infrastructure exists (same API contract), but is not currently running. **This is a valid experimental outcome, not a test bug.** Model failures during the experiment were recorded as `cannot_determine` with reason `model_failure: ` — not silently repaired. + +### Deterministic Baseline Results (Exp 51 classifier, unchanged) + +Against Domain A (European market entry): +- Known phrasing ("whether to enter"): classified as `could_change_decision` ✓ +- Compliance phrasing: classified as `supports_decision` ✓ +- Paraphrased coherent ("would enough people want it"): classified as `cannot_determine` ✗ +- Paraphrased unrelated ("which firms set standard"): classified as `cannot_determine` ✓ + +The deterministic classifier continues to fail on paraphrases and new domains — exactly as Experiment 51 established. This is the baseline that semantic interpretation is being compared against. + +### Semantic Interpretation Results + +Not obtained — Ollama was not available. The test file (`tests/graph/decision-relevance-semantic.test.js`) contains the complete contract, all fixed human reference labels, three-run stability checks, and cross-domain comparison logic. When Ollama is available on port 11434 with a JSON-capable model (e.g., `llama3.1`), rerunning: + +```bash +npx vitest run tests/graph/decision-relevance-semantic.test.js +``` + +will exercise the semantic interpreter against all test cases. + +### Paraphrase Results + +Not obtained. The semantic contract and paraphrase test cases are in place. Expected outcomes (based on hypothesis): +- Coherent paraphrase ("would enough people there actually want what we offer?") → `could_change_decision` (semantic generalisation) +- Unrelated paraphrase ("which analytics firms set the industry standard?") → `cannot_determine` or `unlikely_to_change_decision` + +### Could-change versus supports Distinction + +Not evaluated. The semantic interpreter must distinguish between direct decision-changing questions and supporting-evidence questions. This requires model execution against Domain B where coherent cases split between these categories. + +### Repeatability Result + +Not obtained (no model). The test file runs each case exactly three times and classifies stability as `stable` or `unstable`. + +### Questionable or Unsupported Findings + +The core finding here is an infrastructure gap: the semantic interpretation hypothesis cannot be tested without an Ollama instance with JSON-capable model support. This is a testing environment limitation, not a failure of the experimental design. + +### Experiment Conclusion + +**Experiment could not be completed with existing infrastructure.** The test file documents the complete semantic contract, evaluation set, and human reference labels. When Ollama (`ollama serve`) is available on port 11434, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will complete the comparison against the deterministic baseline. + +### Focused Test Result + +| Test File | Tests | Passed | Notes | +|---|---|---|---| +| `decision-relevance-semantic.test.js` (Exp 52) | 48 | 15 / 33 fail | 15 pass = deterministic guardrails; 33 fail = Ollama not available | + +### Regression / Validation Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 | +| `question-decision-relevance.test.js` (Exp 21) | 25 | 25 | + +All existing tests unchanged. No regression introduced. + +### Documentation Updated + +- `docs/design-evolution-log.md`: Experiment 51 interpretation corrected; Experiment 52 added +- `docs/current-handoff.md`: Return-to-work note updated + +### Confirmation Production Decision-Relevance Classifier Remained Unchanged + +The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass). The test file includes explicit assertions that known patterns return their original classifications unchanged. + +### Confirmation No Semantic Logic Entered Active Runtime + +The semantic helper is defined exclusively within `tests/graph/decision-relevance-semantic.test.js` as a test-level function. It is never imported by production code. No runtime caller was wired. + +### Confirmation Assessor, Behaviour Selection, Graph Construction and Fixtures Remained Unchanged + +No assessor files loaded or modified. No Behaviour Selection files loaded or modified. No graph construction files loaded or modified. No fixtures loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`. + +### Confirmation Active Engine Behaviour Remained Unchanged + +The decision-relevance classifier has no callers outside its own module. No active user-facing behaviour changed. The semantic helper was never wired into the engine under test. + +--- + +## Experiment 52A — Recover Semantic Evaluation Using Existing Project Configuration (2026-08-07) + +This is a recovery and validation of Experiment 52, not a new reasoning experiment. Its purpose is to determine why the semantic test helper did not use the project's existing configuration mechanism and correct it. + +### Investigation Findings + +| Question | Finding | +|---|---| +| Where is `OLLAMA_BASE_URL` actually loaded? | Production reads directly from `process.env.OLLAMA_BASE_URL`. No production code uses `getConfig()` for this — it reads the env var directly (same as `.env.local`). | +| Does `.env.local` already contain the correct host? | Yes: `http://192.168.1.111:11434`. Ollama confirmed running there with `qwen-claude:latest`. | +| Why did the semantic helper use localhost? | The test helper had a hardcoded fallback: `process.env.OLLAMA_BASE_URL \|\| "http://localhost:11434"`. When vitest ran without `OLLAMA_BASE_URL` in its process env, it silently connected to localhost instead of failing fast. | +| Was provider logic duplicated? | Partially. The test helper re-implements the same fetch-to-Ollama pattern (intentionally, as a minimal inline helper). But the configuration *resolution* diverged: hardcoded defaults instead of using `process.env`. | +| Was configuration bypassed? | Yes — two issues: (1) `OLLAMA_BASE_URL` defaulted to localhost instead of `process.env.OLLAMA_BASE_URL \|\| undefined`, and (2) `EXPERIMENT_52_MODEL` was introduced as a new env var with hardcoded `"llama3.1"` default, bypassing the project's `OLLAMA_MODEL` config in `.env.local`. | +| Is any production code incorrect? | No. Production `lib/llm/provider.js:100` reads from `process.env.OLLAMA_BASE_URL` correctly. `.env.local` has the correct values. Config module validates them via Zod. | +| What is the smallest correction? | (a) Remove localhost fallback so helper fails fast when config is missing, matching production behaviour. (b) Replace `EXPERIMENT_52_MODEL` with existing `OLLAMA_MODEL`. (c) Add dotenv loading from `.env.local` in the test file so vitest can access the project's configuration source. | + +### Smallest Correction Applied + +**File:** `tests/graph/decision-relevance-semantic.test.js` + +Three changes, all in the test helper only: + +1. Removed hardcoded `|| "http://localhost:11434"` fallback — now throws when `OLLAMA_BASE_URL` is missing (matches production). +2. Replaced `process.env.EXPERIMENT_52_MODEL \|\| "llama3.1"` with `process.env.OLLAMA_MODEL \|\| "llama3.1"` — uses project config, not an experiment-specific variable. +3. Added `dotenv.config({ path: ".env.local" })` at the top of the test file — enables vitest to access the project's configuration source (the same source Next.js uses). + +### Configuration Source Resolved + +- Ollama base URL: `http://192.168.1.111:11434` (from `.env.local`) +- Model: `qwen-claude:latest` (from `.env.local`, via `process.env.OLLAMA_MODEL`) + +### Experimental Result: Ollama Performance + +Ollama at 192.168.1.111 responds correctly with `format:json` support and `qwen-claude:latest` available. However, per-request latency averages ~82 seconds (measured via direct API test). The semantic test requires 33 cases × 3 runs = 99 inference calls — impractical to execute (~135 hours estimated). + +**This is a valid experimental outcome:** the configuration recovery succeeded, but the remote Ollama server's performance prevents semantic execution within reasonable time. The infrastructure path is correct; the bottleneck is inference speed on the remote host. + +### Focused Test Result (Experiment 52A) + +| Test File | Tests | Passed | Notes | +|---|---|---|---| +| `decision-relevance-semantic.test.js` (Exp 52 infra fix only, deterministic subset) | Config verified | ✅ | Dotenv loads `.env.local`; Ollama reachable at configured URL; no hardcoded localhost | +| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 | All pass. No production code changed. | + +### Regression Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 | + +All existing tests unchanged. No regression introduced. + +### Production Provider Unchanged + +- `lib/llm/provider.js`: 0 lines changed +- `lib/config.js`: 0 lines changed +- `lib/analysis.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed + +### Duplicate Helper Status + +Retained (not removed). The inline test helper is appropriate for a one-shot evaluation and does not duplicate production logic — it merely mirrors the same fetch-to-Ollama pattern. The configuration *resolution* inside it has been corrected to use the project's existing mechanism. + +### Conclusion + +Experiment 52A resolved the configuration root cause. The semantic helper now uses exactly the same environment variable resolution as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. With a faster Ollama instance or model, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will execute the semantic comparison as Experiment 52 defined. + +**Status: Pending Rob's review.** + +--- + +## Experiment 52B — Small Semantic Probe With the Existing Qwen Model (2026-08-07) + +Experiment 52A recovered configuration but deferred semantic execution due to latency (~82s/request makes 99 calls impractical). This experiment reduces the evaluation to the smallest useful live probe: six cases, one call each. + +### Objective + +Using the existing configured `qwen-claude:latest` model, does semantic interpretation handle a handful of paraphrases and cross-domain cases better than the deterministic keyword classifier? + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Semantic instruction | Same conceptual instruction as Exp 52, with explicit enum added so the model outputs valid category values | + +### Six Cases Evaluated + +| Case | Domain | Question | Human Reference | Purpose | +|---|---|---|---|---| +| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | could_change_decision | Easy in-domain test | +| 2 | A (market) — familiar unrelated | "Can two senior staff members resolve their ongoing disagreement?" | unlikely_to_change_decision | Reject obviously unrelated | +| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | could_change_decision | Known deterministic failure | +| 4 | B (event) — relevant | "Whether there is sufficient weather risk for an outdoor event in September" | could_change_decision | Cross-domain generalisation | +| 5 | B (event) — supporting | "What insurance requirements apply for hosting the event outdoors" | supports_decision | Distinguish decisive vs supportive | +| 6 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | unlikely_to_change_decision | Reject non-relevant in new domain | + +### Results + +#### Human Reference Labels (fixed before evaluation) + +| Case | Human Ref | +|---|---| +| 1 | could_change_decision | +| 2 | unlikely_to_change_decision | +| 3 | could_change_decision | +| 4 | could_change_decision | +| 5 | supports_decision | +| 6 | unlikely_to_change_decision | + +#### Deterministic Baseline Results + +| Case | Deterministic Result | Matches Human Ref? | +|---|---|---| +| 1 | could_change_decision | ✓ | +| 2 | cannot_determine | ✗ | +| 3 | cannot_determine | ✗ | +| 4 | cannot_determine | ✗ | +| 5 | cannot_determine | ✗ | +| 6 | cannot_determine | ✗ | + +**Deterministic agreement with human reference: 1/6** (only the familiar in-domain case matched) + +#### Semantic Results (qwen-claude:latest) + +| Case | Semantic Result | Latency (ms) | Matches Human Ref? | +|---|---|---|---| +| 1 | could_change_decision | 14001 | ✓ | +| 2 | unlikely_to_change_decision | 15633 | ✓ | +| 3 | could_change_decision | 9455 | ✓ | +| 4 | could_change_decision | 26992 | ✓ | +| 5 | could_change_decision | 14658 | ✗ (model classified as decisive rather than supportive — defensible for insurance constraints) | +| 6 | unlikely_to_change_decision | 13859 | ✓ | + +**Semantic agreement with human reference: 5/6** + +### Inference Timing + +- Total inference time: ~94,598 ms (≈95 seconds) +- Average per call: ~15,766 ms (~16 seconds) +- Fastest call: 9,455 ms (Case 3 — paraphrase) +- Slowest call: 26,992 ms (Case 4 — cross-domain) +- All six calls completed successfully + +### Key Findings + +1. **Semantic interpretation correctly handled the known keyword failure (Case 3).** The deterministic classifier returned `cannot_determine` for "Would enough people there actually want what we offer?" — a paraphrase of "Whether to enter the European market for analytics tools." The semantic model classified it as `could_change_decision`, agreeing with human reference. + +2. **Semantic interpretation generalised to a second domain (Cases 4–6).** Despite being trained on market-entry vocabulary, the model correctly classified weather-risk as relevant and board-chairs as unrelated for an outdoor-community-event decision. + +3. **Deterministic classifier cannot generalise.** On all four unseen cases (2–6), the deterministic baseline returned `cannot_determine`. It only matched human reference on the one in-domain case it was trained to recognise. + +4. **One defensible disagreement (Case 5).** The model classified insurance requirements as `could_change_decision` rather than `supports_decision`. For an outdoor event, uncovered insurance costs can make the decision infeasible — so treating it as potentially decisive is a reasonable interpretation. + +5. **Latency improved vs earlier measurements.** Average ~16s/call versus ~82s reported in Experiment 52A. Possible server load variation or model warm-up effects. + +### Agreement Counts + +- Semantic agreement with human reference: **5/6** +- Deterministic agreement with human reference: **1/6** + +### Did Semantic Interpretation Improve Generalisation? + +**Yes.** On this small probe, semantic interpretation correctly classified all four in-domain cases (1–3) plus the cross-domain relevant case (4). The deterministic classifier could only classify the one in-domain training-vocabulary case. + +### Is a Repeatability Experiment Justified? + +**Partially.** The evidence on paraphrase generalisation and cross-domain relevance is strong enough to justify confidence. However: +- This was a single-run probe with `qwen-claude:latest` — stability across runs was not tested. +- The model has the right intuition but tends toward conservative categories (Classified supportive insurance question as decisive). +- A repeatability experiment should test whether results hold across different questions and model variants. + +### Limitations + +- Single-run per case — no stability measurement. +- One model only (`qwen-claude:latest`) — does not generalise to other models. +- Six cases is informative but not statistically robust. +- Remote host latency makes large-scale testing expensive in wall-clock time. +- The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52) because qwen-claude:latest needs explicit category labels rather than prose descriptions. + +### Conclusion + +**"Semantic interpretation shows clear improvement in this small probe"** + +The semantic model correctly classified 5 of 6 cases against human reference, including the critical paraphrase case (Case 3) and both cross-domain cases where it generalised beyond training vocabulary. The deterministic classifier scored 1/6 on the same cases. + +No production code changed. No semantic logic entered active runtime. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the six-case test and results, then `docs/design-evolution-log.md` Experiment 52B section. + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 | +| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 | + +### Regression Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 | + +All existing tests unchanged. No regression introduced. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- `lib/llm/provider.js`: 0 lines changed +- `lib/config.js`: 0 lines changed +- `lib/analysis.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed + +### Files Modified + +- `tests/graph/decision-relevance-semantic.test.js` — replaced Exp 52 corpus with 6-case Exp 52B probe +- `docs/design-evolution-log.md` — added Experiment 52B section +- `docs/current-handoff.md` — updated return-to-work note + + +## Correction to Experiment 52B Conclusion (2026-08-07) + +The qualitative generalisation result is stronger evidence than the headline 5/6 score: + +- Semantic interpretation handled a known paraphrase that the keyword classifier missed; +- Semantic interpretation generalised to a second domain (Cases 4–6); +- Clearly unrelated questions were recognised as unrelated; +- Six live calls completed successfully using the existing `qwen-claude:latest` model. + +However, two experimental-control issues were exposed: + +1. The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52, but this does influence which category the model selects); +2. The disputed insurance case (Case 5 in Exp 52B) was defensible either way — for outdoor events, uncovered insurance costs can make a decision infeasible, so treating it as potentially decisive is reasonable. + +Therefore: the qualitative generalisation result (paraphrase handling + cross-domain relevance) is stronger evidence than the headline score of 5/6. The experimental design should be refined before further quantitative claims. + + +## Experiment 52C — Separate Semantic Meaning From Relevance Labels (2026-08-07) + +Experiment 52B showed encouraging semantic results but exposed two control issues: the instruction contained explicit enum values that could bias category selection, and the qualitative generalisation result deserved more weight than the headline score. This experiment separates understanding from labelling into two independent calls per case. + +### Objective + +Test whether `qwen-claude:latest` understands the relationship between a question and a decision in ordinary language **before** forcing that understanding into the existing four decision-relevance categories. + +> Is the model's semantic understanding better than its ability to express that understanding using our predefined enum labels? + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Meaning-mode instruction | "Explain in one short sentence how answering this question would or would not matter to the stated decision. Do not classify it, score it, or use predefined category names." (+ JSON schema hint `{relationship: "..."}` for output format) | +| Enum-mode instruction | Same constrained instruction as Exp 52B (four categories) | + +### Five Fixed Cases + +| Case | Domain | Question | Expected Relationship | Expected Enum | +|------|--------|----------|----------------------|---------------| +| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | Resolving demand could materially change whether market entry is worthwhile. | `could_change_decision` | +| 2 | A (market) — familiar supporting | "Whether European regulatory compliance is suitable for our analytics product" | Compliance suitability is an important condition supporting the decision, but not itself the whole decision. | `supports_decision` | +| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | Another way of asking whether enough demand exists for entering the market. | `could_change_decision` | +| 4 | B (event) — second-domain relevant | "Whether there is sufficient weather risk for an outdoor event in September" | Weather risk could materially affect whether holding the event outdoors is viable. | `could_change_decision` | +| 5 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | Board chairs has no meaningful bearing on outdoor event decision. | `unlikely_to_change_decision` | + +### Results + +#### Meaning-mode responses + +| Case | Meaning captured intended relationship? | Mode A response (truncated to 80 chars) | +|------|----------------------------------------|------------------------------------------| +| 1 | ✓ | "Answering this question directly determines whether entering the European market..." | +| 2 | ✓ | "Answering this question is critical because European data regulations will deter..." | +| 3 | ✓ | "Answering this question is critical because confirming sufficient customer deman..." | +| 4 | ✓ | "Answering this question is essential because the level of weather risk directly ..." | +| 5 | ✓ | "Answering this question is irrelevant because replacing meeting room chairs has ..." | + +**Meaning-correct count: 5/5** + +#### Enum-mode responses + +| Case | Expected Enum | Mode B Result | Reason (truncated) | Match? | +|------|--------------|---------------|-------------------|--------| +| 1 | `could_change_decision` | `could_change_decision` | "Customer demand is a fundamental viability factor..." | ✓ | +| 2 | `supports_decision` | `could_change_decision` | "Meeting European data regulations is a legal prerequisite... confirming non-compliance would make market entry unviable" | ✗ | +| 3 | `could_change_decision` | `could_change_decision` | "Validating sufficient customer demand is fundamental..." | ✓ | +| 4 | `could_change_decision` | `could_change_decision` | "Weather risk is a primary factor for hosting outdoors..." | ✓ | +| 5 | `unlikely_to_change_decision` | `unlikely_to_change_decision` | "The question addresses board furniture maintenance..." | ✓ | + +**Enum-match count: 4/5** + +#### Meaning-correct / enum-mismatch cases + +**Case 2**: Mode A correctly identified compliance as a supporting condition ("critical because European data regulations..."). Mode B classified it as `could_change_decision` with reason noting "legal prerequisite" and "non-compliance would make market entry unviable." The model treated regulatory compliance as potentially decisive rather than supportive — defensible interpretation for a SaaS product in Europe where non-compliance blocks operation entirely, but it diverges from the expected `supports_decision` label. This is a case where both meaning and reason are correct, but enum differs. + +### Inference Timing + +- Total inference time: ~147,050 ms (≈147 seconds) +- Average per call: ~14,705 ms (~15 seconds) +- Fastest call: ~9,500 ms +- Slowest call: ~27,000 ms +- All 10 calls completed successfully + +### Key Findings + +1. **Meaning mode scored 5/5 — perfect on this probe.** Free-language explanations captured the intended relationship for all five cases without any category hints. + +2. **Enum classification scored 4/5.** One mismatch (Case 2) where both meaning and reason described supporting conditions correctly, but the model chose `could_change_decision` instead of `supports_decision`. + +3. **The known paraphrase retained its meaning without enum hints (Case 3).** The model explained demand relevance in free language identical to Case 1's approach — no category priming was needed. + +4. **Cross-domain generalisation held without enum hints (Case 4).** Weather risk was correctly explained as materially affecting the outdoor event decision, matching Case 1's pattern of causal explanation. + +5. **Unrelated case remained clearly unrelated (Case 5).** Free-language mode explicitly stated irrelevance ("Answering this question is irrelevant because..."), confirming the model does not force false connections when none exist. + +6. **Supplying enum names did materially change interpretation.** When categories were supplied, the model tended to be more conservative in its classifications — e.g., Case 2's compliance question was classified as potentially decisive rather than supportive, likely because "legal prerequisite" triggered a higher-stakes category choice. This is evidence that semantic interpretation and normalisation may benefit from being separate conceptual jobs. + +### Limitations + +- Single-run probe with `qwen-claude:latest` — stability not measured. +- Five cases only — sufficient for a diagnostic but not statistically robust. +- Remote host latency (~15s/call) limits scope of repeatability testing. +- Meaning-mode evaluation used keyword regex patterns rather than LLM-based assessment, which itself has limitations. +- Case 2's supporting-vs-decisive boundary is inherently fuzzy; the disagreement may reflect legitimate interpretive difference rather than error. + +### Conclusion + +**"Meaning is stronger than enum classification in this probe."** + +The model correctly explained how every question relates to its decision in free language (5/5) while misclassifying one case into enum labels (4/5). The single mismatch (Case 2) was still semantically defensible — both modes described supporting conditions accurately, only the label diverged. This supports treating semantic interpretation and engine-contract normalisation as separate conceptual jobs: the model understands relationships reliably even when it struggles to express that understanding using our predefined categories. + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-semantic-normalisation.test.js` (Exp 52C) | 29 | 29 | + +### Regression Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | + +All existing tests pass. No regression introduced. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- `lib/llm/provider.js`: 0 lines changed +- `lib/config.js`: 0 lines changed +- `lib/analysis.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed + +### Files Created + +- `tests/graph/decision-relevance-semantic-normalisation.test.js` — Exp 52C probe (29 tests, 10 live calls) + +### Files Modified + +- `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section +- `docs/current-handoff.md` — updated return-to-work note + +## Experiment 52D — Can Free-Language Meaning Be Normalised Into the Existing Decision-Relevance Contract? (2026-08-07) + +Experiment 52C found that free-language semantic understanding scored 5/5 while enum classification scored 4/5, with the compliance case consistently misclassified as `could_change_decision` instead of `supports_decision`. This experiment isolated the normalisation step: the model receives **only** a correct free-language relationship statement (no decision target, no question) and maps it into the existing four categories. + +### Objective + +Test whether a separate normalisation step — given an already-correct meaning statement — can reliably map that meaning into the engine's existing enum contract without keyword matching or altering the meaning itself. + +> Once the meaning has already been understood correctly, can we reliably translate that meaning into the engine's existing categories? + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | "You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories..." (+ definitions + JSON schema) | +| Input per case | `{"relationship": ""}` only | +| No input | Original decision target, original unknown question, domain examples, or previous model outputs | + +### Domain-Neutral Category Definitions Used + +These faithfully reflect the production contract in `lib/graph/question-decision-relevance.js`: + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Five Fixed Relationship Statements + +| Case | Source | Relationship Statement (verbatim) | Expected Enum | +|------|--------|-----------------------------------|---------------| +| 1 — Demand | Exp 52C Case 1 | "Answering whether genuine customer demand exists could materially determine whether entering the European market is worthwhile." | `could_change_decision` | +| 2 — Compliance | Exp 52C Case 2 | "Knowing whether the product can satisfy European regulatory requirements is an important condition that supports the market-entry decision." | `supports_decision` | +| 3 — Paraphrased demand | Exp 52C Case 3 | "Knowing whether enough people there actually want the product would materially affect whether entering that market is worthwhile." | `could_change_decision` | +| 4 — Weather (cross-domain) | Exp 52C Case 4 | "Knowing the weather risk could materially determine whether holding the community event outdoors is viable." | `could_change_decision` | +| 5 — Unrelated chairs | Exp 52C Case 5 | "Whether the board replaces its meeting-room chairs has no meaningful bearing on whether the community event should be held outdoors." | `unlikely_to_change_decision` | + +### Results + +| Case | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency | +|------|--------------|---------------|--------|-------------------|---------| +| 1 — Demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that answering could materially determine whether market entry is worthwhile..." | 15,937ms | +| 2 — Compliance | `supports_decision` | `could_change_decision` | ✗ mismatch | "Regulatory compliance is a fundamental viability constraint for market entry, functioning as a go/no-go condition where failure to satisfy it would directly reverse the proposed action." | 28,021ms | +| 3 — Paraphrased demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that the answer would materially affect whether entering the market is worthwhile..." | 13,239ms | +| 4 — Weather (cross-domain) | `could_change_decision` | `could_change_decision` | ✓ match | "The relationship explicitly states that weather risk materially determines the event's viability..." | 8,419ms | +| 5 — Unrelated chairs | `unlikely_to_change_decision` | `unlikely_to_change_decision` | ✓ match | "The statement explicitly notes that answering the question has no meaningful bearing on the decision..." | 8,461ms | + +**Enum-match count: 4/5** + +### Key Findings + +1. **Normalisation matched expected enum on 4/5 cases.** The same four categories normalised cleanly when the meaning was already correct. + +2. **The compliance boundary disagreement persisted.** Case 2 (regulatory requirements as a supporting condition) still maps to `could_change_decision`. The model's reason — "Regulatory compliance is a fundamental viability constraint... functioning as a go/no-go condition" — is faithful to the relationship statement itself, not an invented interpretation. Both `supports_decision` and `could_change_decision` are defensible: compliance *supports* the decision by building evidence, but non-compliance would *reverse* it (blocking entry entirely). The model chose the latter reading because the category definition for `could_change_decision` includes "go/no-go condition" which aligns with a regulatory blocker. + +3. **The paraphrase-derived meaning normalised identically to the familiar demand meaning.** Cases 1 and 3 both returned `could_change_decision` with matching reasoning ("materially affect/determine whether entering the market is worthwhile"). Meaning preservation through paraphrase held when only normalisation was tested. + +4. **Cross-domain generalisation held.** The weather case (Case 4) normalised correctly to `could_change_decision` without any domain-specific tuning. The model applied the category definitions consistently across domains. + +5. **The unrelated relationship normalised correctly.** Case 5 mapped cleanly to `unlikely_to_change_decision` with a faithful reason referencing "no meaningful bearing." + +6. **The model did not attempt to reinterpret missing context.** All five reasons were grounded in the supplied relationship statement. None fabricated information that was not present in the input. + +7. **The four-category contract is sufficiently clear for normalisation** in three of four boundary zones (demand, weather, unrelated all normalised correctly). The remaining ambiguity lies specifically at the `supports_decision` ↔ `could_change_decision` boundary. + +8. **Evidence points to category definitions as the remaining problem.** Not semantic understanding (already solved by Exp 52C's meaning mode), not normalisation mechanism (which works for 4/5 cases), but the definition of `could_change_decision` which includes "go/no-go condition" — a phrase that both a compliance blocker and a demand question could satisfy. + +### Compliance Boundary Analysis + +The persistent disagreement on Case 2 is not a model error or a normalisation failure. It is evidence of genuine ambiguity in the category definitions: + +- **Relationship statement (meaning):** "...is an important condition that supports the market-entry decision." +- **Model's reading:** "Regulatory compliance is a fundamental viability constraint... go/no-go condition." +- **Expected:** `supports_decision` — because the relationship says "supports" +- **Actual:** `could_change_decision` — because non-compliance would reverse the action + +Both readings are faithful to the same relationship statement. The model applied the category definitions literally: if a condition's negation would reverse the decision, it is a "go/no-go condition" under `could_change_decision`. This interpretation is internally consistent and not an error. **Reference-category boundary appears questionable.** + +### Inference Timing + +- Total inference time: 74,077 ms (~74 seconds) +- Average per call: ~14,815 ms (~15 seconds) +- Fastest call: 8,419 ms (Case 5 — unrelated chairs) +- Slowest call: 28,021 ms (Case 2 — compliance) + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-normalisation.test.js` (Exp 52D) | 20 | 20 | + +### Regression Result + +Regression tests ran against Exp 52C (`decision-relevance-semantic-normalisation.test.js`) and core classifier (`question-decision-relevance.test.js`) — no regressions introduced. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified + +### Files Created + +- `tests/graph/decision-relevance-normalisation.test.js` — Exp 52D probe (20 tests, 5 live calls) + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Five cases only — sufficient for a diagnostic but not statistically robust. +- Remote host latency (~15s/call) limits scope of repeatability testing. +- The compliance boundary disagreement was not resolved; further analysis is needed on whether the existing definitions can distinguish "supports" from "could change" when both interpretations are faithful to the same relationship statement. + +### Conclusion + +**"Normalisation works but one category boundary remains ambiguous."** + +The model correctly mapped four of five correct meaning statements into the expected enum categories when given only the relationship statement and the category definitions — no original decision context was needed. The single remaining disagreement (Case 2, compliance) is not a normalisation failure or a semantic understanding problem: both `supports_decision` and `could_change_decision` are faithful readings of the same relationship statement under the current definitions. The evidence suggests the remaining problem lies in **category definitions** — specifically, the phrase "go/no-go condition" in `could_change_decision` captures compliance blockers that should arguably be classified as supporting evidence rather than decision-reversing conditions. + +### Status + +**Closed.** Pending resolution by Experiment 52E: does the existing category boundary hold when relationship statements explicitly distinguish a blocker from supporting evidence? + +--- + +## Experiment 52E — Is the `supports_decision` / `could_change_decision` Boundary Actually Coherent? (2026-08-07) + +Experiment 52D found that normalisation works cleanly on four of five meaning statements, but one compliance case consistently misclassified as `could_change_decision`. The open question was whether this reflected an ambiguous category boundary or a poorly specified reference statement. Experiment 52E tests the boundary directly using three explicit contrast pairs (blocker vs supporting-evidence) across three distinct domains, with no domain overlap from previous experiments except market entry (Pair 1). + +### Objective + +Test whether the existing distinction between `could_change_decision` and `supports_decision` holds consistently when relationship statements explicitly differentiate a go/no-go blocker from supporting evidence. + +> Can the current category definitions reliably distinguish a condition that could reverse a decision from evidence that merely strengthens confidence in it? + +This is a passive contract-boundary experiment using clearer contrast statements than Experiment 52D's compliance case. + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Same as Experiment 52D — domain-neutral, category definitions included, JSON schema enforced | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples. | +| No input | Original decision target, original unknown question, domain examples, or previous model outputs | + +### Domain-Neutral Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Three Contrast Pairs + +**Pair 1 — Market Entry (one new domain-referenced pair; one cross-domain)** + +| Item | Relationship Statement | Expected Enum | +|------|----------------------|---------------| +| 1A (blocker) | "If the product cannot legally satisfy the required European regulations, entering the market cannot proceed." | `could_change_decision` | +| 1B (supporting) | "Independent customer interviews showing strong interest would increase confidence that entering the European market is worthwhile, but would not determine the decision by themselves." | `supports_decision` | + +**Pair 2 — Community Event** + +| Item | Relationship Statement | Expected Enum | +|------|----------------------|---------------| +| 2A (blocker) | "If the forecast shows dangerous weather conditions on the event date, holding the event outdoors would no longer be viable." | `could_change_decision` | +| 2B (supporting) | "Positive feedback from previous attendees about outdoor events would strengthen confidence in choosing an outdoor venue, but would not decide the issue by itself." | `supports_decision` | + +**Pair 3 — Hiring Decision (fresh domain)** + +| Item | Relationship Statement | Expected Enum | +|------|----------------------|---------------| +| 3A (blocker) | "If the candidate does not hold the legally required professional licence, they cannot be appointed to the role." | `could_change_decision` | +| 3B (supporting) | "Strong references from previous employers would increase confidence that the candidate is suitable, but would not determine the hiring decision alone." | `supports_decision` | + +### Results + +| Pair | Item | Relationship Statement (truncated) | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency | +|------|------|-----------------------------------|---------------|---------------|--------|-------------------|---------| +| 1A | blocker | "If the product cannot legally satisfy..." | `could_change_decision` | `could_change_decision` | match | "Legal compliance defined as strict prerequisite, functioning as go/no-go condition" | 11,283ms | +| 1B | supporting | "Independent customer interviews showing..." | `supports_decision` | `supports_decision` | match | "Explicitly increases confidence but would not alone determine or reverse the decision" | 11,011ms | +| 2A | blocker | "If the forecast shows dangerous weather..." | `could_change_decision` | `could_change_decision` | match | "Dangerous weather defined as condition that would make event non-viable (go/no-go)" | 17,441ms | +| 2B | supporting | "Positive feedback from previous attendees..." | `supports_decision` | `supports_decision` | match | "Strengthens confidence but would not alone determine outcome" | 14,289ms | +| 3A | blocker | "If candidate does not hold licence..." | `could_change_decision` | `could_change_decision` | match | "Mandatory legal requirement serves as definitive go/no-go condition" | 17,588ms | +| 3B | supporting | "Strong references from previous employers..." | `supports_decision` | `supports_decision` | match | "Improves confidence in suitability without being sole determinant" | 12,653ms | + +**Enum-match count: 6/6** + +### Evaluation Questions — Answered + +1. **Did all three direct-blocker cases map to `could_change_decision`?** Yes — 3/3 blockers classified as `could_change_decision`. +2. **Did all three supporting-evidence cases map to `supports_decision`?** Yes — 3/3 supporting-evidence cases classified as `supports_decision`. +3. **Did the same distinction survive across all three domains?** Yes — Market Entry, Community Event, and Hiring Decision all produced clean contrast pairs with consistent categorisation. +4. **Did the model ever treat supporting evidence as a potential decision-reverser?** No — zero supporting-evidence cases were classified as `could_change_decision`. +5. **Did the model ever treat an explicit blocker as merely supportive?** No — zero blocker cases were classified as `supports_decision`. +6. **Does the existing wording create a stable distinction when relationships are unambiguous?** Yes — when the relationship statement explicitly distinguishes a blocker from supporting evidence, the model consistently and correctly applies the category definitions. +7. **Does Experiment 52D's compliance disagreement now look more like a bad reference label, an ambiguous relationship statement, or an ambiguous category boundary?** The most accurate answer is: **an ambiguous relationship statement**. The existing category definitions work cleanly when the input explicitly frames the relationship (as in all six test cases). Experiment 52D Case 2's statement ("...is an important condition that supports the market-entry decision") did not explicitly frame whether compliance was a blocker or supporting evidence — it used "supports" as a verb describing its role but left the go/no-go implication implicit. The model read both meanings, which are both valid under the current definitions. + +### Key Findings + +1. **All six cases classified cleanly.** Every direct-blocker statement mapped to `could_change_decision` and every supporting-evidence statement mapped to `supports_decision` with 100% accuracy across three distinct domains. + +2. **Cross-domain consistency confirmed.** The same distinction held in Market Entry, Community Event, and Hiring Decision — no domain-specific tuning or phrasing was required. Each contrast pair showed a clear category split between the blocker and supporting items. + +3. **The model did not confuse blocker with supporting under any condition.** No supporting-evidence case produced `could_change_decision`, and no blocker case produced `supports_decision`. The boundary held cleanly for unambiguous inputs. + +4. **Experiment 52D's compliance case is resolved as an ambiguous reference statement, not a broken contract.** When the relationship explicitly framed the nature of the condition (as in Pair 1A: "cannot legally satisfy... cannot proceed"), the model correctly classified it as `could_change_decision`. The earlier disagreement arose because the phrase "important condition that supports" did not contain enough signal to distinguish go/no-go from supporting evidence. Both readings were valid — but the input was insufficient to select one definitively. + +5. **The existing category definitions are workable.** The contract does not need modification for cases where the relationship statement is sufficiently explicit. The current definitions ("go/no-go condition" vs "improves confidence") correctly distinguish blockers from supporting evidence when the input provides that distinction. + +### Focused Test Result + +| Test File | Tests | Passed | +|---|---|---| +| `decision-relevance-category-boundary.test.js` (Exp 52E) | 27 | 27 | +| `decision-relevance-normalisation.test.js` (Exp 52D regression, fresh run) | 20 | 20 | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | + +### Regression Result + +Experiment 52D results confirmed on fresh run: still 4/5 matches with case 2 compliance mismatching. This is consistent — the compliance reference wording remains ambiguous between blocker and supporting interpretations. Core classifier (Exp 21, deterministic) continues to produce correct classifications for all test cases with zero regressions. + +### Inference Timing + +- Total inference time: 84,265 ms (~84 seconds) +- Average per call: ~14,044 ms (~14 seconds) +- Fastest call: 11,011 ms (Pair 1B supporting — customer interviews) +- Slowest call: 17,588 ms (Pair 3A blocker — candidate licence) + +### Normalisation Failures + +None. All six relationships normalised cleanly to one of the four existing categories without error or ambiguity. + +### Questionable or Unsupported Findings + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured. +- Six cases only — sufficient for a diagnostic conclusion but not statistically robust. +- Remote host latency (~14s/call) limits scope of repeatability testing. +- Pair 1 (Market Entry) overlaps with Experiment 52D's original domain; however, the reference statements are different enough to provide independent evidence. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-category-boundary.test.js` — Exp 52E probe (27 tests, 6 live calls) + +### Conclusion + +**"Existing boundary is coherent for clear contrast cases."** + +When relationship statements explicitly distinguish a go/no-go blocker from supporting evidence, the existing category definitions produce clean, consistent classification across multiple domains. The experiment confirms that the four-category contract works correctly for unambiguous inputs. Experiment 52D's compliance disagreement was caused by an ambiguous reference statement — not by a broken contract. The phrase "important condition that supports" in the earlier case allowed two equally valid readings (supporting evidence vs go/no-go blocker), whereas the explicit contrast statements used here contained sufficient signal for the model to select the correct category every time. + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Six cases only — a diagnostic, not a statistical study. +- Remote host latency (~14s/call) limits scope of repeatability testing. +- Does not test paraphrase robustness or out-of-vocabulary language for boundary edge cases. + +### Status + +**Closed.** The category boundary is usable for clear contrast cases. Remaining uncertainty: whether less explicit phrasing (between fully ambiguous and fully explicit) still produces consistent results. Pending resolution by Experiment 52F — will genuinely ambiguous relationship statements remain `cannot_determine` or get forced into stronger categories? + +## Experiment 52F — Will the Normaliser Admit When the Category Boundary Is Genuinely Unclear? (2026-08-07) + +Experiment 52E confirmed the existing boundary is coherent for clear contrast cases. The remaining question was whether the contract can **own uncertainty** when the relationship statement itself does not contain enough information to choose cleanly between categories. This experiment tests two genuinely ambiguous regulatory-position statements against `cannot_determine`, using two clear controls to confirm the blocker/supporting boundary still works. + +### Objective + +Test whether the existing normalisation step honestly returns `cannot_determine` for ambiguous relationship statements, or forces them into a stronger category. + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Same as Experiment 52E — no coaching toward any category | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples, no external knowledge. | + +### Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Four Fixed Relationship Statements + +**Case 1 — Clear blocker control** + +Relationship: "If the product cannot satisfy the required regulations, entering the market cannot legally proceed." + +Expected enum: `could_change_decision` + +Purpose: Confirm the known blocker boundary still behaves as Experiment 52E established. + +--- + +**Case 2 — Clear support control** + +Relationship: "Evidence that the product already meets commonly expected regulatory standards would increase confidence in entering the market, but would not determine the decision by itself." + +Expected enum: `supports_decision` + +Purpose: Confirm the known supporting-evidence boundary still behaves cleanly. + +--- + +**Case 3 — Genuinely ambiguous** + +Relationship: "Understanding the regulatory position would be important to the market-entry decision." + +Expected enum: `cannot_determine` + +Purpose: The statement tells us the issue matters but does **not** tell us whether it is a blocker, supporting evidence, or something else. Do not add context. + +--- + +**Case 4 — Ambiguous condition** + +Relationship: "Regulatory compliance is an important condition to consider when deciding whether to enter the market." + +Expected enum: `cannot_determine` + +Purpose: Deliberately resembles the ambiguity exposed in Experiment 52D. It says the condition matters but does not establish whether failure would prevent action or merely affect confidence. + +--- + +### Results + +| Case | Description | Expected Enum | Returned Enum | Match? | Reason | Latency | +|------|-------------|---------------|---------------|--------|--------|---------| +| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | match | "The relationship explicitly identifies regulatory compliance as a mandatory legal requirement for market entry, meaning a negative answer would directly reverse or block the proposed action." | 11,702ms | +| 2 | Clear support control | `supports_decision` | `supports_decision` | match | "The statement explicitly indicates that answering would increase confidence in the decision but would not determine it alone, which directly matches the provided definition of supports_decision." | 11,887ms | +| 3 | Genuinely ambiguous | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory compliance typically acts as a critical go/no-go condition for market entry, meaning its answer could directly reverse or prevent the proposed action." | 17,907ms | +| 4 | Ambiguous condition | `cannot_determine` | `could_change_decision` | mismatch | "The statement identifies regulatory compliance as an important condition for market entry, indicating that meeting or failing it serves as a go/no-go barrier that could directly reverse the decision to proceed." | 7,909ms | + +**Clear-control match count: 2/2** + +**Ambiguous `cannot_determine` count: 0/2** + +### Evaluation Questions — Answered + +1. **Did the clear blocker still map to `could_change_decision`?** Yes — Case 1 classified correctly. +2. **Did the clear supporting statement still map to `supports_decision`?** Yes — Case 2 classified correctly. +3. **Did Case 3 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that "regulatory compliance typically acts as a critical go/no-go condition for market entry," importing external domain knowledge not present in the statement. +4. **Did Case 4 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that regulatory compliance "serves as a go/no-go barrier that could directly reverse the decision to proceed," again importing its own regulatory-domain assumption. +5. **What information in the supplied statement did the reason rely on for Cases 3 and 4?** Neither case's statement says anything about blocking or reversing. The model introduced the concept of "go/no-go barrier" from its domain knowledge that regulation is typically mandatory, not from what either relationship statement actually stated. +6. **Did the model introduce outside assumptions?** Yes. Case 3: "typically acts as a critical go/no-go condition." Case 4: "serves as a go/no-go barrier." These are external-domain assumptions about regulatory compliance, not derivations from the supplied statements. The supplied statements only say the issue is "important" or an "important condition to consider." +7. **Does `cannot_determine` function as a real uncertainty-preserving category in the current normalisation contract?** No — for cases where the model's domain knowledge suggests regulation matters, it bypasses `cannot_determine` entirely and forces the statement into `could_change_decision`. The category exists but is not triggered when the model has strong prior beliefs about the subject matter. +8. **Does Experiment 52D's compliance disagreement now look like something the contract can represent honestly without redefining the categories?** No — Experiment 52F shows that even with deliberately ambiguous phrasing ("important condition to consider"), the contract cannot preserve this uncertainty because the model substitutes its own domain knowledge for the supplied meaning. The existing `cannot_determine` category is not a real escape route when domain priors are strong enough. + +### External-Assumption Findings + +| Case | Grounding classification | Evidence in reason | +|------|------------------------|-------------------| +| 3 (ambiguous) | `introduced_external_assumption` | "typically acts as a critical go/no-go condition" — not present in the statement | +| 4 (ambiguous condition) | `introduced_external_assumption` | "serves as a go/no-go barrier" — not present in the statement | + +Both ambiguous cases introduced external assumptions about regulatory compliance being inherently blocking. The model's reasoning relied on its domain knowledge that regulation = mandatory requirement, not on what either supplied relationship actually said. + +### Key Findings + +1. **Clear controls work.** Cases 1 and 2 confirmed the existing blocker/supporting boundary holds for explicit contrast statements — both matched expected enums correctly. + +2. **`cannot_determine` is bypassed for domain-prior cases.** When the model has strong domain knowledge about regulation (i.e., that it is typically mandatory), it uses that knowledge to classify ambiguous statements as `could_change_decision` instead of honestly returning `cannot_determine`. + +3. **The model substitutes domain knowledge for supplied meaning.** Neither Case 3 nor Case 4's statement says compliance can block the decision. Both say only that it "matters" or is an "important condition." The model added the blocker interpretation from its own regulatory-domain assumptions. + +4. **Experiment 52D's compliance disagreement is confirmed as a contract-level problem.** Experiment 52F reproduces the same pattern: when regulation appears in an ambiguous context, the model forces it into `could_change_decision` because its domain knowledge says regulation is typically blocking — even though the supplied statement does not say that. + +### Focused Test Result + +| Test File | Tests | Passed | Failed | +|---|---|---|---| +| `decision-relevance-ambiguity.test.js` (Exp 52F) | 30 | 27 | 3 | +| `decision-relevance-category-boundary.test.js` (Exp 52E regression) | 27 | 27 | — | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — | + +### Regression Result + +Experiment 52E results confirmed on fresh run: all six cases still classify correctly. The clear blocker/supporting boundary remains intact for explicit contrast statements. Experiment 21 deterministic classifier: zero regressions across all 25 tests. + +### Inference Timing + +- Total inference time: 49,405 ms (~49 seconds) +- Average per call: ~12,351 ms (~12 seconds) +- Fastest call: 7,909 ms (Case 4 — ambiguous condition) +- Slowest call: 17,907 ms (Case 3 — genuinely ambiguous) + +### Normalisation Failures + +No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified both ambiguous cases into `could_change_decision` rather than preserving uncertainty as `cannot_determine`. + +### Questionable or Unsupported Findings + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured. +- Both ambiguous cases use regulatory-domain language — the pattern may differ for other domains where regulation is less of a default assumption. +- The external-assumption diagnostic uses heuristic keyword matching; manual review of reasons confirms both cases introduced domain priors not present in the statements. +- Remote host latency (~12s/call) limits scope of repeatability testing. + +### Conclusion + +**"Current contract sometimes forces ambiguous meaning into stronger categories."** + +The existing four-category contract cannot preserve uncertainty when the model's domain knowledge conflicts with the ambiguity in the supplied statement. For regulatory compliance appearing in an ambiguous context, the model consistently defaults to `could_change_decision` because its domain knowledge says regulation is typically a go/no-go condition — even though the supplied relationship statement does not state this. + +The two clear controls (Cases 1 and 2) confirmed the blocker/supporting boundary still works for explicit contrast statements. But `cannot_determine` does not function as a real uncertainty-preserving category in practice when strong domain priors exist. The model will substitute its own knowledge rather than admit insufficient information from the supplied statement. + +This means Experiment 52D's compliance disagreement is a contract-level problem: the contract has the words `cannot_determine` but no reliable mechanism to trigger it when the model has competing domain beliefs about the subject matter. + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Both ambiguous cases use regulatory-domain language; results may vary for domains with weaker default assumptions. +- External-assumption diagnostic uses heuristic keyword matching of reasoning text. +- Remote host latency (~12s/call) limits scope of repeatability testing. + +### Status + +**Open.** Pending Rob's review. The contract cannot reliably preserve ambiguity when domain priors are strong. Potential resolution paths: (a) modify the normalisation instruction to more strongly anchor the model to "what this statement says" vs "what you know about regulation," (b) add a constraint layer that prevents the model from inferring blocker status without explicit go/no-go language in the statement, or (c) accept that `cannot_determine` is only available when domain priors are weak. No production code has been changed. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-ambiguity.test.js` — Exp 52F probe (30 tests, 4 live calls) + + +## Experiment 52G — Does the Model Fill Ambiguous Meaning With Domain Expectations? (2026-08-07) + +Experiment 52F showed that two ambiguous regulatory statements were forced into `could_change_decision` instead of `cannot_determine`. Both cases used regulation, so it was unknown whether this was a strong regulatory prior or a general tendency to complete ambiguous meaning using domain knowledge. Experiment 52G tests the same structurally identical ambiguity across four different domains to isolate that question. + +### Objective + +Test whether the normaliser's failure to preserve ambiguity in Experiment 52F was specifically caused by strong regulatory knowledge, or whether it more generally fills incomplete relationship statements using its own domain expectations. + +> **When several relationship statements have the same deliberately incomplete structure but refer to different domains, does the model preserve `cannot_determine`, or invent different relevance categories from what it already knows about each subject?** + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Same as Experiment 52F — no coaching toward any category, identical text confirmed | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples. | + +### Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Four Structurally Matched Ambiguous Statements + +All four use the template: **"Understanding [X] would be important to [decision]."** + +| Case | Domain | Relationship Statement | Expected Enum | +|------|--------|----------------------|---------------| +| 1 | Regulation | "Understanding the regulatory position would be important to the market-entry decision." | `cannot_determine` | +| 2 | Weather | "Understanding the weather outlook would be important to the outdoor-event decision." | `cannot_determine` | +| 3 | Employment References | "Understanding what the candidate's references say would be important to the hiring decision." | `cannot_determine` | +| 4 | Customer Feedback | "Understanding what customers think would be important to the product-launch decision." | `cannot_determine` | + +### Results + +| Case | Domain | Expected Enum | Returned Enum | Match? | Reason (summary) | Latency | +|------|--------|---------------|---------------|--------|-------------------|---------| +| 1 | Regulation | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory position as a critical viability factor for market entry, implying go/no-go condition" | 15,979ms | +| 2 | Weather | `cannot_determine` | `could_change_decision` | mismatch | "Weather identified as important to the decision, indicating go/no-go condition that could reverse whether event proceeds" | 17,209ms | +| 3 | Employment Refs | `cannot_determine` | `could_change_decision` | mismatch | "Reference feedback identified as material factor that could reasonably reverse or confirm outcome — go/no-go condition" | 23,499ms | +| 4 | Customer Feedback | `cannot_determine` | `could_change_decision` | mismatch | "Customer sentiment identified as critical go/no-go factor for product launch impacting viability" | 21,218ms | + +**Clear-control match count:** N/A (no controls in this experiment — controlled by 52F) +**Ambiguous `cannot_determine` count: 0/4** + +### Evaluation Questions — Answered + +1. **How many of four ambiguous statements returned `cannot_determine`?** Zero. All four were forced into `could_change_decision`. +2. **Did regulation again become `could_change_decision`?** Yes — consistent with Experiment 52F. +3. **Did weather produce a stronger category from assumed risk?** Yes — the model inferred that "important to [weather]" implies go/no-go relevance to the outdoor-event decision. The supplied statement did not say bad weather would cancel the event; it only said understanding the outlook matters. +4. **Did employment references produce a stronger category from assumed hiring practice?** Yes — the model treated references as a material factor that could "reverse or confirm" the outcome. The statement did not say whether references are decisive, supportive, or routine. +5. **Did customer feedback produce a stronger category from assumed commercial importance?** Yes — the model interpreted "important to [product launch]" as implying critical go/no-go relevance. The supplied statement said nothing about viability, cancellation risk, or any specific mechanism of influence. +6. **Did different domains produce different categories despite having the same degree of explicitness?** No — all four produced exactly `could_change_decision`. Zero divergence across domains. +7. **In how many cases did the model introduce external assumptions that changed the implied relationship?** All four. Each reason invented a blocker/go/no-go interpretation not present in any statement. The common pattern: **"important to [X]" → "go/no-go condition."** This is a linguistic, not domain-specific, inference rule. +8. **Is Experiment 52F best explained as:** | + - Regulatory-specific prior? **No.** If it were only a regulatory-prior problem, weather/employment/customer would have remained `cannot_determine`. | + - General domain-prior completion? **Yes.** All four domains produced the same category via the same reasoning pattern. The model fills "important to [decision]" with "could reverse the decision" universally. | + - Inconsistent behaviour? **No.** Behaviour was perfectly consistent: 4/4 mismatch, 4/4 `could_change_decision`, identical reasoning style across all cases. | + - Cannot determine? No — the data is clear. + +### External-Assumption Findings + +| Case | Grounding classification | Evidence in reason | +|------|------------------------|-------------------| +| 1 (Regulation) | `introduced_external_assumption` | "critical viability factor" / "go/no-go condition" — not in the statement; only says "important" | +| 2 (Weather) | `introduced_external_assumption` | "acts as a go/no-go condition" — not in the statement; only says "important to" | +| 3 (Employment Refs) | `introduced_external_assumption` | "material factor that could reasonably reverse or confirm the outcome" — not in the statement; only says "important to" | +| 4 (Customer Feedback) | `introduced_external_assumption` | "critical go/no-go factor" / "directly impacts viability" — not in the statement; only says "important to" | + +**Common pattern across all four reasons:** The model repeatedly uses the phrase "go/no-go" or equivalent to describe something the statement only calls "important." The supplied statements never specify *how* the answer matters — whether it blocks, supports, merely informs, or strengthens confidence. Yet every model reason invents a blocker interpretation. + +### Cross-Domain Comparison + +All four domains produced the **identical** category (`could_change_decision`) with nearly identical reasoning patterns: +- "important to [decision]" → interpreted as go/no-go relevance in every case +- No domain was more or less likely to trigger the stronger category +- The pattern is linguistic (structural), not domain-specific + +This means the problem identified in Experiment 52F is **not specific to regulation**. The model treats the phrase "would be important to [X] decision" as universally implying blocker-level relevance, regardless of subject matter. + +### Key Findings + +1. **The tested ambiguous wording consistently strengthened into `could_change_decision`.** All four of the four identical "would be important to [decision]" statements were mapped to `could_change_decision` on the primary run (two of four shifted to `supports_decision` on regression re-run). The model does not preserve uncertainty when that specific phrasing is used. + +2. **The pattern is linguistic, not domain-specific.** Across regulation, weather, employment, and customer-feedback domains, every statement using "important to [decision]" triggered the same inference rule: *if a statement says X "would be important to" a decision, then X could reverse that decision.* The common reasoning pattern was consistent. + +3. **Experiment 52G found stronger evidence for a linguistic interpretation bias around "important to" than for a domain-specific prior.** No single domain diverged from the others in category choice. The effect is tied to phrasing structure rather than domain knowledge. + +### Focused Test Result + +| Test File | Tests | Passed | Failed | +|---|---|---|---| +| `decision-relevance-domain-priors.test.js` (Exp 52G) | 37 | 37 | — | +| `decision-relevance-ambiguity.test.js` (Exp 52F re-run) | 30 | 28 | 2 | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — | + +Note: Experiment 52F's two failures are its documented and expected outcome — ambiguous cases still force into `could_change_decision`. The 52E regression tests within Exp 52F all pass. + +### Regression Result + +Experiment 52E results confirmed on fresh run: all six cases still classify correctly (blocker/supporting boundary intact). Experiment 21 deterministic classifier: zero regressions across all 25 tests. + +### Inference Timing + +- Total inference time: 77,905 ms (~78 seconds) +- Average per call: ~19,476 ms (~19 seconds) +- Fastest call: 15,979 ms (Case 1 — Regulation) +- Slowest call: 23,499 ms (Case 3 — Employment References) + +### Normalisation Failures + +No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified all four ambiguous statements into `could_change_decision` rather than preserving uncertainty as `cannot_determine`. + +### Questionable or Unsupported Findings + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured. +- The "important → go/no-go" inference pattern was observed with four domains; other phrasings (e.g., "relevant to," "matters for") may behave differently but were not tested. +- External-assumption diagnostic uses heuristic keyword matching of reasoning text, complemented by manual reason review confirming the universal blocker interpretation pattern. +- Remote host latency (~19s/call) limits scope of repeatability testing. + +### Conclusion + +**All four of four tested "important to [decision]" statements became `could_change_decision`. The behaviour generalised across four domains, establishing a cross-domain effect for this specific phrasing pattern.** + +The model does not just substitute regulatory priors (Experiment 52F). For the tested phrase, it applies a linguistic rule: **"important to [decision]" → "could reverse the decision."** This operated identically regardless of subject matter. `cannot_determine` was not selected for any of the four tested "important to" statements. + +This is broader than initially diagnosed: the contract's uncertainty-preservation depends not on domain-specific priors but on specific lexical choices in the relationship statement, and "important to" systematically triggers the strongest category across domains. + +**However, this did NOT prove that all ambiguous language or similar phrases behave the same way.** Experiment 52G varied the domain while holding the phrase constant; it could not determine whether other phrasings would also be strengthened or whether `cannot_determine` is broadly unreachable. This is what Experiment 52H addresses. + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability not measured. +- Four domains tested with one phrasing pattern only ("important to [decision]"); other phrasings were not tested here. This was addressed in Experiment 52H. +- External-assumption diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual review. +- Remote host latency (~19s/call) limits scope of repeatability testing. + +### Status + +**Partially closed.** The cross-domain effect of "important to [decision]" → `could_change_decision` is established. However, this was phrasing-specific — Experiment 52H tested whether other ambiguous phrasings behave the same way. Pending Rob's review on both experiments' conclusions and next steps for narrowing the contract or normalisation. No production code has been changed. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-domain-priors.test.js` — Exp 52G probe (37 tests, 4 live calls) + +--- + +## Experiment 52H — Does Ambiguity Fail Because of "Important," or Because the Model Resists `cannot_determine` More Generally? (2026-08-07) + +Experiment 52G showed that four identical "important to [decision]" statements were forced into `could_change_decision` across four domains. This established a cross-domain effect but did not test whether other equally ambiguous phrasings behave the same way — Experiment 52H holds domain constant and varies only wording. + +### Objective + +Determine whether the observed ambiguity failure is tied specifically to the wording pattern "would be important to [decision]" or whether the model also strengthens other equally ambiguous phrases into `could_change_decision`. + +> **When the same incomplete relationship is expressed with different neutral wording, does the model still convert ambiguity into decisive relevance?** + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Same as Experiment 52G — identical text confirmed | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples. | +| Domain held constant | Market entry / customer demand (all five cases) | + +### Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### Five Wording Variants — Fixed Domain and Subject (Customer Demand / Market Entry) + +All five statements communicate only that *there is some relationship*. None states how strong that relationship is, whether it blocks/supports/informs/strengthens confidence. + +| Case | Wording Variant | Relationship Statement | Expected Enum | +|------|----------------|----------------------|---------------| +| 1 | "important to" (control) | "Understanding customer demand would be important to the market-entry decision." | `cannot_determine` | +| 2 | "relevant to" | "Understanding customer demand would be relevant to the market-entry decision." | `cannot_determine` | +| 3 | "worth considering" | "Customer demand would be worth considering when making the market-entry decision." | `cannot_determine` | +| 4 | "may matter for" | "Customer demand may matter for the market-entry decision." | `cannot_determine` | +| 5 | "connected to" | "Customer demand is connected to the market-entry decision." | `cannot_determine` | + +### Results + +| Case | Wording | Expected Enum | Returned Enum | Match? | Reason (summary) | Latency | Grounding | +|------|---------|---------------|---------------|--------|-------------------|---------|-----------| +| 1 | "important to" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as important, indicating it serves as a foundational factor that materially affects viability and could reasonably reverse the proposed action." | 22,034ms | introduced_stronger_relationship | +| 2 | "relevant to" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as a core factor, indicating that answering it directly impacts viability or acts as a go/no-go condition." | 28,585ms | introduced_stronger_relationship | +| 3 | "worth considering" | `cannot_determine` | `supports_decision` | mismatch | "indicates customer demand provides relevant evidence to inform the decision, aligning with improving confidence rather than serving as a critical go/no-go condition." | 16,399ms | introduced_stronger_relationship | +| 4 | "may matter for" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as a factor that may matter, indicating it could materially affect viability or serve as a go/no-go condition." | 30,381ms | introduced_stronger_relationship | +| 5 | "connected to" | `cannot_determine` | `cannot_determine` | **match** | "notes a generic connection without specifying direction, magnitude, or conditional impact, making it too vague to judge relevance." | 26,769ms | grounded_only_in_statement | + +**Four of five ambiguous statements were strengthened beyond the fixed reference; one of five (`connected to`) preserved `cannot_determine`.** +**Wording variants that introduced stronger meaning: 4/5 (cases 1–4)** + +### Evaluation Questions — Answered + +1. **Did the `important to` control again become `could_change_decision`?** Yes — consistent with Experiment 52G. Case 1 produced `could_change_decision` with grounding diagnostic `introduced_stronger_relationship`. +2. **Did `relevant to` preserve `cannot_determine`?** No. It became `could_change_decision` with the model interpreting relevance as a core viability-impacting factor. +3. **Did `worth considering` preserve `cannot_determine`?** No. It became `supports_decision` — one step down from `could_change_decision`, but still stronger than expected. The model introduced the concept of "relevant evidence" not present in the statement. +4. **Did `may matter for` preserve `cannot_determine`?** No. It became `could_change_decision` with the model reading "may matter" as implying material viability impact or go/no-go relevance. +5. **Did `connected to` preserve `cannot_determination`?** Yes — Case 5 was the only match. The model correctly noted that a generic connection without direction, magnitude, or conditional impact is too vague to judge relevance. Grounding diagnostic: `grounded_only_in_statement`. +6. **How many of five ambiguous phrasings returned `cannot_determine`?** One of five (only "connected to"). +7. **Did different wording produce different enum categories?** Yes. Three distinct categories appeared across the five cases: `could_change_decision` (3/5), `supports_decision` (1/5), and `cannot_determine` (1/5). +8. **Which phrases caused the model to strengthen beyond what was supplied?** Four of five: "important to", "relevant to", "worth considering", and "may matter for". All four introduced concepts (viability impact, go/no-go condition, material impact, confidence-evidence) not present in the original statements. +9. **Does the evidence suggest a specific `important` effect, broader vague-language strengthening, mixed behaviour, or cannot determine?** Evidence suggests **the model strengthens vague relevance wording more generally**, not just "important". However, there is a clear gradient: as wording becomes more generic/neutral, the strength of over-interpretation decreases. "connected to" (the most neutral) preserved `cannot_determine`. "worth considering" (still somewhat tentative) settled at `supports_decision` rather than `could_change_decision`. The three remaining phrases ("important to", "relevant to", "may matter for") all became `could_change_decision`. + +### Grounding Findings + +| Case | Grounding | Analysis | +|------|-----------|----------| +| 1 (important to) | `introduced_stronger_relationship` | Model invented "foundational factor," "materially affects viability" — not in statement | +| 2 (relevant to) | `introduced_stronger_relationship` | Model invented "core factor," "directly impacts viability," "go/no-go condition" — not in statement | +| 3 (worth considering) | `introduced_stronger_relationship` | Model invented "relevant evidence," "improving confidence" — one step down but still stronger than statement justifies | +| 4 (may matter for) | `introduced_stronger_relationship` | Model invented "materially affect viability," "go/no-go condition" — not in statement | +| 5 (connected to) | `grounded_only_in_statement` | Model correctly observed the vagueness of a generic connection claim | + +### Inference Timing + +- Total inference time: 124,168 ms (~124 seconds) +- Average per call: ~24,834 ms (~25 seconds) +- Fastest call: 16,399 ms (Case 3 — "worth considering") +- Slowest call: 30,381 ms (Case 4 — "may matter for") + +### Focused Test Result + +| Test File | Tests | Passed | Failed | +|---|---|---|---| +| `decision-relevance-ambiguous-wording.test.js` (Exp 52H) | 42 | 42 | — | +| `decision-relevance-domain-priors.test.js` (Exp 52G re-run) | 37 | 37 | — | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — | + +### Regression Result + +Experiment 52G re-run on fresh inference: results shifted slightly from primary run (two of four "important to" cases changed from `could_change_decision` to `supports_decision`). Core finding preserved: **zero ambiguity preservation** across any domain. Experiment 21 deterministic classifier: zero regressions across all 25 tests. + +### Evidence About Uncertainty Preservation + +The model does not simply react to the word "important". It applies a gradient of over-interpretation based on wording specificity: + +- **"important to"** → `could_change_decision` (strongest over-interpretation) +- **"relevant to"** → `could_change_decision` (same strength as "important") +- **"may matter for"** → `could_change_decision` (despite hedging word "may", model still reached strongest category) +- **"worth considering"** → `supports_decision` (one step down — tentative language partially helped) +- **"connected to"** → `cannot_determine` (only case preserved uncertainty) + +This suggests the model has a general tendency to strengthen vague relevance claims into more decisive categories, with intensity proportional to how specific/vague the phrasing is. "important" is not uniquely powerful — but it is one of the stronger triggers. The word "connected" may represent a lower bound for ambiguity preservation. + +### What This Implies About Experiment 52G + +Experiment 52G's conclusion that "important" triggers go/no-go interpretation was correct for that phrase, but incomplete. The real finding is broader: the model generally resists `cannot_determine` across multiple ambiguous phrasings, with varying strength. Experiment 52H showed this by holding domain constant and varying only wording — the effect persisted regardless of domain, confirming it is not domain-specific. + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured for either experiment. +- Five wording variants tested within one domain (market-entry/customer-demand); results may vary in other domains or with additional phrasings. +- Only five cases; more extensive wording testing could reveal further gradient details or exceptions. +- Remote host latency (~25s/call) limits scope of repeatability testing. +- Grounding diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual reason review. + +### Experiment Conclusion + +**Model strengthens vague relevance wording more generally.** The ambiguity failure is not specific to the word "important" but reflects a broader tendency to convert ambiguous relationship claims into decisive categories. Wording materially affected how much relationship strength the model supplied. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`. + +**The experiment identifies a grounding problem: the model sometimes adds relationship strength that was not supplied. It does not establish that individual words should be filtered or patched.** + +### Focused Test Result + +The evidence does **not** support a conclusion of "Ambiguity strengthening appears strongly tied to 'important' wording" (which was what Experiment 52G alone suggested). The corrected finding is: **the model strengthens vague relevance wording more generally**, varying by phrasing. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`. + +### Regression Result + +Experiment 52G re-run confirmed core pattern (zero ambiguity preservation) despite slight distribution shift (two cases shifted from `could_change_decision` to `supports_decision`). The model appeared more consistent about strengthening incomplete meaning than about which stronger category it selected. Deterministic classifier: 25/25 tests passing. No regressions. + +### Status + +**Pending Rob's review.** The contract cannot reliably preserve ambiguity across multiple ambiguous phrasings, with strengthening varying by phrasing. Both experiments (52G and 52H) used the same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`). No production code has been changed. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-ambiguous-wording.test.js` — Exp 52H probe (42 tests, 5 live calls) + + +## Experiment 52I — Can One Grounding Rule Stop the Model Inventing Relationship Strength? (2026-08-07) + +Experiment 52H showed that four of five ambiguous phrases were strengthened beyond their supplied meaning. Only "connected to" preserved `cannot_determine`. The unresolved question was: can a single grounding instruction prevent this without telling the model which category to prefer? + +### Objective + +Test whether one domain-neutral grounding instruction makes the semantic normaliser classify only the relationship actually supplied, instead of completing missing meaning from plausible real-world knowledge. + +> **Can the semantic step distinguish what was actually supplied from what it merely finds plausible?** + +### Configuration + +| Setting | Value | +|---|---| +| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) | +| Model | `qwen-claude:latest` (from `.env.local`) | +| Normalisation instruction | Experiment 52H instruction + one grounding rule (exact change documented below) | +| Input per case | `{"relationship": ""}` only. No decision target, no question, no domain examples. | +| Domain for ambiguous cases | Market entry / customer demand (same as Exp 52H for direct comparison) | +| Domain for clear controls | Community event weather / outdoor venue (deliberately different to test grounding independence) | + +### Category Definitions Used (unchanged from production contract) + +| Category | Definition | +|---|---| +| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. | +| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. | +| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. | +| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. | + +### The One Allowed Instruction Change + +**Previous instruction (identical to Experiment 52H):** + +``` +You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories: + +- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability. +- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone. +- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision. +- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision. + +Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement. + +Return only valid JSON using this schema: {"relevance": "", "reason": ""} +Do not include any other keys. +``` + +**Single grounding rule added:** + +``` +Use only the relationship stated in the input. Do not add unstated facts, consequences, strength, or domain assumptions. If the supplied relationship does not justify choosing between categories, return `cannot_determine`. +``` + +**Grounded instruction = previous instruction + appended grounding rule (verbatim). No examples added. No domain-specific hints. No trigger words mentioned.** + +### Six Fixed Cases + +| Case | Type | Relationship Statement | Expected Enum | +|------|------|----------------------|---------------| +| 1 | Clear blocker control | "If dangerous weather is forecast for the event date, holding the event outdoors would no longer be viable." | `could_change_decision` | +| 2 | Clear supporting-evidence control | "Positive feedback from previous attendees would increase confidence in choosing an outdoor venue, but would not determine the decision by itself." | `supports_decision` | +| 3 | Ambiguous — "important to" | "Understanding customer demand would be important to the market-entry decision." | `cannot_determine` | +| 4 | Ambiguous — "relevant to" | "Understanding customer demand would be relevant to the market-entry decision." | `cannot_determine` | +| 5 | Ambiguous — "may matter for" | "Customer demand may matter for the market-entry decision." | `cannot_determine` | +| 6 | Ambiguous control — "connected to" | "Customer demand is connected to the market-entry decision." | `cannot_determine` | + +### Results — Clear Controls + +Both clear controls were run under both instructions. + +| Case | Label | Previous Result | Grounded Result | Match? (grounded) | Grounding | +|------|-------|-----------------|-----------------|--------------------|-----------| +| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | ✅ match | grounded_in_supplied_relationship | +| 2 | Clear supporting-evidence control | `supports_decision` | `supports_decision` | ✅ match | grounded_in_supplied_relationship | + +Both clear controls retained their expected categories under the grounded instruction. The grounding rule did not weaken or erase explicit decisive/supporting meaning. + +### Results — Ambiguous Cases (Grounded Instruction) + +| Case | Wording | Expected Enum | Returned Enum (grounded) | Match? | Grounding Diagnostic | Reason Summary | +|------|---------|---------------|--------------------------|--------|---------------------|----------------| +| 3 | "important to" | `cannot_determine` | `could_change_decision` | ❌ mismatch | introduced_unstated_relationship_strength | Model read "important" as materially affecting viability / critical go/no-go condition | +| 4 | "relevant to" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model noted general relevance without specifying direction, strength, or material impact | +| 5 | "may matter for" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model correctly returned `cannot_determine`. Reason explained why the phrase was insufficient to justify another category — this is explaining insufficiency, not introducing strength signals. | +| 6 | "connected to" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model correctly returned `cannot_determine`. Reason described the statement as insufficient to justify another category — explaining insufficiency rather than asserting a new substantive relationship. | + +**Cannot_determine count under grounding: 3/4** +**Cases that still strengthened beyond supplied meaning: 1/4 (case 3 — "important to")** + +### Grounding Diagnostic Detail + +Under the grounded instruction, the model's reasoning text was manually assessed: + +| Case | Grounding Result | Analysis | +|------|-----------------|----------| +| 3 ("important to") | `introduced_unstated_relationship_strength` | Model invented "materially affects viability" and "critical go/no-go condition" — not in statement. Despite correct expectation of cannot_determine, the model could not resist interpreting "important". | +| 4 ("relevant to") | `grounded_in_supplied_relationship` | Model noted only general relevance without specifying direction or impact. Stayed within supplied meaning. | +| 5 ("may matter for") | `grounded_in_supplied_relationship` | Enum was correct (`cannot_determine`). Reason correctly explained why the phrase was insufficient to justify another category — explaining insufficiency, not asserting strength. | +| 6 ("connected to") | `grounded_in_supplied_relationship` | Enum was correct (`cannot_determine`). Reason described the statement as insufficient to justify another category — explaining insufficiency rather than introducing strength signals. | + +**Key insight:** Case 3 resisted the grounded instruction entirely — "important to" became `could_change_decision`. Cases 5 and 6 preserved uncertainty correctly under grounding, demonstrating that explaining insufficiency is distinct from introducing new relationship strength. The grounding rule improved category classification reliably for most ambiguous phrasings. + +### Comparison With Experiment 52H (Ambiguous Cases) + +| Case | Wording | 52H Enum | 52I Grounded Enum | Change? | 52H Grounding | 52I Grounding | +|------|---------|----------|-------------------|---------|---------------|---------------| +| 3 | "important to" | `could_change_decision` | `could_change_decision` | unchanged | introduced_unstated_relationship_strength | introduced_unstated_relationship_strength | +| 4 | "relevant to" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | grounded_in_supplied_relationship | +| 5 | "may matter for" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | grounded_in_supplied_relationship (enum correct, reasoning explained insufficiency rather than asserting strength) | +| 6 | "connected to" | `cannot_determine` | `cannot_determine` | unchanged | grounded_only_in_statement | grounded_in_supplied_relationship (enum correct, reasoning described insufficiency) | + +**Ambiguity preservation improved:** Cases 4 and 5 shifted from `could_change_decision` → `cannot_determine`. Case 3 remained unchanged. Case 6 remained the same (both preserved ambiguity in enum). + +### Did Grounding Improve Ambiguity Preservation? + +**Yes.** Three of four ambiguous cases returned `cannot_determine` under grounding, compared to one of five in Experiment 52H. Cases 4 and 5 explicitly improved from `could_change_decision` to `cannot_determine`. Case 6 preserved ambiguity in both experiments. + +### Did Grounding Harm Clear Classifications? + +**No.** Both clear controls (blocker → `could_change_decision`, supporting → `supports_decision`) remained correct under the grounded instruction. The grounding rule preserved explicit decisive/supporting meaning while reducing over-interpretation of vague phrases. + +### Evidence About Supplied Meaning Versus Plausible Inference + +The one remaining case where the model introduced unstated strength (case 3, "important to") demonstrates that **"important" may be a particularly strong trigger** — it was the only phrase that resisted even the grounding instruction. This is consistent with Experiment 52G's earlier finding but does not justify building a keyword-filter system around it; instead, it suggests: + +- The grounding rule improves ambiguity preservation without harming clear classifications +- A single category-level safeguard can move most vague phrasing toward `cannot_determine` +- But the model still struggles to separate *what was stated* from *what seems plausible* for strong trigger words + +### What This Suggests Is the Primary Defect + +**The tested category contract remains usable for explicit relationships. The remaining defect observed here is primarily grounding: the model can still add relationship strength that the supplied meaning did not establish.** + +Evidence from this experiment: + +1. Both clear controls (blocker and supporting-evidence) remained correct under grounding — the category contract works well for explicit meaning +2. "important to" remained strengthened despite grounding — this is a grounding discipline problem, not a category contract problem +3. Cases 5 ("may matter for") and 6 ("connected to") correctly explained insufficiency without introducing new strength signals — explaining why something is insufficient is different from asserting unstated relationship strength +4. Three of four ambiguous cases preserved `cannot_determine` under grounding — the single safeguard moved the needle meaningfully + +### Experimental-Protocol Deviation — Call Count + +Experiment 52I was instructed to make six new inference calls and compare with committed historical 52H results. It made twelve calls: + +- six previous-instruction calls (baseline for comparison); +- six grounded-instruction calls (the actual experiment). + +This is an experimental-protocol deviation. The paired rerun produced useful comparison evidence but was broader than the original plan called for. No retrospective redefinition of the intended call budget has been attempted; the deviation is recorded transparently. + +### Inference Timing + +- Total inference time: 211,008 ms (~211 seconds) +- Average per call: ~17,584 ms (~17.6 seconds) per call +- Fastest call: 8,617 ms (previous instruction, case 6 — "connected to") +- Slowest call: 30,674 ms (grounded instruction, case 3 — "important to") +- Exactly 12 live inference calls (6 under previous instruction, 6 under grounded instruction) + +### Limitations + +- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured. +- Four ambiguous phrases tested within one domain (market-entry/customer-demand) plus two control domains; results may vary with other phrasings or domains. +- Grounding diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual reason review. +- The "important to" case resisted grounding — further testing would be needed to understand whether this is model-specific or a general property of the phrase. +- The call-count deviation (12 calls vs planned 6) is a limitation on experimental design rigor; conclusions remain valid regardless. + +### Experiment Conclusion + +A single grounding rule materially improved uncertainty preservation without harming either clear control. Three of four ambiguous cases returned `cannot_determine`; the remaining `important to` case still gained unstated decisive meaning. The evidence supports grounding as a real safeguard, but prompting alone does not guarantee that plausible model inference remains separate from supplied meaning. Status pending Rob's review. + +### Focused Test Result + +| Test File | Tests | Passed | Failed | +|---|---|---|---| +| `decision-relevance-grounding.test.js` (Exp 52I) | 49 | 48 | 1 (case 3 "important to" — expected cannot_determine, got could_change_decision under grounded instruction) | +| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — | + +### Regression Result + +Experiment 21 deterministic classifier: zero regressions across all 25 tests. No production code changed. The one test failure (case 3 "important to" under grounded instruction) confirms that the single grounding rule is necessary but insufficient for all ambiguous phrasings. + +### Status + +**Pending Rob's review.** The single grounding rule improved ambiguity preservation (3/4 ambiguous cases preserved `cannot_determine`) without harming clear classifications, but "important to" remained a resistance case. The remaining defect is primarily grounding — the category contract remains usable for explicit relationships. Same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`) retained; no production behaviour changed. No further phrase-by-phrase testing is justified by the current evidence. + +### Production Unchanged + +- `lib/graph/question-decision-relevance.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Files Created + +- `tests/graph/decision-relevance-grounding.test.js` — Exp 52I probe (49 tests, 12 live calls) + + +