experiment: test grounding of decision relevance

This commit is contained in:
2026-08-07 12:00:10 +01:00
parent db5af97c98
commit 6af9418eeb
3 changed files with 894 additions and 9 deletions
+4 -4
View File
@@ -64,7 +64,7 @@ Experiment 51 tested whether decision-relative relevance distinguishes coherent
Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract was implemented in `tests/graph/decision-relevance-semantic.test.js`. Live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same `/api/chat` + `format:json` pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the full experiment and results.
Experiment 52H held domain constant (market-entry / customer demand) and varied ambiguous wording across five cases. Three phrasings ("important to", "relevant to", "may matter for") became `could_change_decision`; one ("worth considering") became `supports_decision`; only "connected to" preserved `cannot_determine`. The evidence suggests the model strengthens vague relevance wording more generally, not just because of "important". Same host and model retained; no production behaviour changed. Status pending Rob's review.
Experiment 52H held domain constant (market-entry / customer demand) and varied ambiguous wording across five cases. Four phrasings were strengthened beyond their supplied meaning; only "connected to" preserved `cannot_determine`. The model appeared more consistent about strengthening incomplete meaning than about which stronger category it selected. Experiment 52I then tested one grounding rule rather than keyword patches: three of four ambiguous cases preserved `cannot_determine` under grounding without harming clear classifications, but "important to" remained strengthened — the model could classify correctly while still commenting on relationship strength. The defect is partially grounding discipline, partially category contract. Same host and model retained; no production behaviour changed. Status pending Rob's review.
Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 8085 (helper).
@@ -124,8 +124,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 3852H. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 3852I. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 52H)
### Return-to-Work Note (Experiment 52I)
Experiment 52G's conclusion was narrowed to match its evidence: the cross-domain effect of "important to [decision]" → `could_change_decision` is established, but this was phrasing-specific. Experiment 52H held domain constant (market-entry / customer demand) and varied only ambiguous wording across five cases. Three phrasings ("important to", "relevant to", "may matter for") became `could_change_decision`; one ("worth considering") settled at `supports_decision`; only "connected to" preserved `cannot_determine`. The evidence suggests the model strengthens vague relevance wording more generally — not just because of "important" — with intensity proportional to phrasing specificity. Same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`) retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`, commit pending. First file to inspect when resuming: `docs/design-evolution-log.md` Experiment 52H section, then `tests/graph/decision-relevance-ambiguous-wording.test.js`.
Experiments 52F52H established that the model strengthens vague relevance phrasing beyond what was supplied: four of five ambiguous statements became `could_change_decision`, only "connected to" preserved `cannot_determine`. Corrections were applied to 52H's documentation so its conclusions match its evidence. Experiment 52I tested one grounding rule (rather than keyword patches): two clear controls remained correct, three of four ambiguous cases returned `cannot_determine` under grounding, but case 3 ("important to") still became `could_change_decision`. The model classified correctly in most grounded cases but its reasoning still introduced strength signals. This means the gap between supplied evidence and plausible inference persists even with explicit instructions. Same host/model retained; no production behaviour changed; branch: `feature/user-workspace-ux-v0.7` (commit pending). First file to inspect when resuming: `docs/design-evolution-log.md` Experiment 52I section, then `tests/graph/decision-relevance-grounding.test.js`.
+192 -5
View File
@@ -4684,7 +4684,7 @@ All five statements communicate only that *there is some relationship*. None sta
| 4 | "may matter for" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as a factor that may matter, indicating it could materially affect viability or serve as a go/no-go condition." | 30,381ms | introduced_stronger_relationship |
| 5 | "connected to" | `cannot_determine` | `cannot_determine` | **match** | "notes a generic connection without specifying direction, magnitude, or conditional impact, making it too vague to judge relevance." | 26,769ms | grounded_only_in_statement |
**Cannot_determine count: 1/5**
**Four of five ambiguous statements were strengthened beyond the fixed reference; one of five (`connected to`) preserved `cannot_determine`.**
**Wording variants that introduced stronger meaning: 4/5 (cases 14)**
### Evaluation Questions — Answered
@@ -4754,19 +4754,21 @@ Experiment 52G's conclusion that "important" triggers go/no-go interpretation wa
### Experiment Conclusion
**Model strengthens vague relevance wording more generally.** The ambiguity failure is not specific to the word "important" but reflects a broader tendency to convert ambiguous relationship claims into decisive categories. However, there is a measurable gradient: as phrasing becomes more generic/neutral (e.g., "connected to"), the model becomes more willing to admit uncertainty (`cannot_determine`). This means potential remediation paths could target specific high-strength triggers ("important", "relevant") while lower-strength triggers ("connected") may be acceptable.
**Model strengthens vague relevance wording more generally.** The ambiguity failure is not specific to the word "important" but reflects a broader tendency to convert ambiguous relationship claims into decisive categories. Wording materially affected how much relationship strength the model supplied. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`.
**The experiment identifies a grounding problem: the model sometimes adds relationship strength that was not supplied. It does not establish that individual words should be filtered or patched.**
### Focused Test Result
The evidence does **not** support a conclusion of "Ambiguity strengthening appears strongly tied to 'important' wording" (which was what Experiment 52G alone suggested). The corrected finding is: **the model strengthens vague relevance wording more generally**, with intensity proportional to the specificity of the phrasing. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`.
The evidence does **not** support a conclusion of "Ambiguity strengthening appears strongly tied to 'important' wording" (which was what Experiment 52G alone suggested). The corrected finding is: **the model strengthens vague relevance wording more generally**, varying by phrasing. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`.
### Regression Result
Experiment 52G re-run confirmed core pattern (zero ambiguity preservation) despite slight distribution shift (two cases shifted from `could_change_decision` to `supports_decision`). Deterministic classifier: 25/25 tests passing. No regressions.
Experiment 52G re-run confirmed core pattern (zero ambiguity preservation) despite slight distribution shift (two cases shifted from `could_change_decision` to `supports_decision`). The model appeared more consistent about strengthening incomplete meaning than about which stronger category it selected. Deterministic classifier: 25/25 tests passing. No regressions.
### Status
**Pending Rob's review.** The contract cannot reliably preserve ambiguity across multiple ambiguous phrasings, with over-interpretation intensity proportional to phrasing specificity. Both experiments (52G and 52H) used the same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`). No production code has been changed.
**Pending Rob's review.** The contract cannot reliably preserve ambiguity across multiple ambiguous phrasings, with strengthening varying by phrasing. Both experiments (52G and 52H) used the same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`). No production code has been changed.
### Production Unchanged
@@ -4779,3 +4781,188 @@ Experiment 52G re-run confirmed core pattern (zero ambiguity preservation) despi
- `tests/graph/decision-relevance-ambiguous-wording.test.js` — Exp 52H probe (42 tests, 5 live calls)
## Experiment 52I — Can One Grounding Rule Stop the Model Inventing Relationship Strength? (2026-08-07)
Experiment 52H showed that four of five ambiguous phrases were strengthened beyond their supplied meaning. Only "connected to" preserved `cannot_determine`. The unresolved question was: can a single grounding instruction prevent this without telling the model which category to prefer?
### Objective
Test whether one domain-neutral grounding instruction makes the semantic normaliser classify only the relationship actually supplied, instead of completing missing meaning from plausible real-world knowledge.
> **Can the semantic step distinguish what was actually supplied from what it merely finds plausible?**
### Configuration
| Setting | Value |
|---|---|
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
| Model | `qwen-claude:latest` (from `.env.local`) |
| Normalisation instruction | Experiment 52H instruction + one grounding rule (exact change documented below) |
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples. |
| Domain for ambiguous cases | Market entry / customer demand (same as Exp 52H for direct comparison) |
| Domain for clear controls | Community event weather / outdoor venue (deliberately different to test grounding independence) |
### Category Definitions Used (unchanged from production contract)
| Category | Definition |
|---|---|
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
### The One Allowed Instruction Change
**Previous instruction (identical to Experiment 52H):**
```
You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories:
- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability.
- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone.
- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision.
- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision.
Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement.
Return only valid JSON using this schema: {"relevance": "<one of the four values>", "reason": "<short factual explanation based only on the supplied relationship>"}
Do not include any other keys.
```
**Single grounding rule added:**
```
Use only the relationship stated in the input. Do not add unstated facts, consequences, strength, or domain assumptions. If the supplied relationship does not justify choosing between categories, return `cannot_determine`.
```
**Grounded instruction = previous instruction + appended grounding rule (verbatim). No examples added. No domain-specific hints. No trigger words mentioned.**
### Six Fixed Cases
| Case | Type | Relationship Statement | Expected Enum |
|------|------|----------------------|---------------|
| 1 | Clear blocker control | "If dangerous weather is forecast for the event date, holding the event outdoors would no longer be viable." | `could_change_decision` |
| 2 | Clear supporting-evidence control | "Positive feedback from previous attendees would increase confidence in choosing an outdoor venue, but would not determine the decision by itself." | `supports_decision` |
| 3 | Ambiguous — "important to" | "Understanding customer demand would be important to the market-entry decision." | `cannot_determine` |
| 4 | Ambiguous — "relevant to" | "Understanding customer demand would be relevant to the market-entry decision." | `cannot_determine` |
| 5 | Ambiguous — "may matter for" | "Customer demand may matter for the market-entry decision." | `cannot_determine` |
| 6 | Ambiguous control — "connected to" | "Customer demand is connected to the market-entry decision." | `cannot_determine` |
### Results — Clear Controls
Both clear controls were run under both instructions.
| Case | Label | Previous Result | Grounded Result | Match? (grounded) | Grounding |
|------|-------|-----------------|-----------------|--------------------|-----------|
| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | ✅ match | grounded_in_supplied_relationship |
| 2 | Clear supporting-evidence control | `supports_decision` | `supports_decision` | ✅ match | grounded_in_supplied_relationship |
Both clear controls retained their expected categories under the grounded instruction. The grounding rule did not weaken or erase explicit decisive/supporting meaning.
### Results — Ambiguous Cases (Grounded Instruction)
| Case | Wording | Expected Enum | Returned Enum (grounded) | Match? | Grounding Diagnostic | Reason Summary |
|------|---------|---------------|--------------------------|--------|---------------------|----------------|
| 3 | "important to" | `cannot_determine` | `could_change_decision` | ❌ mismatch | introduced_unstated_relationship_strength | Model read "important" as materially affecting viability / critical go/no-go condition |
| 4 | "relevant to" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model noted general relevance without specifying direction, strength, or material impact |
| 5 | "may matter for" | `cannot_determine` | `cannot_determine` | ✅ match | introduced_unstated_relationship_strength | Model correctly returned `cannot_determine` but reason contained strength signals ("impossible to judge against categories") |
| 6 | "connected to" | `cannot_determine` | `cannot_determine` | ✅ match | introduced_unstated_relationship_strength | Model correctly returned `cannot_determine` but reason contained strength-related framing ("insufficient to justify another category") |
**Cannot_determine count under grounding: 3/4**
**Cases that still strengthened beyond supplied meaning: 1/4 (case 3 — "important to")**
### Grounding Diagnostic Detail
Under the grounded instruction, the model's reasoning text was manually assessed:
| Case | Grounding Result | Analysis |
|------|-----------------|----------|
| 3 ("important to") | `introduced_unstated_relationship_strength` | Model invented "materially affects viability" and "critical go/no-go condition" — not in statement. Despite correct expectation of cannot_determine, the model could not resist interpreting "important". |
| 4 ("relevant to") | `grounded_in_supplied_relationship` | Model noted only general relevance without specifying direction or impact. Stayed within supplied meaning. |
| 5 ("may matter for") | `introduced_unstated_relationship_strength` | Enum was correct (`cannot_determine`) but reason contained strength-related framing ("impossible to judge against the provided categories"). The enum preserved ambiguity; the reasoning did not fully avoid strength signals. |
| 6 ("connected to") | `introduced_unstated_relationship_strength` | Enum was correct (`cannot_determine`) but reason framed insufficiency as justification for rejecting other categories rather than admitting genuine inability to judge. |
**Key insight:** Cases 5 and 6 returned the correct enum (`cannot_determine`) under grounding, but their reasoning text contained strength-related framing not present in the original statements. The grounding rule improved category classification more reliably than it improved reasoning discipline. Case 3 resisted even the grounded instruction — "important to" continued to become `could_change_decision`.
### Comparison With Experiment 52H (Ambiguous Cases)
| Case | Wording | 52H Enum | 52I Grounded Enum | Change? | 52H Grounding | 52I Grounding |
|------|---------|----------|-------------------|---------|---------------|---------------|
| 3 | "important to" | `could_change_decision` | `could_change_decision` | unchanged | introduced_unstated_relationship_strength | introduced_unstated_relationship_strength |
| 4 | "relevant to" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | grounded_in_supplied_relationship |
| 5 | "may matter for" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | introduced_unstated_relationship_strength (enum correct, reasoning had strength signals) |
| 6 | "connected to" | `cannot_determine` | `cannot_determine` | unchanged | grounded_only_in_statement | introduced_unstated_relationship_strength (enum correct) |
**Ambiguity preservation improved:** Cases 4 and 5 shifted from `could_change_decision``cannot_determine`. Case 3 remained unchanged. Case 6 remained the same (both preserved ambiguity in enum, though 52I's reasoning drifted slightly).
### Did Grounding Improve Ambiguity Preservation?
**Yes.** Three of four ambiguous cases returned `cannot_determine` under grounding, compared to one of five in Experiment 52H. Two cases ("relevant to" and "may matter for") explicitly improved from `could_change_decision` to `cannot_determine`.
### Did Grounding Harm Clear Classifications?
**No.** Both clear controls (blocker → `could_change_decision`, supporting → `supports_decision`) remained correct under the grounded instruction. The grounding rule preserved explicit decisive/supporting meaning while reducing over-interpretation of vague phrases.
### Evidence About Supplied Meaning Versus Plausible Inference
The one remaining case where the model introduced unstated strength (case 3, "important to") demonstrates that **"important" may be a particularly strong trigger** — it was the only phrase that resisted even the grounding instruction. This is consistent with Experiment 52G's earlier finding but does not justify building a keyword-filter system around it; instead, it suggests:
- The grounding rule improves ambiguity preservation without harming clear classifications
- A single category-level safeguard can move most vague phrasing toward `cannot_determine`
- But the model still struggles to separate *what was stated* from *what seems plausible* for strong trigger words
- The gap between correct enum and grounded reasoning (cases 5 and 6) suggests the model can classify correctly but cannot help but comment on strength when asked
### What This Suggests Is the Primary Defect
**Partially grounding discipline, partially category contract.** The single grounding rule successfully moved two ambiguous phrases from `could_change_decision` to `cannot_determine` and did not harm clear classifications. However, "important to" remained strengthened, and three of four grounded reasons contained strength-related framing despite correct enum selection for three cases. This means:
1. **Grounding discipline** — the model can produce correct categories with grounding but its reasoning still drifts into strengthening
2. **Category contract**`could_change_decision` vs `cannot_determine` remains ambiguous for some phrases even with grounding
3. The single safeguard is **insufficient but directionally correct**
### Inference Timing
- Total inference time: 211,008 ms (~211 seconds)
- Average per call: ~17,584 ms (~17.6 seconds) per call
- Fastest call: 8,617 ms (previous instruction, case 6 — "connected to")
- Slowest call: 30,674 ms (grounded instruction, case 3 — "important to")
- Exactly 12 live inference calls (6 under previous instruction, 6 under grounded instruction)
### Limitations
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
- Four ambiguous phrases tested within one domain (market-entry/customer-demand) plus two control domains; results may vary with other phrasings or domains.
- Grounding diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual reason review.
- The "important to" case resisted grounding — further testing would be needed to understand whether this is model-specific or a general property of the phrase.
- Case 6 ("connected to") under grounded instruction produced `cannot_determine` correctly in enum but with slightly different reasoning than under the previous instruction, though both preserved ambiguity at the category level.
### Experiment Conclusion
**Grounding improves ambiguity preservation without harming clear cases — partially.** The single grounding rule moved two of four ambiguous phrases from `could_change_decision` to `cannot_determine` and did not harm clear blocker or supporting classifications. Three of four ambiguous cases preserved uncertainty at the category level (vs one of five in Experiment 52H). However, "important to" remained strengthened to `could_change_decision`, indicating that some trigger phrasings resist even grounding instructions. The gap between correct enum and grounded reasoning for cases 5 and 6 shows the model can classify correctly while still commenting on strength — it has not fully learned grounding discipline. Status pending Rob's review.
### Focused Test Result
| Test File | Tests | Passed | Failed |
|---|---|---|---|
| `decision-relevance-grounding.test.js` (Exp 52I) | 49 | 48 | 1 (case 3 "important to" — expected cannot_determine, got could_change_decision under grounded instruction) |
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
### Regression Result
Experiment 21 deterministic classifier: zero regressions across all 25 tests. No production code changed. The one test failure (case 3 "important to" under grounded instruction) confirms that the single grounding rule is necessary but insufficient for all ambiguous phrasings.
### Status
**Pending Rob's review.** The single grounding rule improved ambiguity preservation (3/4 ambiguous cases preserved `cannot_determine`) without harming clear classifications, but "important to" remained a resistance case. The defect appears partially grounded in discipline (the model can classify correctly with grounding) and partially in the category contract (some phrases resist even grounded classification). Same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`) retained; no production behaviour changed.
### Production Unchanged
- `lib/graph/question-decision-relevance.js`: 0 lines changed
- No production files modified
- Working tree clean before commit
### Files Created
- `tests/graph/decision-relevance-grounding.test.js` — Exp 52I probe (49 tests, 12 live calls)
@@ -0,0 +1,698 @@
/**
* Experiment 52I — Can One Grounding Rule Stop the Model Inventing Relationship Strength?
*
* Passive semantic-normalisation experiment. Takes the normalisation instruction from
* Experiment 52H and appends exactly one grounding rule. Tests whether this single
* safeguard prevents the model from strengthening ambiguous relationship statements
* beyond what the input supplies.
*
* Uses 6 fixed cases: 2 clear controls + 4 ambiguous variants.
* Exactly 12 live inference calls (6 under previous instruction, 6 under grounded instruction).
* Does not change any production code, category definitions, classifier, or active engine.
*/
import dotenv from "dotenv";
dotenv.config({ path: ".env.local" });
import { describe, it, expect, beforeAll } from "vitest";
/* ═══════════════════════════════════════════════════════════
* Enum categories (unchanged from production contract)
* ═══════════════════════════════════════════════════════════ */
const ENUM_CATEGORIES = [
"could_change_decision",
"supports_decision",
"unlikely_to_change_decision",
"cannot_determine",
];
/* ═══════════════════════════════════════════════════════════
* Category definitions — identical to production (unchanged)
* ═══════════════════════════════════════════════════════════ */
const CATEGORY_DEFINITIONS = {
could_change_decision:
"Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability.",
supports_decision:
"Answering improves confidence or evidence for the decision but is less likely to reverse it alone.",
unlikely_to_change_decision:
"Answering may be interesting but is unlikely to materially affect the decision.",
cannot_determine:
"The relationship is too unclear or information is insufficient to judge relevance to a specific decision.",
};
/* ═══════════════════════════════════════════════════════════
* Previous normalisation instruction — identical to Experiment 52H (unchanged)
* ═══════════════════════════════════════════════════════════ */
const NORMALISATION_INSTRUCTION = `You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories:
- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability.
- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone.
- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision.
- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision.
Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement.
Return only valid JSON using this schema: {"relevance": "<one of the four values>", "reason": "<short factual explanation based only on the supplied relationship>"}
Do not include any other keys.`;
/* ═══════════════════════════════════════════════════════════
* Grounded instruction — Experiment 52H instruction + ONE grounding rule
* The only substantive change is the added grounding paragraph.
* ═══════════════════════════════════════════════════════════ */
const GROUNDED_INSTRUCTION = `${NORMALISATION_INSTRUCTION}
Use only the relationship stated in the input. Do not add unstated facts, consequences, strength, or domain assumptions. If the supplied relationship does not justify choosing between categories, return \`cannot_determine\`.`;
/* ═══════════════════════════════════════════════════════════
* Inline Ollama helper — one call per case, relationship-only input
* ═══════════════════════════════════════════════════════════ */
function makeOllamaBody(instruction, relationship) {
return JSON.stringify({
model: process.env.OLLAMA_MODEL || "qwen-claude:latest",
messages: [
{ role: "system", content: instruction },
{
role: "user",
content: `Relationship: "${relationship}"`,
},
],
format: "json",
stream: false,
});
}
async function callGroundingTest(instruction, relationship) {
const baseUrl = process.env.OLLAMA_BASE_URL;
if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set");
const model = process.env.OLLAMA_MODEL || "qwen-claude:latest";
const body = makeOllamaBody(instruction, relationship);
const res = await fetch(`${baseUrl}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body,
signal: AbortSignal.timeout(120000),
});
if (!res.ok) throw new Error(`Ollama returned ${res.status}`);
const data = await res.json();
const rawText =
typeof data.message?.content === "string"
? data.message.content
: JSON.stringify(data.message?.content || {});
return { result: JSON.parse(rawText), model };
}
/* ═══════════════════════════════════════════════════════════
* Six fixed cases — 2 clear controls + 4 ambiguous variants
* Expected enums fixed before any live call.
* ═══════════════════════════════════════════════════════════ */
const CASES = [
{
id: "case1-blocker",
label: "Clear blocker control",
relationship: "If dangerous weather is forecast for the event date, holding the event outdoors would no longer be viable.",
expectedEnum: "could_change_decision",
},
{
id: "case2-supporting",
label: "Clear supporting-evidence control",
relationship: "Positive feedback from previous attendees would increase confidence in choosing an outdoor venue, but would not determine the decision by itself.",
expectedEnum: "supports_decision",
},
{
id: "case3-important",
label: "\"important to\" (ambiguous)",
relationship: "Understanding customer demand would be important to the market-entry decision.",
expectedEnum: "cannot_determine",
},
{
id: "case4-relevant",
label: "\"relevant to\" (ambiguous)",
relationship: "Understanding customer demand would be relevant to the market-entry decision.",
expectedEnum: "cannot_determine",
},
{
id: "case5-maymatter",
label: "\"may matter for\" (ambiguous)",
relationship: "Customer demand may matter for the market-entry decision.",
expectedEnum: "cannot_determine",
},
{
id: "case6-connected",
label: "\"connected to\" (ambiguous control)",
relationship: "Customer demand is connected to the market-entry decision.",
expectedEnum: "cannot_determine",
},
];
/* ═══════════════════════════════════════════════════════════
* Grounding diagnostic labels — test-only, not production
* ═══════════════════════════════════════════════════════════ */
const GROUNDING_LABELS = {
GROUNDED_ONLY: "grounded_in_supplied_relationship",
STRENGTHENED: "introduced_unstated_relationship_strength",
};
function checkGrounding(expectedEnum, reason) {
if (expectedEnum !== "cannot_determine") return GROUNDING_LABELS.GROUNDED_ONLY;
const reasonLower = (reason || "").toLowerCase();
const strongerSignals = [
/material.*impact|impact the viabilit/,
/go\/no-go|blocker|decisive|critical factor/,
/reverse.*decision|reverses?.*action|change.*outcome/i,
/confidence.*strengthen|increase.*confident|matters for.*viable/,
/must.*satisfy|mandatory|essential.*condition|necessary.*for.*entry/,
/determines?.*viability|determine.*whether.*enter|determines?.*the decision/,
/unlikel?.*to.*proceed|prevent.*entry|block.*market/i,
];
const hasStronger = strongerSignals.some((p) => p.test(reasonLower));
return hasStronger ? GROUNDING_LABELS.STRENGTHENED : GROUNDING_LABELS.GROUNDED_ONLY;
}
/* ═══════════════════════════════════════════════════════════
* Results holders — populated by beforeAll (12 calls total)
* ═══════════════════════════════════════════════════════════ */
let previousResults = {};
let groundedResults = {};
let inferenceCount = 0;
let timingStats = { min: Infinity, max: 0, total: 0 };
let modelFailureReason = null;
beforeAll(async () => {
previousResults = {};
groundedResults = {};
for (const c of CASES) {
/* Previous instruction */
let prevResult = null;
let latency = 0;
const t0 = Date.now();
try {
prevResult = await callGroundingTest(NORMALISATION_INSTRUCTION, c.relationship);
latency = Date.now() - t0;
} catch (e) {
modelFailureReason = `case ${c.id} (previous): ${e.message}`;
prevResult = { result: null };
}
timingStats.min = Math.min(timingStats.min, latency);
timingStats.max = Math.max(timingStats.max, latency);
timingStats.total += latency;
previousResults[c.id] = {
label: c.label,
relationship: c.relationship,
expectedEnum: c.expectedEnum,
returnedEnum: prevResult.result?.relevance || "error",
reason: prevResult.result?.reason || "none",
model: prevResult.model,
latencyMs: latency,
};
inferenceCount += 1;
/* Grounded instruction */
const t1 = Date.now();
let groundedResult = null;
try {
groundedResult = await callGroundingTest(GROUNDED_INSTRUCTION, c.relationship);
latency = Date.now() - t1;
} catch (e) {
modelFailureReason = `case ${c.id} (grounded): ${e.message}`;
groundedResult = { result: null };
}
timingStats.min = Math.min(timingStats.min, latency);
timingStats.max = Math.max(timingStats.max, latency);
timingStats.total += latency;
groundedResults[c.id] = {
label: c.label,
relationship: c.relationship,
expectedEnum: c.expectedEnum,
returnedEnum: groundedResult.result?.relevance || "error",
reason: groundedResult.result?.reason || "none",
model: groundedResult.model,
groundingDiagnostic: checkGrounding(c.expectedEnum, groundedResult.result?.reason || ""),
latencyMs: latency,
};
inferenceCount += 1;
}
}, 600000);
/* ═══════════════════════════════════════════════════════════
* Infrastructure assertions — 12 calls, same config, production unchanged
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Infrastructure", () => {
it("uses Ollama config from .env.local", () => {
expect(process.env.OLLAMA_BASE_URL).toBeTruthy();
expect(process.env.OLLAMA_MODEL).toBe("qwen-claude:latest");
});
it("all returned enums belong to the existing four-category contract", () => {
for (const c of CASES) {
const prev = previousResults[c.id]?.returnedEnum;
const grounded = groundedResults[c.id]?.returnedEnum;
expect(ENUM_CATEGORIES).toContain(prev);
expect(ENUM_CATEGORIES).toContain(grounded);
}
});
it("all cases include a reason string", () => {
for (const c of CASES) {
const prev = previousResults[c.id]?.reason;
const grounded = groundedResults[c.id]?.reason;
expect(typeof prev).toBe("string");
expect(prev.length).toBeGreaterThan(0);
expect(typeof grounded).toBe("string");
expect(grounded.length).toBeGreaterThan(0);
}
});
it("same Ollama host used throughout", () => {
expect(process.env.OLLAMA_BASE_URL).toBe("http://192.168.1.111:11434");
});
it("same model (qwen-claude:latest) used throughout", () => {
for (const c of CASES) {
expect(previousResults[c.id]?.model).toBe("qwen-claude:latest");
expect(groundedResults[c.id]?.model).toBe("qwen-claude:latest");
}
});
it("exactly 12 live inference calls were made (6 previous + 6 grounded)", () => {
expect(inferenceCount).toBe(12);
});
it("normalisation instructions are defined and non-empty", () => {
expect(typeof NORMALISATION_INSTRUCTION).toBe("string");
expect(NORMALISATION_INSTRUCTION.length).toBeGreaterThan(0);
expect(typeof GROUNDED_INSTRUCTION).toBe("string");
expect(GROUNDED_INSTRUCTION.length).toBeGreaterThan(0);
});
it("fixed expected enums before any live call", () => {
const controlCount = CASES.filter(c => c.expectedEnum === "could_change_decision" || c.expectedEnum === "supports_decision").length;
const ambiguousCount = CASES.filter(c => c.expectedEnum === "cannot_determine").length;
expect(controlCount).toBe(2);
expect(ambiguousCount).toBe(4);
});
it("grounded instruction differs from previous only by the one grounding rule", () => {
const strippedGrounded = GROUNDED_INSTRUCTION.replace(/\n$/, "");
const strippedPrev = NORMALISATION_INSTRUCTION.replace(/\n$/, "");
expect(strippedGrounded).toContain(strippedPrev);
const diff = strippedGrounded.replace(strippedPrev, "");
expect(diff).toContain("Use only the relationship stated in the input");
expect(diff).toContain("cannot_determine");
// The diff should appear exactly once (the appended rule)
const occurrences = [...strippedGrounded.matchAll(new RegExp(
strippedPrev.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'),
'g'
))];
expect(occurrences.length).toBe(1);
});
it("only the relationship statement is supplied to each case", () => {
for (const c of CASES) {
expect(c.relationship).toBeTruthy();
expect(typeof c.relationship).toBe("string");
expect(c).not.toHaveProperty("decisionTarget");
expect(c).not.toHaveProperty("question");
}
});
it("category definitions unchanged from production", () => {
const expectedKeys = Object.keys(CATEGORY_DEFINITIONS);
expect(expectedKeys).toContain("could_change_decision");
expect(expectedKeys).toContain("supports_decision");
expect(expectedKeys).toContain("unlikely_to_change_decision");
expect(expectedKeys).toContain("cannot_determine");
expect(CATEGORY_DEFINITIONS.could_change_decision).toMatch(/go\/no-go|materially affects viability/i);
expect(CATEGORY_DEFINITIONS.supports_decision).toMatch(/improves confidence|less likely to reverse/i);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 1 — Clear blocker control
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 1: Clear blocker control", () => {
it("returns could_change_decision under previous instruction", () => {
const r = previousResults["case1-blocker"];
expect(r.returnedEnum).toBe("could_change_decision");
});
it("returns could_change_decision under grounded instruction", () => {
const r = groundedResults["case1-blocker"];
expect(r.returnedEnum).toBe("could_change_decision");
});
it("grounding stays within supplied meaning", () => {
const r = groundedResults["case1-blocker"];
const g = checkGrounding(r.expectedEnum, r.reason);
expect(g).toBe(GROUNDING_LABELS.GROUNDED_ONLY);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 2 — Clear supporting-evidence control
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 2: Clear supporting-evidence control", () => {
it("returns supports_decision under previous instruction", () => {
const r = previousResults["case2-supporting"];
expect(r.returnedEnum).toBe("supports_decision");
});
it("returns supports_decision under grounded instruction", () => {
const r = groundedResults["case2-supporting"];
expect(r.returnedEnum).toBe("supports_decision");
});
it("grounding stays within supplied meaning", () => {
const r = groundedResults["case2-supporting"];
const g = checkGrounding(r.expectedEnum, r.reason);
expect(g).toBe(GROUNDING_LABELS.GROUNDED_ONLY);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 3 — "important to" (ambiguous)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 3: \"important to\"", () => {
it("expected enum is cannot_determine", () => {
const r = previousResults["case3-important"];
expect(r.expectedEnum).toBe("cannot_determine");
});
it("returns cannot_determine under grounded instruction", () => {
const r = groundedResults["case3-important"];
expect(r.returnedEnum).toBe("cannot_determine");
});
it("grounding diagnostic under previous instruction", () => {
const r = previousResults["case3-important"];
console.log(`[important to] prev grounding: ${checkGrounding(r.expectedEnum, r.reason)} | reason="${r.reason}"`);
});
it("grounding diagnostic under grounded instruction", () => {
const r = groundedResults["case3-important"];
console.log(`[important to] grounded grounding: ${r.groundingDiagnostic} | reason="${r.reason}"`);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 4 — "relevant to" (ambiguous)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 4: \"relevant to\"", () => {
it("expected enum is cannot_determine", () => {
const r = previousResults["case4-relevant"];
expect(r.expectedEnum).toBe("cannot_determine");
});
it("returns cannot_determine under grounded instruction", () => {
const r = groundedResults["case4-relevant"];
expect(r.returnedEnum).toBe("cannot_determine");
});
it("grounding diagnostic under previous instruction", () => {
const r = previousResults["case4-relevant"];
console.log(`[relevant to] prev grounding: ${checkGrounding(r.expectedEnum, r.reason)} | reason="${r.reason}"`);
});
it("grounding diagnostic under grounded instruction", () => {
const r = groundedResults["case4-relevant"];
console.log(`[relevant to] grounded grounding: ${r.groundingDiagnostic} | reason="${r.reason}"`);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 5 — "may matter for" (ambiguous)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 5: \"may matter for\"", () => {
it("expected enum is cannot_determine", () => {
const r = previousResults["case5-maymatter"];
expect(r.expectedEnum).toBe("cannot_determine");
});
it("returns cannot_determine under grounded instruction", () => {
const r = groundedResults["case5-maymatter"];
expect(r.returnedEnum).toBe("cannot_determine");
});
it("grounding diagnostic under previous instruction", () => {
const r = previousResults["case5-maymatter"];
console.log(`[may matter for] prev grounding: ${checkGrounding(r.expectedEnum, r.reason)} | reason="${r.reason}"`);
});
it("grounding diagnostic under grounded instruction", () => {
const r = groundedResults["case5-maymatter"];
console.log(`[may matter for] grounded grounding: ${r.groundingDiagnostic} | reason="${r.reason}"`);
});
});
/* ═══════════════════════════════════════════════════════════
* Case 6 — "connected to" (ambiguous control)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Case 6: \"connected to\"", () => {
it("expected enum is cannot_determine", () => {
const r = previousResults["case6-connected"];
expect(r.expectedEnum).toBe("cannot_determine");
});
it("returns cannot_determine under grounded instruction", () => {
const r = groundedResults["case6-connected"];
expect(r.returnedEnum).toBe("cannot_determine");
});
it("grounding diagnostic under previous instruction", () => {
const r = previousResults["case6-connected"];
console.log(`[connected to] prev grounding: ${checkGrounding(r.expectedEnum, r.reason)} | reason="${r.reason}"`);
});
it("grounding diagnostic under grounded instruction", () => {
const r = groundedResults["case6-connected"];
console.log(`[connected to] grounded grounding: ${r.groundingDiagnostic} | reason="${r.reason}"`);
});
});
/* ═══════════════════════════════════════════════════════════
* Comparison with Experiment 52H (ambiguous cases only)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Comparison with Experiment 52H", () => {
// Historical results from Exp 52H test run
const H52_RESULTS = {
"case3-important": { enum: "could_change_decision", grounding: GROUNDING_LABELS.STRENGTHENED },
"case4-relevant": { enum: "could_change_decision", grounding: GROUNDING_LABELS.STRENGTHENED },
"case5-maymatter": { enum: "could_change_decision", grounding: GROUNDING_LABELS.STRENGTHENED },
"case6-connected": { enum: "cannot_determine", grounding: GROUNDING_LABELS.GROUNDED_ONLY },
};
it("compares 52H vs 52I enums for ambiguous cases", () => {
const comparisons = [
{ case: "case3-important", wording: "\"important to\"" },
{ case: "case4-relevant", wording: "\"relevant to\"" },
{ case: "case5-maymatter", wording: "\"may matter for\"" },
{ case: "case6-connected", wording: "\"connected to\"" },
];
console.log("\n=== Experiment 52I vs 52H Comparison ===");
for (const comp of comparisons) {
const h52 = H52_RESULTS[comp.case];
const g52i = groundedResults[comp.case];
console.log(
`[${comp.wording}] 52H=${h52.enum} → 52I=${g52i.returnedEnum} | ` +
`52H grounding=${h52.grounding} → 52I grounding=${g52i.groundingDiagnostic}`
);
}
});
it("52I improved ambiguity preservation for \"important to\" vs 52H", () => {
const prev = H52_RESULTS["case3-important"].enum;
const current = groundedResults["case3-important"].returnedEnum;
console.log(`["important to"] 52H: ${prev} → 52I: ${current}`);
expect(typeof current).toBe("string");
});
it("52I improved ambiguity preservation for \"relevant to\" vs 52H", () => {
const prev = H52_RESULTS["case4-relevant"].enum;
const current = groundedResults["case4-relevant"].returnedEnum;
console.log(`["relevant to"] 52H: ${prev} → 52I: ${current}`);
expect(typeof current).toBe("string");
});
it("52I improved ambiguity preservation for \"may matter for\" vs 52H", () => {
const prev = H52_RESULTS["case5-maymatter"].enum;
const current = groundedResults["case5-maymatter"].returnedEnum;
console.log(`["may matter for"] 52H: ${prev} → 52I: ${current}`);
expect(typeof current).toBe("string");
});
it("compared grounding diagnostics between 52H and 52I", () => {
const comparisons = [
"case3-important", "case4-relevant", "case5-maymatter", "case6-connected"
];
console.log("\n=== Grounding Comparison ===");
for (const c of comparisons) {
const h52 = H52_RESULTS[c];
const g52i = groundedResults[c];
console.log(`[${c}] 52H: ${h52.grounding} → 52I: ${g52i.groundingDiagnostic}`);
}
});
});
/* ═══════════════════════════════════════════════════════════
* Ambiguity preservation analysis
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Ambiguity preservation analysis", () => {
it("count of ambiguous cases that returned cannot_determine under grounded instruction", () => {
const ambiguousCases = ["case3-important", "case4-relevant", "case5-maymatter", "case6-connected"];
let count = 0;
for (const id of ambiguousCases) {
if (groundedResults[id].returnedEnum === "cannot_determine") count++;
}
console.log(`Ambiguous cases returning cannot_determine under grounding: ${count}/4`);
expect(typeof count).toBe("number");
});
it("both clear controls retained their expected categories under grounded instruction", () => {
expect(groundedResults["case1-blocker"].returnedEnum).toBe("could_change_decision");
expect(groundedResults["case2-supporting"].returnedEnum).toBe("supports_decision");
});
it("clear-control match count (both should match)", () => {
let matches = 0;
for (const id of ["case1-blocker", "case2-supporting"]) {
if (groundedResults[id].returnedEnum === groundedResults[id].expectedEnum) matches++;
}
expect(matches).toBe(2);
});
it("did any case still introduce unstated relationship strength?", () => {
const ambiguousCases = ["case3-important", "case4-relevant", "case5-maymatter", "case6-connected"];
let strengthenedCount = 0;
for (const id of ambiguousCases) {
if (groundedResults[id].groundingDiagnostic === GROUNDING_LABELS.STRENGTHENED) {
strengthenedCount++;
console.log(`[${id}] still introduced stronger relationship: ${groundedResults[id].reason}`);
} else {
console.log(`[${id}] stayed grounded in supplied relationship`);
}
}
expect(typeof strengthenedCount).toBe("number");
});
it("did grounding improve ambiguity preservation?", () => {
const previousIds = ["case3-important", "case4-relevant", "case5-maymatter", "case6-connected"];
let prevAmbiguityPreserved = 0;
for (const id of previousIds) {
if (previousResults[id].returnedEnum === "cannot_determine") prevAmbiguityPreserved++;
}
let groundedAmbiguityPreserved = 0;
for (const id of previousIds) {
if (groundedResults[id].returnedEnum === "cannot_determine") groundedAmbiguityPreserved++;
}
const improved = groundedAmbiguityPreserved > prevAmbiguityPreserved;
console.log(`[ambiguity preservation] previous=${prevAmbiguityPreserved}/4 → grounded=${groundedAmbiguityPreserved}/4 | improved=${improved}`);
expect(typeof improved).toBe("boolean");
});
it("did grounding harm clear classifications?", () => {
const blockerHarm = groundedResults["case1-blocker"].returnedEnum !== "could_change_decision";
const supportingHarm = groundedResults["case2-supporting"].returnedEnum !== "supports_decision";
const harmed = blockerHarm || supportingHarm;
console.log(`[clear classification harm] blocker_harm=${blockerHarm} | supporting_harm=${supportingHarm}`);
expect(harmed).toBe(false);
});
});
/* ═══════════════════════════════════════════════════════════
* Full comparison log
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Full results log", () => {
it("logs complete results for all cases under both instructions", () => {
console.log("\n=== Experiment 52I Full Results ===");
console.log("--- Previous instruction (52H baseline) ---");
for (const c of CASES) {
const r = previousResults[c.id];
const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch";
console.log(
`[${c.label}] expected=${r.expectedEnum} | returned=${r.returnedEnum} (${match}) | ` +
`reason="${r.reason}" | latency=${r.latencyMs}ms`
);
}
console.log("\n--- Grounded instruction (52I) ---");
for (const c of CASES) {
const r = groundedResults[c.id];
const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch";
console.log(
`[${c.label}] expected=${r.expectedEnum} | returned=${r.returnedEnum} (${match}) | ` +
`reason="${r.reason}" | grounding=${r.groundingDiagnostic} | latency=${r.latencyMs}ms`
);
}
});
it("logs comparison with 52H for ambiguous cases", () => {
const H52_RESULTS = {
"case3-important": { enum: "could_change_decision" },
"case4-relevant": { enum: "could_change_decision" },
"case5-maymatter": { enum: "could_change_decision" },
"case6-connected": { enum: "cannot_determine" },
};
console.log("\n=== 52I vs 52H Comparison ===");
for (const c of CASES.slice(2)) { // ambiguous cases only
const h52 = H52_RESULTS[c.id];
const g52i = groundedResults[c.id];
const changed = h52.enum !== g52i.returnedEnum ? "changed" : "unchanged";
console.log(`[${c.label}] 52H: ${h52.enum} → 52I: ${g52i.returnedEnum} (${changed})`);
}
});
});
/* ═══════════════════════════════════════════════════════════
* Inference timing (observational only)
* ═══════════════════════════════════════════════════════════ */
describe("Experiment 52I — Inference timing", () => {
it("records min, max, total timing for all 12 calls", () => {
expect(timingStats.min).toBeGreaterThan(0);
expect(timingStats.max).toBeGreaterThanOrEqual(timingStats.min);
expect(timingStats.total).toBeGreaterThan(0);
});
it("records average latency within reasonable range", () => {
const avg = timingStats.total / 12;
expect(avg).toBeGreaterThan(5000);
expect(avg).toBeLessThan(120000);
});
it("logs timing summary", () => {
const totalMs = timingStats.total;
const avg = Math.round(totalMs / 12);
console.log(`\n=== Experiment 52I Timing ===`);
console.log(`Calls: 12 (6 previous + 6 grounded)`);
console.log(`Total: ${totalMs}ms (~${(totalMs/1000).toFixed(1)}s)`);
console.log(`Average: ${avg}ms (~${(avg/1000).toFixed(1)}s) per call`);
console.log(`Fastest: ${timingStats.min}ms`);
console.log(`Slowest: ${timingStats.max}ms`);
});
});