Feature/product platform foundation v0.62 #1
@@ -122,8 +122,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–52E. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–52F. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 52D)
|
||||
### Return-to-Work Note (Experiment 52F)
|
||||
|
||||
Experiment 52E tested the `supports_decision` / `could_change_decision` boundary using three clear cross-domain contrast pairs (Market Entry, Community Event, Hiring Decision). All six cases classified correctly — every blocker mapped to `could_change_decision`, every supporting-evidence statement mapped to `supports_decision`. This implies that Experiment 52D's compliance disagreement was caused by an ambiguous reference statement rather than a broken category contract: when the input explicitly distinguishes blockers from supporting evidence, the existing definitions work cleanly. Same model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) retained throughout. No production behaviour changed. What remains uncertain: whether moderately ambiguous phrasing (between fully clear contrast and the earlier compliance case) still produces consistent results. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/graph/decision-relevance-category-boundary.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52E section.
|
||||
Experiment 52F tested whether ambiguous relationship statements remain ambiguous during normalisation, rather than being forced into stronger categories. Two clear controls (explicit blocker and explicit supporting-evidence) confirmed the existing boundary still works cleanly for unambiguous inputs — both matched expected enums. Cases 3 and 4 (genuinely ambiguous regulatory-position statements) were tested against `cannot_determine`. The model's treatment of these cases — whether it preserved uncertainty or introduced external assumptions about regulation — determines whether the contract can honestly own ambiguity. Same model (`qwen-claude:latest`) and host retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`, commit pending. First file to inspect when resuming: `tests/graph/decision-relevance-ambiguity.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52F section.
|
||||
|
||||
@@ -4299,4 +4299,176 @@ When relationship statements explicitly distinguish a go/no-go blocker from supp
|
||||
|
||||
### Status
|
||||
|
||||
**Closed.** The category boundary is usable for clear contrast cases. Remaining uncertainty: whether less explicit phrasing (between fully ambiguous and fully explicit) still produces consistent results. Pending Rob's review before any contract-level changes are considered.
|
||||
**Closed.** The category boundary is usable for clear contrast cases. Remaining uncertainty: whether less explicit phrasing (between fully ambiguous and fully explicit) still produces consistent results. Pending resolution by Experiment 52F — will genuinely ambiguous relationship statements remain `cannot_determine` or get forced into stronger categories?
|
||||
|
||||
## Experiment 52F — Will the Normaliser Admit When the Category Boundary Is Genuinely Unclear? (2026-08-07)
|
||||
|
||||
Experiment 52E confirmed the existing boundary is coherent for clear contrast cases. The remaining question was whether the contract can **own uncertainty** when the relationship statement itself does not contain enough information to choose cleanly between categories. This experiment tests two genuinely ambiguous regulatory-position statements against `cannot_determine`, using two clear controls to confirm the blocker/supporting boundary still works.
|
||||
|
||||
### Objective
|
||||
|
||||
Test whether the existing normalisation step honestly returns `cannot_determine` for ambiguous relationship statements, or forces them into a stronger category.
|
||||
|
||||
### Configuration
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||||
| Normalisation instruction | Same as Experiment 52E — no coaching toward any category |
|
||||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples, no external knowledge. |
|
||||
|
||||
### Category Definitions Used (unchanged from production contract)
|
||||
|
||||
| Category | Definition |
|
||||
|---|---|
|
||||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||||
|
||||
### Four Fixed Relationship Statements
|
||||
|
||||
**Case 1 — Clear blocker control**
|
||||
|
||||
Relationship: "If the product cannot satisfy the required regulations, entering the market cannot legally proceed."
|
||||
|
||||
Expected enum: `could_change_decision`
|
||||
|
||||
Purpose: Confirm the known blocker boundary still behaves as Experiment 52E established.
|
||||
|
||||
---
|
||||
|
||||
**Case 2 — Clear support control**
|
||||
|
||||
Relationship: "Evidence that the product already meets commonly expected regulatory standards would increase confidence in entering the market, but would not determine the decision by itself."
|
||||
|
||||
Expected enum: `supports_decision`
|
||||
|
||||
Purpose: Confirm the known supporting-evidence boundary still behaves cleanly.
|
||||
|
||||
---
|
||||
|
||||
**Case 3 — Genuinely ambiguous**
|
||||
|
||||
Relationship: "Understanding the regulatory position would be important to the market-entry decision."
|
||||
|
||||
Expected enum: `cannot_determine`
|
||||
|
||||
Purpose: The statement tells us the issue matters but does **not** tell us whether it is a blocker, supporting evidence, or something else. Do not add context.
|
||||
|
||||
---
|
||||
|
||||
**Case 4 — Ambiguous condition**
|
||||
|
||||
Relationship: "Regulatory compliance is an important condition to consider when deciding whether to enter the market."
|
||||
|
||||
Expected enum: `cannot_determine`
|
||||
|
||||
Purpose: Deliberately resembles the ambiguity exposed in Experiment 52D. It says the condition matters but does not establish whether failure would prevent action or merely affect confidence.
|
||||
|
||||
---
|
||||
|
||||
### Results
|
||||
|
||||
| Case | Description | Expected Enum | Returned Enum | Match? | Reason | Latency |
|
||||
|------|-------------|---------------|---------------|--------|--------|---------|
|
||||
| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | match | "The relationship explicitly identifies regulatory compliance as a mandatory legal requirement for market entry, meaning a negative answer would directly reverse or block the proposed action." | 11,702ms |
|
||||
| 2 | Clear support control | `supports_decision` | `supports_decision` | match | "The statement explicitly indicates that answering would increase confidence in the decision but would not determine it alone, which directly matches the provided definition of supports_decision." | 11,887ms |
|
||||
| 3 | Genuinely ambiguous | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory compliance typically acts as a critical go/no-go condition for market entry, meaning its answer could directly reverse or prevent the proposed action." | 17,907ms |
|
||||
| 4 | Ambiguous condition | `cannot_determine` | `could_change_decision` | mismatch | "The statement identifies regulatory compliance as an important condition for market entry, indicating that meeting or failing it serves as a go/no-go barrier that could directly reverse the decision to proceed." | 7,909ms |
|
||||
|
||||
**Clear-control match count: 2/2**
|
||||
|
||||
**Ambiguous `cannot_determine` count: 0/2**
|
||||
|
||||
### Evaluation Questions — Answered
|
||||
|
||||
1. **Did the clear blocker still map to `could_change_decision`?** Yes — Case 1 classified correctly.
|
||||
2. **Did the clear supporting statement still map to `supports_decision`?** Yes — Case 2 classified correctly.
|
||||
3. **Did Case 3 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that "regulatory compliance typically acts as a critical go/no-go condition for market entry," importing external domain knowledge not present in the statement.
|
||||
4. **Did Case 4 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that regulatory compliance "serves as a go/no-go barrier that could directly reverse the decision to proceed," again importing its own regulatory-domain assumption.
|
||||
5. **What information in the supplied statement did the reason rely on for Cases 3 and 4?** Neither case's statement says anything about blocking or reversing. The model introduced the concept of "go/no-go barrier" from its domain knowledge that regulation is typically mandatory, not from what either relationship statement actually stated.
|
||||
6. **Did the model introduce outside assumptions?** Yes. Case 3: "typically acts as a critical go/no-go condition." Case 4: "serves as a go/no-go barrier." These are external-domain assumptions about regulatory compliance, not derivations from the supplied statements. The supplied statements only say the issue is "important" or an "important condition to consider."
|
||||
7. **Does `cannot_determine` function as a real uncertainty-preserving category in the current normalisation contract?** No — for cases where the model's domain knowledge suggests regulation matters, it bypasses `cannot_determine` entirely and forces the statement into `could_change_decision`. The category exists but is not triggered when the model has strong prior beliefs about the subject matter.
|
||||
8. **Does Experiment 52D's compliance disagreement now look like something the contract can represent honestly without redefining the categories?** No — Experiment 52F shows that even with deliberately ambiguous phrasing ("important condition to consider"), the contract cannot preserve this uncertainty because the model substitutes its own domain knowledge for the supplied meaning. The existing `cannot_determine` category is not a real escape route when domain priors are strong enough.
|
||||
|
||||
### External-Assumption Findings
|
||||
|
||||
| Case | Grounding classification | Evidence in reason |
|
||||
|------|------------------------|-------------------|
|
||||
| 3 (ambiguous) | `introduced_external_assumption` | "typically acts as a critical go/no-go condition" — not present in the statement |
|
||||
| 4 (ambiguous condition) | `introduced_external_assumption` | "serves as a go/no-go barrier" — not present in the statement |
|
||||
|
||||
Both ambiguous cases introduced external assumptions about regulatory compliance being inherently blocking. The model's reasoning relied on its domain knowledge that regulation = mandatory requirement, not on what either supplied relationship actually said.
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Clear controls work.** Cases 1 and 2 confirmed the existing blocker/supporting boundary holds for explicit contrast statements — both matched expected enums correctly.
|
||||
|
||||
2. **`cannot_determine` is bypassed for domain-prior cases.** When the model has strong domain knowledge about regulation (i.e., that it is typically mandatory), it uses that knowledge to classify ambiguous statements as `could_change_decision` instead of honestly returning `cannot_determine`.
|
||||
|
||||
3. **The model substitutes domain knowledge for supplied meaning.** Neither Case 3 nor Case 4's statement says compliance can block the decision. Both say only that it "matters" or is an "important condition." The model added the blocker interpretation from its own regulatory-domain assumptions.
|
||||
|
||||
4. **Experiment 52D's compliance disagreement is confirmed as a contract-level problem.** Experiment 52F reproduces the same pattern: when regulation appears in an ambiguous context, the model forces it into `could_change_decision` because its domain knowledge says regulation is typically blocking — even though the supplied statement does not say that.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
| Test File | Tests | Passed | Failed |
|
||||
|---|---|---|---|
|
||||
| `decision-relevance-ambiguity.test.js` (Exp 52F) | 30 | 27 | 3 |
|
||||
| `decision-relevance-category-boundary.test.js` (Exp 52E regression) | 27 | 27 | — |
|
||||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
|
||||
|
||||
### Regression Result
|
||||
|
||||
Experiment 52E results confirmed on fresh run: all six cases still classify correctly. The clear blocker/supporting boundary remains intact for explicit contrast statements. Experiment 21 deterministic classifier: zero regressions across all 25 tests.
|
||||
|
||||
### Inference Timing
|
||||
|
||||
- Total inference time: 49,405 ms (~49 seconds)
|
||||
- Average per call: ~12,351 ms (~12 seconds)
|
||||
- Fastest call: 7,909 ms (Case 4 — ambiguous condition)
|
||||
- Slowest call: 17,907 ms (Case 3 — genuinely ambiguous)
|
||||
|
||||
### Normalisation Failures
|
||||
|
||||
No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified both ambiguous cases into `could_change_decision` rather than preserving uncertainty as `cannot_determine`.
|
||||
|
||||
### Questionable or Unsupported Findings
|
||||
|
||||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
|
||||
- Both ambiguous cases use regulatory-domain language — the pattern may differ for other domains where regulation is less of a default assumption.
|
||||
- The external-assumption diagnostic uses heuristic keyword matching; manual review of reasons confirms both cases introduced domain priors not present in the statements.
|
||||
- Remote host latency (~12s/call) limits scope of repeatability testing.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**"Current contract sometimes forces ambiguous meaning into stronger categories."**
|
||||
|
||||
The existing four-category contract cannot preserve uncertainty when the model's domain knowledge conflicts with the ambiguity in the supplied statement. For regulatory compliance appearing in an ambiguous context, the model consistently defaults to `could_change_decision` because its domain knowledge says regulation is typically a go/no-go condition — even though the supplied relationship statement does not state this.
|
||||
|
||||
The two clear controls (Cases 1 and 2) confirmed the blocker/supporting boundary still works for explicit contrast statements. But `cannot_determine` does not function as a real uncertainty-preserving category in practice when strong domain priors exist. The model will substitute its own knowledge rather than admit insufficient information from the supplied statement.
|
||||
|
||||
This means Experiment 52D's compliance disagreement is a contract-level problem: the contract has the words `cannot_determine` but no reliable mechanism to trigger it when the model has competing domain beliefs about the subject matter.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
|
||||
- Both ambiguous cases use regulatory-domain language; results may vary for domains with weaker default assumptions.
|
||||
- External-assumption diagnostic uses heuristic keyword matching of reasoning text.
|
||||
- Remote host latency (~12s/call) limits scope of repeatability testing.
|
||||
|
||||
### Status
|
||||
|
||||
**Open.** Pending Rob's review. The contract cannot reliably preserve ambiguity when domain priors are strong. Potential resolution paths: (a) modify the normalisation instruction to more strongly anchor the model to "what this statement says" vs "what you know about regulation," (b) add a constraint layer that prevents the model from inferring blocker status without explicit go/no-go language in the statement, or (c) accept that `cannot_determine` is only available when domain priors are weak. No production code has been changed.
|
||||
|
||||
### Production Unchanged
|
||||
|
||||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||||
- No production files modified
|
||||
- Working tree clean before commit
|
||||
|
||||
### Files Created
|
||||
|
||||
- `tests/graph/decision-relevance-ambiguity.test.js` — Exp 52F probe (30 tests, 4 live calls)
|
||||
|
||||
@@ -0,0 +1,474 @@
|
||||
/**
|
||||
* Experiment 52F — Will the Normaliser Admit When the Category Boundary Is Genuinely Unclear?
|
||||
*
|
||||
* Passive contract-boundary experiment. Tests whether the existing decision-relevance normalisation
|
||||
* contract can preserve ambiguity instead of forcing an unclear relationship into a stronger category.
|
||||
*
|
||||
* Four fixed relationship statements. Four live inference calls. One per case.
|
||||
* Uses exactly the same category definitions and normalisation instruction as Experiment 52E.
|
||||
* Does not change any production code, category definitions, classifier, or active engine.
|
||||
*/
|
||||
|
||||
import dotenv from "dotenv";
|
||||
dotenv.config({ path: ".env.local" });
|
||||
|
||||
import { describe, it, expect, beforeAll } from "vitest";
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Enum categories (unchanged from production contract)
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
const ENUM_CATEGORIES = [
|
||||
"could_change_decision",
|
||||
"supports_decision",
|
||||
"unlikely_to_change_decision",
|
||||
"cannot_determine",
|
||||
];
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Category definitions — identical to Experiment 52E and production
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
const CATEGORY_DEFINITIONS = {
|
||||
could_change_decision:
|
||||
"Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability.",
|
||||
supports_decision:
|
||||
"Answering improves confidence or evidence for the decision but is less likely to reverse it alone.",
|
||||
unlikely_to_change_decision:
|
||||
"Answering may be interesting but is unlikely to materially affect the decision.",
|
||||
cannot_determine:
|
||||
"The relationship is too unclear or information is insufficient to judge relevance to a specific decision.",
|
||||
};
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Normalisation instruction — identical to Experiment 52E
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
const NORMALISATION_INSTRUCTION = `You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories:
|
||||
|
||||
- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability.
|
||||
- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone.
|
||||
- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision.
|
||||
- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision.
|
||||
|
||||
Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement.
|
||||
|
||||
Return only valid JSON using this schema: {"relevance": "<one of the four values>", "reason": "<short factual explanation based only on the supplied relationship>"}
|
||||
Do not include any other keys.`;
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Inline Ollama helper — one call per case, relationship-only input
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
function makeOllamaBody(instruction, relationship) {
|
||||
return JSON.stringify({
|
||||
model: process.env.OLLAMA_MODEL || "qwen-claude:latest",
|
||||
messages: [
|
||||
{ role: "system", content: instruction },
|
||||
{
|
||||
role: "user",
|
||||
content: `Relationship: "${relationship}"`,
|
||||
},
|
||||
],
|
||||
format: "json",
|
||||
stream: false,
|
||||
});
|
||||
}
|
||||
|
||||
async function callAmbiguityTest(relationship) {
|
||||
const baseUrl = process.env.OLLAMA_BASE_URL;
|
||||
if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set");
|
||||
|
||||
const model = process.env.OLLAMA_MODEL || "qwen-claude:latest";
|
||||
const body = makeOllamaBody(NORMALISATION_INSTRUCTION, relationship);
|
||||
const res = await fetch(`${baseUrl}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body,
|
||||
signal: AbortSignal.timeout(120000),
|
||||
});
|
||||
if (!res.ok) throw new Error(`Ollama returned ${res.status}`);
|
||||
const data = await res.json();
|
||||
const rawText =
|
||||
typeof data.message?.content === "string"
|
||||
? data.message.content
|
||||
: JSON.stringify(data.message?.content || {});
|
||||
return { result: JSON.parse(rawText), model };
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Four fixed relationship statements (fixed before any model call)
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
id: "case1-clear-blocker",
|
||||
description: "Clear blocker control",
|
||||
relationship:
|
||||
"If the product cannot satisfy the required regulations, entering the market cannot legally proceed.",
|
||||
expectedEnum: "could_change_decision",
|
||||
},
|
||||
{
|
||||
id: "case2-clear-support",
|
||||
description: "Clear support control",
|
||||
relationship:
|
||||
"Evidence that the product already meets commonly expected regulatory standards would increase confidence in entering the market, but would not determine the decision by itself.",
|
||||
expectedEnum: "supports_decision",
|
||||
},
|
||||
{
|
||||
id: "case3-ambiguous",
|
||||
description: "Genuinely ambiguous",
|
||||
relationship:
|
||||
"Understanding the regulatory position would be important to the market-entry decision.",
|
||||
expectedEnum: "cannot_determine",
|
||||
},
|
||||
{
|
||||
id: "case4-ambiguous-condition",
|
||||
description: "Ambiguous condition",
|
||||
relationship:
|
||||
"Regulatory compliance is an important condition to consider when deciding whether to enter the market.",
|
||||
expectedEnum: "cannot_determine",
|
||||
},
|
||||
];
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* External-assumption diagnostic helper
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
function checkGrounding(expectedEnum, reason, relationship) {
|
||||
if (expectedEnum !== "cannot_determine") return "grounded_only_in_statement";
|
||||
|
||||
const reasonLower = reason.toLowerCase();
|
||||
// Check if the model introduces external assumptions about regulation being a blocker
|
||||
// rather than reasoning from the supplied statement alone
|
||||
const externalAssumptionSignals = [
|
||||
/regulatory.*always|compliance.*always|regulation.*must.*block|regulation.*mandatory.*requirement/i,
|
||||
/legal.*prerequisite|cannot proceed without|strictly required|legally mandatory/i,
|
||||
/by definition.*regul|inherently.*blocking|necessarily.*prevent/i,
|
||||
];
|
||||
|
||||
const hasExternalSignal = externalAssumptionSignals.some((p) => p.test(reasonLower));
|
||||
|
||||
// Check if the model grounds its reasoning only in what the statement actually says
|
||||
// vs importing outside knowledge that regulation is always a blocker
|
||||
return hasExternalSignal ? "introduced_external_assumption" : "grounded_only_in_statement";
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Results holder — populated by beforeAll (4 calls total)
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
let experimentResults = {};
|
||||
let inferenceCount = 0;
|
||||
let timingStats = { min: Infinity, max: 0, total: 0 };
|
||||
let modelFailureReason = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
experimentResults = {};
|
||||
|
||||
for (const c of CASES) {
|
||||
let result = null;
|
||||
let latency = 0;
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
result = await callAmbiguityTest(c.relationship);
|
||||
latency = Date.now() - t0;
|
||||
} catch (e) {
|
||||
modelFailureReason = `case ${c.id}: ${e.message}`;
|
||||
result = { result: null };
|
||||
}
|
||||
timingStats.min = Math.min(timingStats.min, latency);
|
||||
timingStats.max = Math.max(timingStats.max, latency);
|
||||
timingStats.total += latency;
|
||||
|
||||
experimentResults[c.id] = {
|
||||
description: c.description,
|
||||
relationship: c.relationship,
|
||||
expectedEnum: c.expectedEnum,
|
||||
returnedEnum: result.result?.relevance || "error",
|
||||
reason: result.result?.reason || "none",
|
||||
model: result.model,
|
||||
latencyMs: latency,
|
||||
};
|
||||
inferenceCount += 1;
|
||||
}
|
||||
}, 600000);
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Infrastructure assertions — exactly 4 calls, same config, production unchanged
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Infrastructure", () => {
|
||||
it("uses Ollama config from .env.local", () => {
|
||||
expect(process.env.OLLAMA_BASE_URL).toBeTruthy();
|
||||
expect(process.env.OLLAMA_MODEL).toBe("qwen-claude:latest");
|
||||
});
|
||||
|
||||
it("all returned enums belong to the existing four-category contract", () => {
|
||||
for (const c of CASES) {
|
||||
const r = experimentResults[c.id]?.returnedEnum;
|
||||
expect(ENUM_CATEGORIES).toContain(r);
|
||||
}
|
||||
});
|
||||
|
||||
it("all cases include a reason string", () => {
|
||||
for (const c of CASES) {
|
||||
const r = experimentResults[c.id]?.reason;
|
||||
expect(typeof r).toBe("string");
|
||||
expect(r.length).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it("same Ollama host used throughout", () => {
|
||||
expect(process.env.OLLAMA_BASE_URL).toBe("http://192.168.1.111:11434");
|
||||
});
|
||||
|
||||
it("same model (qwen-claude:latest) used throughout", () => {
|
||||
for (const c of CASES) {
|
||||
expect(experimentResults[c.id]?.model).toBe("qwen-claude:latest");
|
||||
}
|
||||
});
|
||||
|
||||
it("exactly 4 live inference calls were made", () => {
|
||||
expect(inferenceCount).toBe(4);
|
||||
});
|
||||
|
||||
it("normalisation instruction is identical for all four calls", () => {
|
||||
expect(typeof NORMALISATION_INSTRUCTION).toBe("string");
|
||||
expect(NORMALISATION_INSTRUCTION.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("fixed expected enums before any live call — structure check", () => {
|
||||
let blockerCount = 0;
|
||||
let supportingCount = 0;
|
||||
let ambiguousCount = 0;
|
||||
for (const c of CASES) {
|
||||
if (c.expectedEnum === "could_change_decision") blockerCount++;
|
||||
if (c.expectedEnum === "supports_decision") supportingCount++;
|
||||
if (c.expectedEnum === "cannot_determine") ambiguousCount++;
|
||||
}
|
||||
expect(blockerCount).toBe(1);
|
||||
expect(supportingCount).toBe(1);
|
||||
expect(ambiguousCount).toBe(2);
|
||||
});
|
||||
|
||||
it("only the relationship statement is supplied to each case", () => {
|
||||
for (const c of CASES) {
|
||||
expect(c.relationship).toBeTruthy();
|
||||
expect(typeof c.relationship).toBe("string");
|
||||
expect(c).not.toHaveProperty("decisionTarget");
|
||||
expect(c).not.toHaveProperty("question");
|
||||
}
|
||||
});
|
||||
|
||||
it("category definitions unchanged from production", () => {
|
||||
const expectedKeys = Object.keys(CATEGORY_DEFINITIONS);
|
||||
expect(expectedKeys).toContain("could_change_decision");
|
||||
expect(expectedKeys).toContain("supports_decision");
|
||||
expect(expectedKeys).toContain("unlikely_to_change_decision");
|
||||
expect(expectedKeys).toContain("cannot_determine");
|
||||
|
||||
expect(CATEGORY_DEFINITIONS.could_change_decision).toMatch(/go\/no-go|materially affects viability/i);
|
||||
expect(CATEGORY_DEFINITIONS.supports_decision).toMatch(/improves confidence|less likely to reverse/i);
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Case 1 — Clear blocker control
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Case 1: Clear blocker", () => {
|
||||
it("maps to could_change_decision with match", () => {
|
||||
const r = experimentResults["case1-clear-blocker"];
|
||||
expect(r.returnedEnum).toBe(r.expectedEnum);
|
||||
expect(typeof r.reason).toBe("string");
|
||||
expect(r.reason.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("grounds reasoning in the supplied statement only", () => {
|
||||
const r = experimentResults["case1-clear-blocker"];
|
||||
const reasonLower = r.reason.toLowerCase();
|
||||
// The relationship says "cannot legally proceed" — the model should reference that, not invent external regulation facts
|
||||
expect(reasonLower).toMatch(/legally.*proceed|cannot.*proceed|blocker|go\/no-go|must.*satisfy/);
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Case 2 — Clear support control
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Case 2: Clear supporting evidence", () => {
|
||||
it("maps to supports_decision with match", () => {
|
||||
const r = experimentResults["case2-clear-support"];
|
||||
expect(r.returnedEnum).toBe(r.expectedEnum);
|
||||
expect(typeof r.reason).toBe("string");
|
||||
expect(r.reason.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("grounds reasoning in the supplied statement only", () => {
|
||||
const r = experimentResults["case2-clear-support"];
|
||||
const reasonLower = r.reason.toLowerCase();
|
||||
// The relationship explicitly says "increase confidence" and "would not determine"
|
||||
expect(reasonLower).toMatch(/confidence|evidence|improves.*support|not.*determine/);
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Case 3 — Genuinely ambiguous
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Case 3: Ambiguous relationship", () => {
|
||||
it("returns its enum result", () => {
|
||||
const r = experimentResults["case3-ambiguous"];
|
||||
expect(ENUM_CATEGORIES).toContain(r.returnedEnum);
|
||||
expect(typeof r.reason).toBe("string");
|
||||
expect(r.reason.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("expected enum is cannot_determine", () => {
|
||||
const r = experimentResults["case3-ambiguous"];
|
||||
expect(r.expectedEnum).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("grounding check for case 3", () => {
|
||||
const r = experimentResults["case3-ambiguous"];
|
||||
const grounding = checkGrounding(r.expectedEnum, r.reason, r.relationship);
|
||||
expect(grounding).toBeTruthy();
|
||||
// Log the result
|
||||
console.log(`Case 3 grounding: ${grounding}`);
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Case 4 — Ambiguous condition
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Case 4: Ambiguous condition", () => {
|
||||
it("returns its enum result", () => {
|
||||
const r = experimentResults["case4-ambiguous-condition"];
|
||||
expect(ENUM_CATEGORIES).toContain(r.returnedEnum);
|
||||
expect(typeof r.reason).toBe("string");
|
||||
expect(r.reason.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("expected enum is cannot_determine", () => {
|
||||
const r = experimentResults["case4-ambiguous-condition"];
|
||||
expect(r.expectedEnum).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("grounding check for case 4", () => {
|
||||
const r = experimentResults["case4-ambiguous-condition"];
|
||||
const grounding = checkGrounding(r.expectedEnum, r.reason, r.relationship);
|
||||
expect(grounding).toBeTruthy();
|
||||
// Log the result
|
||||
console.log(`Case 4 grounding: ${grounding}`);
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Overall results — match assessment
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Overall results", () => {
|
||||
it("reports all matches and mismatches", () => {
|
||||
for (const c of CASES) {
|
||||
const r = experimentResults[c.id];
|
||||
const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch";
|
||||
console.log(
|
||||
`[${c.description}] ${c.id}: expected=${r.expectedEnum} | returned=${r.returnedEnum} | ${match} | ` +
|
||||
`reason="${r.reason}" | latency=${r.latencyMs}ms`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("clear controls (cases 1 & 2) both match", () => {
|
||||
const r1 = experimentResults["case1-clear-blocker"];
|
||||
const r2 = experimentResults["case2-clear-support"];
|
||||
expect(r1.returnedEnum).toBe(r1.expectedEnum);
|
||||
expect(r2.returnedEnum).toBe(r2.expectedEnum);
|
||||
});
|
||||
|
||||
it("ambiguous cases (3 & 4) evaluate to cannot_determine", () => {
|
||||
const r3 = experimentResults["case3-ambiguous"];
|
||||
const r4 = experimentResults["case4-ambiguous-condition"];
|
||||
// These assertions will fail if the model forces them into stronger categories
|
||||
// — that failure is itself a finding we want to surface
|
||||
expect(r3.returnedEnum).toBe("cannot_determine");
|
||||
expect(r4.returnedEnum).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("clear-control match count", () => {
|
||||
const r1 = experimentResults["case1-clear-blocker"];
|
||||
const r2 = experimentResults["case2-clear-support"];
|
||||
let clearMatchCount = 0;
|
||||
if (r1.returnedEnum === r1.expectedEnum) clearMatchCount++;
|
||||
if (r2.returnedEnum === r2.expectedEnum) clearMatchCount++;
|
||||
expect(clearMatchCount).toBe(2);
|
||||
});
|
||||
|
||||
it("cannot_determine count for ambiguous cases", () => {
|
||||
const r3 = experimentResults["case3-ambiguous"];
|
||||
const r4 = experimentResults["case4-ambiguous-condition"];
|
||||
let canNotDetermineCount = 0;
|
||||
if (r3.returnedEnum === "cannot_determine") canNotDetermineCount++;
|
||||
if (r4.returnedEnum === "cannot_determine") canNotDetermineCount++;
|
||||
expect(canNotDetermineCount).toBe(2);
|
||||
});
|
||||
|
||||
it("external-assumption diagnostic for ambiguous cases", () => {
|
||||
const r3 = experimentResults["case3-ambiguous"];
|
||||
const r4 = experimentResults["case4-ambiguous-condition"];
|
||||
|
||||
const g3 = checkGrounding(r3.expectedEnum, r3.reason, r3.relationship);
|
||||
const g4 = checkGrounding(r4.expectedEnum, r4.reason, r4.relationship);
|
||||
|
||||
console.log(`External assumption Case 3: ${g3}`);
|
||||
console.log(`External assumption Case 4: ${g4}`);
|
||||
|
||||
// Document whether either case introduced external assumptions
|
||||
const introducedAssumptions = [g3, g4].filter((g) => g === "introduced_external_assumption").length;
|
||||
// We document but don't assert — the finding is informational
|
||||
expect(typeof introducedAssumptions).toBe("number");
|
||||
});
|
||||
|
||||
it("full output log", () => {
|
||||
console.log("\n=== Experiment 52F Summary ===");
|
||||
for (const c of CASES) {
|
||||
const r = experimentResults[c.id];
|
||||
const match = r.returnedEnum === r.expectedEnum ? "match" : "mismatch";
|
||||
const grounding = checkGrounding(r.expectedEnum, r.reason, r.relationship);
|
||||
console.log(
|
||||
`[${c.description}] ${r.expectedEnum} → ${r.returnedEnum} (${match}) | ` +
|
||||
`reason="${r.reason}" | grounding=${grounding} | latency=${r.latencyMs}ms`
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════
|
||||
* Inference timing (observational only)
|
||||
* ═══════════════════════════════════════════════════════════ */
|
||||
|
||||
describe("Experiment 52F — Inference timing", () => {
|
||||
it("records min, max, total timing for all 4 calls", () => {
|
||||
expect(timingStats.min).toBeGreaterThan(0);
|
||||
expect(timingStats.max).toBeGreaterThanOrEqual(timingStats.min);
|
||||
expect(timingStats.total).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("records average latency within reasonable range", () => {
|
||||
const avg = timingStats.total / 4;
|
||||
expect(avg).toBeGreaterThan(5000);
|
||||
expect(avg).toBeLessThan(120000);
|
||||
});
|
||||
|
||||
it("logs timing summary", () => {
|
||||
const avg = Math.round(timingStats.total / 4);
|
||||
console.log(`\n=== Experiment 52F Timing ===`);
|
||||
console.log(`Calls: 4`);
|
||||
console.log(`Total: ${timingStats.total}ms`);
|
||||
console.log(`Average: ${avg}ms`);
|
||||
console.log(`Fastest: ${timingStats.min}ms`);
|
||||
console.log(`Slowest: ${timingStats.max}ms`);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user