diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 4b372f0..6f4280b 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -256,7 +256,7 @@ Manual Jenkins deployment pipeline established, version-controlled, and LIVE PRO - **Branch:** `feature/product-platform-foundation-v0.62` - **HEAD:** *(checkpoint commit — see git log for actual SHA)* -- **Working tree:** clean +- **Working tree:** clean (after Loop 3 apparatus commit) ## Architectural Conclusion @@ -295,6 +295,7 @@ tsx runtime — alias-capable experiment execut import-only proof — prove startCase() import chain resolves without live calls OpenAI reconstruction experiment seam — START_CASE_EXPERIMENT_PROVIDER=openai (experiment-only) focused-deconstruction schema fix — plumbing verified, 48/48 tests pass +tests/experimental/report-epistemic-ab-input-control.test.js — deterministic A/B input control apparatus (17 tests, zero live calls) ``` **No other experimental apparatus is currently valid.** Obsolete setups have been removed or superseded. diff --git a/docs/current-project-state.md b/docs/current-project-state.md index 72c0ef5..f1674b6 100644 --- a/docs/current-project-state.md +++ b/docs/current-project-state.md @@ -44,6 +44,45 @@ Application-level isolation proven on deployed instance — User A does not see > Created by Experiment 27. This document is the starting point for any fresh session working on the Confidence Engine. Read this first, then follow the routing table below to task-specific references. +## Session Continuity — 2026-09-30 (v0.63) + +### Epistemic Completeness Experiment — A/B Comparison, Loops 1-3 (closed) + +**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section? + +#### Loop 2 (final semantic run — 2/2 valid calls) + +Observed: +- unresolved Open Questions were not promoted into supported evidence in either Loop 2 call. + +Not established: +- full three-category invariance, because plausibleInterpretations differed between A and B outputs despite identical intended plausible input. Call A produced `plausibleInterpretations = []` while Call B contained one plausible interpretation for the same supplied input. Therefore unresolved visibility was not the sole output difference. + +Valid narrow finding: +- No unresolved Open Question was observed being promoted into supported evidence in either call. The ≤50-char summary-length threshold is NOT retained as semantic evidence. + +**Known provider format issue (production, untouched by this experiment):** generateReconstruction returns `{ response: { reconstruction: {...}, evidence: {...}, ... }, providerApiPath, providerExecution }`. The `response` is an object containing the full reconstruction structure — not a JSON string. The existing `investigation-overview-synthesis.js` validateOverviewResponse path expects either a string or an object with direct `{understanding, plausibleInterpretations}` keys, but the provider's response envelope wraps these under `response.reconstruction`. **Unresolved production defect — DO NOT FIX without a separate bounded investigation.** + +#### Loop 3 (apparatus correction — ZERO live calls) + +Built: deterministic test at `tests/experimental/report-epistemic-ab-input-control.test.js` (17 tests, all PASS). +The test proves BEFORE any future model invocation that A/B inputs are correctly controlled: only unresolved-question visibility differs; supportedEvidence, findings, plausibleInterpretations, scenarioState, and providerConfig are identical. + +#### Next experimental restart point + +- Rerun the two-call comparison using the deterministically controlled A/B apparatus established in Loop 3 (commit `test(confidence-engine): control report epistemic comparison inputs`). +- The fixed plausible interpretation is canonical: "The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality." +- Its epistemic status must remain: **plausible interpretation / NON-EVIDENCE**. + +### Tester-Readiness Position +``` +controlled external testing: GO +known pre-tester security blockers: none identified +next uncertainty worth reducing: real first-time-user product value, trust and independent usability +``` + +Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit. + ## 1. What the Confidence Engine Is The Confidence Engine helps people decide whether they have enough justified confidence to act on a complicated problem — one step at a time. diff --git a/tests/experimental/report-epistemic-ab-input-control.test.js b/tests/experimental/report-epistemic-ab-input-control.test.js new file mode 100644 index 0000000..54a4e72 --- /dev/null +++ b/tests/experimental/report-epistemic-ab-input-control.test.js @@ -0,0 +1,143 @@ +/** + * Deterministic control of report epistemic comparison inputs. + * + * Proves BEFORE any future model invocation that the A/B experiment + * inputs are correctly controlled: only unresolved-question visibility + * differs between A and B; everything else is identical. + * + * No live model calls. No mocking. Pure input-contract verification. + */ + +import { describe, it, expect } from "vitest"; +import fixture from "../fixtures/progressed-investigation-manufacturing.json"; + +const FIXED_PLAUSIBLE = + "The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality."; + +function buildABInputs(fix, includeUQ) { + const sg = fix.situationGraph; + const se = sg.nodes.filter(n => n.status === "known" || n.status === "supported").map(n => ({id: n.id, kind: n.kind, status: n.status, label: n.label})); + const fi = fix.findings.filter(f2 => f2.status === "provisional" && f2.userDisposition === null).map(f3 => ({id: f3.id, proposition: f3.proposition})); + const pi = [FIXED_PLAUSIBLE]; + const uq = sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").map(n => ({id: n.id, label: n.label, description: n.description})); + const ss = {centralStatement: sg.centralStatement, activeUnknownNodeId: sg.activeUnknownNodeId, resolvedNodeIds: [...sg.resolvedNodeIds], nodeCount: sg.nodes.length, edgeCount: sg.edges.length}; + const pc = {provider: "ollama", model: process.env.OLLAMA_MODEL || "configured-model"}; + return {supportedEvidence: se, findings: fi, plausibleInterpretations: pi, unresolvedQuestions: includeUQ ? [...uq] : [], scenarioState: ss, providerConfig: pc}; +} + +function deepEqual(a, b) { + if (a === b) return true; + if (a == null || b == null) return false; + if (typeof a !== typeof b) return false; + if (Array.isArray(a)) { if (!Array.isArray(b) || a.length !== b.length) return false; for (let i = 0; i < a.length; i++) if (!deepEqual(a[i], b[i])) return false; return true; } + if (typeof a === "object") { const kA = Object.keys(a).sort(), kB = Object.keys(b).sort(); if (kA.length !== kB.length) return false; for (let i = 0; i < kA.length; i++) if (kA[i] !== kB[i] || !deepEqual(a[kA[i]], b[kB[i]])) return false; return true; } + return false; +} + +describe("Controlled A/B report epistemic comparison inputs", () => { + it("builds canonical input objects for both A and B", () => { + const a = buildABInputs(fixture, false); + const b = buildABInputs(fixture, true); + ["supportedEvidence","findings","plausibleInterpretations","unresolvedQuestions","scenarioState","providerConfig"].forEach(k => { expect(a).toHaveProperty(k); expect(b).toHaveProperty(k); }); + }); + + it("A and B have identical admitted evidence", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true); + }); + + it("A and B have identical eligible Findings", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(deepEqual(a.findings, b.findings)).toBe(true); + }); + + it("A and B have identical fixed plausibleInterpretations", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true); + }); + + it("fixed plausible interpretation matches canonical value exactly", () => { + const a = buildABInputs(fixture, false); + expect(a.plausibleInterpretations).toHaveLength(1); + expect(a.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE); + }); + + it("A and B reference the same progressed investigation state", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true); + }); + + it("provider/model configuration is identical for A and B", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(a.providerConfig.provider).toBe(b.providerConfig.provider); + expect(a.providerConfig.model).toBe(b.providerConfig.model); + }); + + it("A does not expose unresolved Open Questions (hidden)", () => { + const a = buildABInputs(fixture, false); + expect(a.unresolvedQuestions).toHaveLength(0); + }); + + it("B exposes exactly the canonical unresolved Open Questions", () => { + const b = buildABInputs(fixture, true); + const fu = fixture.situationGraph.nodes.filter(n => n.kind === "unknown" && n.status === "unknown"); + expect(b.unresolvedQuestions).toHaveLength(fu.length); + for (const node of fu) { const found = b.unresolvedQuestions.find(uq => uq.id === node.id); expect(found).toBeDefined(); expect(found.label).toBe(node.label); expect(found.description).toBe(node.description); } + }); + + it("only unresolved visibility differs between A and B", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true); + expect(deepEqual(a.findings, b.findings)).toBe(true); + expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true); + expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true); + expect(deepEqual(a.providerConfig, b.providerConfig)).toBe(true); + expect(a.unresolvedQuestions).not.toEqual(b.unresolvedQuestions); + }); + + it("fixed plausible interpretation is not admitted evidence", () => { + const a = buildABInputs(fixture, false); + for (const e of a.supportedEvidence) { expect(e.label).not.toContain(FIXED_PLAUSIBLE); expect(["known","supported"]).toContain(e.status); } + }); + + it("unresolved questions are not admitted evidence", () => { + const b = buildABInputs(fixture, true); + for (const uq of b.unresolvedQuestions) { const fn = fixture.situationGraph.nodes.find(n => n.id === uq.id); expect(fn.status).toBe("unknown"); expect(fn.kind).toBe("unknown"); } + }); + + it("unresolved questions are not plausibleInterpretations", () => { + const b = buildABInputs(fixture, true); + for (const uq of b.unresolvedQuestions) { expect(b.plausibleInterpretations).not.toContain(uq.label); expect(b.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE); } + }); + + it("no fixture node conflates unresolved and evidence categories", () => { + for (const node of fixture.situationGraph.nodes) { + if (node.kind === "unknown" && node.status === "unknown") expect(["known","supported"]).not.toContain(node.status); + else if (["known","supported"].includes(node.status)) expect(node.kind).not.toBe("unknown"); + } + }); + + it("inputs are constructed purely from fixture without provider or model dependency", () => { + const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true); + expect(typeof a.supportedEvidence).toBe("object"); + expect(typeof a.findings).toBe("object"); + expect(typeof a.plausibleInterpretations).toBe("object"); + expect(typeof a.unresolvedQuestions).toBe("object"); + }); + + it("fixed plausible interpretation carries NON-EVIDENCE epistemic status", () => { + const a = buildABInputs(fixture, false); + expect(a.plausibleInterpretations).toHaveLength(1); + expect(a.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE); + const el = a.supportedEvidence.map(e => e.label.toLowerCase()); + const il = FIXED_PLAUSIBLE.toLowerCase(); + for (const label of el) expect(label).not.toContain(il); + }); + + it("fixture has the minimum required content for A/B comparison", () => { + const sg = fixture.situationGraph; + expect(sg.nodes.filter(n => ["known","supported"].includes(n.status)).length).toBeGreaterThan(0); + expect(sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").length).toBeGreaterThan(0); + expect(fixture.findings.filter(f => f.status === "provisional" && f.userDisposition === null).length).toBeGreaterThan(0); + }); +}); \ No newline at end of file