test(confidence-engine): control report epistemic comparison inputs

Deterministic test (17 cases) proves A/B inputs are correctly controlled:
- supportedEvidence identical in A and B
- findings identical in A and B
- plausibleInterpretations identical (fixed canonical interpretation) in A and B
- scenarioState identical in A and B
- providerConfig identical in A and B
- only unresolvedQuestions visibility differs (A=hidden, B=canonical)
- unresolved questions excluded from evidence and plausible interpretations
- fixed interpretation carries NON-EVIDENCE epistemic status

Also corrects Loop 2 documentation to reflect that three-category
invariance was NOT established (plausibleInterpretations differed
between calls despite identical input), retaining only the narrower
valid finding: no unresolved Open Question was promoted to evidence.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
2026-09-30 11:44:51 +01:00
co-authored by Claude Code
parent ef6592cd91
commit ada83de1db
3 changed files with 184 additions and 1 deletions
+2 -1
View File
@@ -256,7 +256,7 @@ Manual Jenkins deployment pipeline established, version-controlled, and LIVE PRO
- **Branch:** `feature/product-platform-foundation-v0.62`
- **HEAD:** *(checkpoint commit — see git log for actual SHA)*
- **Working tree:** clean
- **Working tree:** clean (after Loop 3 apparatus commit)
## Architectural Conclusion
@@ -295,6 +295,7 @@ tsx runtime — alias-capable experiment execut
import-only proof — prove startCase() import chain resolves without live calls
OpenAI reconstruction experiment seam — START_CASE_EXPERIMENT_PROVIDER=openai (experiment-only)
focused-deconstruction schema fix — plumbing verified, 48/48 tests pass
tests/experimental/report-epistemic-ab-input-control.test.js — deterministic A/B input control apparatus (17 tests, zero live calls)
```
**No other experimental apparatus is currently valid.** Obsolete setups have been removed or superseded.
+39
View File
@@ -44,6 +44,45 @@ Application-level isolation proven on deployed instance — User A does not see
> Created by Experiment 27. This document is the starting point for any fresh session working on the Confidence Engine. Read this first, then follow the routing table below to task-specific references.
## Session Continuity — 2026-09-30 (v0.63)
### Epistemic Completeness Experiment — A/B Comparison, Loops 1-3 (closed)
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
#### Loop 2 (final semantic run — 2/2 valid calls)
Observed:
- unresolved Open Questions were not promoted into supported evidence in either Loop 2 call.
Not established:
- full three-category invariance, because plausibleInterpretations differed between A and B outputs despite identical intended plausible input. Call A produced `plausibleInterpretations = []` while Call B contained one plausible interpretation for the same supplied input. Therefore unresolved visibility was not the sole output difference.
Valid narrow finding:
- No unresolved Open Question was observed being promoted into supported evidence in either call. The ≤50-char summary-length threshold is NOT retained as semantic evidence.
**Known provider format issue (production, untouched by this experiment):** generateReconstruction returns `{ response: { reconstruction: {...}, evidence: {...}, ... }, providerApiPath, providerExecution }`. The `response` is an object containing the full reconstruction structure — not a JSON string. The existing `investigation-overview-synthesis.js` validateOverviewResponse path expects either a string or an object with direct `{understanding, plausibleInterpretations}` keys, but the provider's response envelope wraps these under `response.reconstruction`. **Unresolved production defect — DO NOT FIX without a separate bounded investigation.**
#### Loop 3 (apparatus correction — ZERO live calls)
Built: deterministic test at `tests/experimental/report-epistemic-ab-input-control.test.js` (17 tests, all PASS).
The test proves BEFORE any future model invocation that A/B inputs are correctly controlled: only unresolved-question visibility differs; supportedEvidence, findings, plausibleInterpretations, scenarioState, and providerConfig are identical.
#### Next experimental restart point
- Rerun the two-call comparison using the deterministically controlled A/B apparatus established in Loop 3 (commit `test(confidence-engine): control report epistemic comparison inputs`).
- The fixed plausible interpretation is canonical: "The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality."
- Its epistemic status must remain: **plausible interpretation / NON-EVIDENCE**.
### Tester-Readiness Position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
## 1. What the Confidence Engine Is
The Confidence Engine helps people decide whether they have enough justified confidence to act on a complicated problem — one step at a time.