test(confidence-engine): control report epistemic comparison inputs
Deterministic test (17 cases) proves A/B inputs are correctly controlled: - supportedEvidence identical in A and B - findings identical in A and B - plausibleInterpretations identical (fixed canonical interpretation) in A and B - scenarioState identical in A and B - providerConfig identical in A and B - only unresolvedQuestions visibility differs (A=hidden, B=canonical) - unresolved questions excluded from evidence and plausible interpretations - fixed interpretation carries NON-EVIDENCE epistemic status Also corrects Loop 2 documentation to reflect that three-category invariance was NOT established (plausibleInterpretations differed between calls despite identical input), retaining only the narrower valid finding: no unresolved Open Question was promoted to evidence. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -44,6 +44,45 @@ Application-level isolation proven on deployed instance — User A does not see
|
||||
|
||||
> Created by Experiment 27. This document is the starting point for any fresh session working on the Confidence Engine. Read this first, then follow the routing table below to task-specific references.
|
||||
|
||||
## Session Continuity — 2026-09-30 (v0.63)
|
||||
|
||||
### Epistemic Completeness Experiment — A/B Comparison, Loops 1-3 (closed)
|
||||
|
||||
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
|
||||
|
||||
#### Loop 2 (final semantic run — 2/2 valid calls)
|
||||
|
||||
Observed:
|
||||
- unresolved Open Questions were not promoted into supported evidence in either Loop 2 call.
|
||||
|
||||
Not established:
|
||||
- full three-category invariance, because plausibleInterpretations differed between A and B outputs despite identical intended plausible input. Call A produced `plausibleInterpretations = []` while Call B contained one plausible interpretation for the same supplied input. Therefore unresolved visibility was not the sole output difference.
|
||||
|
||||
Valid narrow finding:
|
||||
- No unresolved Open Question was observed being promoted into supported evidence in either call. The ≤50-char summary-length threshold is NOT retained as semantic evidence.
|
||||
|
||||
**Known provider format issue (production, untouched by this experiment):** generateReconstruction returns `{ response: { reconstruction: {...}, evidence: {...}, ... }, providerApiPath, providerExecution }`. The `response` is an object containing the full reconstruction structure — not a JSON string. The existing `investigation-overview-synthesis.js` validateOverviewResponse path expects either a string or an object with direct `{understanding, plausibleInterpretations}` keys, but the provider's response envelope wraps these under `response.reconstruction`. **Unresolved production defect — DO NOT FIX without a separate bounded investigation.**
|
||||
|
||||
#### Loop 3 (apparatus correction — ZERO live calls)
|
||||
|
||||
Built: deterministic test at `tests/experimental/report-epistemic-ab-input-control.test.js` (17 tests, all PASS).
|
||||
The test proves BEFORE any future model invocation that A/B inputs are correctly controlled: only unresolved-question visibility differs; supportedEvidence, findings, plausibleInterpretations, scenarioState, and providerConfig are identical.
|
||||
|
||||
#### Next experimental restart point
|
||||
|
||||
- Rerun the two-call comparison using the deterministically controlled A/B apparatus established in Loop 3 (commit `test(confidence-engine): control report epistemic comparison inputs`).
|
||||
- The fixed plausible interpretation is canonical: "The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality."
|
||||
- Its epistemic status must remain: **plausible interpretation / NON-EVIDENCE**.
|
||||
|
||||
### Tester-Readiness Position
|
||||
```
|
||||
controlled external testing: GO
|
||||
known pre-tester security blockers: none identified
|
||||
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
|
||||
```
|
||||
|
||||
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
|
||||
|
||||
## 1. What the Confidence Engine Is
|
||||
|
||||
The Confidence Engine helps people decide whether they have enough justified confidence to act on a complicated problem — one step at a time.
|
||||
|
||||
Reference in New Issue
Block a user