docs(confidence-engine): checkpoint verified three-part report
This commit is contained in:
@@ -336,6 +336,33 @@ scripts/overview-experiment-helper.cjs — canonical overview live helper
|
||||
- Both manufacturing (with findings) and product-launch (zero Findings) A/B controlled cases pass all determinism checks.
|
||||
- **No live second-case A/B has yet been run.** The behavioural question — whether exposing unresolved uncertainty in the presence of zero Findings produces different model output than without — remains open.
|
||||
|
||||
### Three-Part Report Increment — Loop 4/2 (VERIFIED)
|
||||
|
||||
**Implementation commit:** `56b5f00` — "feat(confidence-engine): show unresolved report questions"
|
||||
|
||||
Deterministic apparatus verified that:
|
||||
- Resolved/unknown nodes are excluded from admitted evidence at prompt construction time;
|
||||
- Passing the full situationGraph with unresolved nodes vs a filtered version produces identical prompt text (18 deterministic tests, all PASS).
|
||||
|
||||
Live rendered-product verification via Playwright established that one actual investigation Report renders three distinct epistemic sections:
|
||||
|
||||
1. **What we understand** — contains evidence-backed understanding; unresolved Open Questions do not appear as supported understanding.
|
||||
2. **What remains plausible** — visibly and semantically separate; section correctly displayed its empty/default state (no supplied plausible interpretations in the verified investigation).
|
||||
3. **What remains unresolved** — derives from canonical unresolved Open Questions; rendered the canonical question text; remained visibly separate from understanding and plausible interpretations; did not answer, reinterpret, rank, prioritise, or imply blockers/readiness/confidence/evidence sufficiency/investigation completion.
|
||||
|
||||
- Deterministic verification: passed.
|
||||
- Rendered-product verification: passed for three-part epistemic separation.
|
||||
- Unresolved questions remained canonical non-evidence.
|
||||
- Zero additional experimental model calls during live verification.
|
||||
- No production code or test changes during verification.
|
||||
- **Evidence boundary:** establishes the implemented three-part boundary in the deterministic apparatus and the one observed rendered investigation. Does not generalise to universally solved, proven across all investigations, or production-complete.
|
||||
|
||||
**Unresolved — `Update report` placement/ownership:** OPEN PRODUCT QUESTION. A prescribed locator (`page.getByRole("button", { name: "Update report" })`) failed during verification, but this is a separate UI-ownership question (the control exists on the investigation detail page; its presence on the report page was not decided). The three-part report semantic/rendering increment is VERIFIED regardless.
|
||||
|
||||
**Report lifecycle generation/reuse:** not re-established by this browser run (zero-model-call budget). Existing lifecycle evidence remains authoritative unless separately retested. The inference that the report is "dynamically generated on page load" is not preserved from the verification record.
|
||||
|
||||
**Next step:** product-level review of the Investigation Report experience — do not automatically change implementation.
|
||||
|
||||
## Current product architecture
|
||||
|
||||
Three distinct routes:
|
||||
|
||||
@@ -50,6 +50,26 @@ Application-level isolation proven on deployed instance — User A does not see
|
||||
|
||||
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
|
||||
|
||||
#### Three-Part Report Increment — Loop 4/2 (VERIFIED)
|
||||
|
||||
**Implementation commit:** `56b5f00`
|
||||
**Deterministic apparatus:** PASS (18 tests in `report-epistemic-ab-input-control.test.js`).
|
||||
**Rendered-product verification via Playwright:** VERIFIED.
|
||||
|
||||
One actual Investigation Report demonstrated three distinct epistemic sections:
|
||||
1. What we understand — evidence-backed; unresolved questions excluded from understanding.
|
||||
2. What remains plausible — visibly separate (empty/default state in verified case).
|
||||
3. What remains unresolved — canonical Open Questions rendered as non-evidence, not answered/rank/interpreted.
|
||||
|
||||
- Three-part report semantic/rendering: **VERIFIED** (one observed investigation, one scenario shape).
|
||||
- Unresolved questions remained non-evidence.
|
||||
- Zero additional experimental model calls during verification; no production/test changes.
|
||||
- `Update report` placement/ownership: OPEN PRODUCT QUESTION (separate from the verified increment).
|
||||
- Report lifecycle generation/reuse not re-established by this run.
|
||||
- Evidence does not generalise to universally solved or production-complete.
|
||||
|
||||
**Next:** product-level review of Investigation Report experience.
|
||||
|
||||
#### Loop 2 (final semantic run — 2/2 valid calls)
|
||||
|
||||
Observed:
|
||||
|
||||
Reference in New Issue
Block a user