docs(confidence-engine): checkpoint verified three-part report
This commit is contained in:
@@ -336,6 +336,33 @@ scripts/overview-experiment-helper.cjs — canonical overview live helper
|
|||||||
- Both manufacturing (with findings) and product-launch (zero Findings) A/B controlled cases pass all determinism checks.
|
- Both manufacturing (with findings) and product-launch (zero Findings) A/B controlled cases pass all determinism checks.
|
||||||
- **No live second-case A/B has yet been run.** The behavioural question — whether exposing unresolved uncertainty in the presence of zero Findings produces different model output than without — remains open.
|
- **No live second-case A/B has yet been run.** The behavioural question — whether exposing unresolved uncertainty in the presence of zero Findings produces different model output than without — remains open.
|
||||||
|
|
||||||
|
### Three-Part Report Increment — Loop 4/2 (VERIFIED)
|
||||||
|
|
||||||
|
**Implementation commit:** `56b5f00` — "feat(confidence-engine): show unresolved report questions"
|
||||||
|
|
||||||
|
Deterministic apparatus verified that:
|
||||||
|
- Resolved/unknown nodes are excluded from admitted evidence at prompt construction time;
|
||||||
|
- Passing the full situationGraph with unresolved nodes vs a filtered version produces identical prompt text (18 deterministic tests, all PASS).
|
||||||
|
|
||||||
|
Live rendered-product verification via Playwright established that one actual investigation Report renders three distinct epistemic sections:
|
||||||
|
|
||||||
|
1. **What we understand** — contains evidence-backed understanding; unresolved Open Questions do not appear as supported understanding.
|
||||||
|
2. **What remains plausible** — visibly and semantically separate; section correctly displayed its empty/default state (no supplied plausible interpretations in the verified investigation).
|
||||||
|
3. **What remains unresolved** — derives from canonical unresolved Open Questions; rendered the canonical question text; remained visibly separate from understanding and plausible interpretations; did not answer, reinterpret, rank, prioritise, or imply blockers/readiness/confidence/evidence sufficiency/investigation completion.
|
||||||
|
|
||||||
|
- Deterministic verification: passed.
|
||||||
|
- Rendered-product verification: passed for three-part epistemic separation.
|
||||||
|
- Unresolved questions remained canonical non-evidence.
|
||||||
|
- Zero additional experimental model calls during live verification.
|
||||||
|
- No production code or test changes during verification.
|
||||||
|
- **Evidence boundary:** establishes the implemented three-part boundary in the deterministic apparatus and the one observed rendered investigation. Does not generalise to universally solved, proven across all investigations, or production-complete.
|
||||||
|
|
||||||
|
**Unresolved — `Update report` placement/ownership:** OPEN PRODUCT QUESTION. A prescribed locator (`page.getByRole("button", { name: "Update report" })`) failed during verification, but this is a separate UI-ownership question (the control exists on the investigation detail page; its presence on the report page was not decided). The three-part report semantic/rendering increment is VERIFIED regardless.
|
||||||
|
|
||||||
|
**Report lifecycle generation/reuse:** not re-established by this browser run (zero-model-call budget). Existing lifecycle evidence remains authoritative unless separately retested. The inference that the report is "dynamically generated on page load" is not preserved from the verification record.
|
||||||
|
|
||||||
|
**Next step:** product-level review of the Investigation Report experience — do not automatically change implementation.
|
||||||
|
|
||||||
## Current product architecture
|
## Current product architecture
|
||||||
|
|
||||||
Three distinct routes:
|
Three distinct routes:
|
||||||
|
|||||||
@@ -50,6 +50,26 @@ Application-level isolation proven on deployed instance — User A does not see
|
|||||||
|
|
||||||
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
|
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
|
||||||
|
|
||||||
|
#### Three-Part Report Increment — Loop 4/2 (VERIFIED)
|
||||||
|
|
||||||
|
**Implementation commit:** `56b5f00`
|
||||||
|
**Deterministic apparatus:** PASS (18 tests in `report-epistemic-ab-input-control.test.js`).
|
||||||
|
**Rendered-product verification via Playwright:** VERIFIED.
|
||||||
|
|
||||||
|
One actual Investigation Report demonstrated three distinct epistemic sections:
|
||||||
|
1. What we understand — evidence-backed; unresolved questions excluded from understanding.
|
||||||
|
2. What remains plausible — visibly separate (empty/default state in verified case).
|
||||||
|
3. What remains unresolved — canonical Open Questions rendered as non-evidence, not answered/rank/interpreted.
|
||||||
|
|
||||||
|
- Three-part report semantic/rendering: **VERIFIED** (one observed investigation, one scenario shape).
|
||||||
|
- Unresolved questions remained non-evidence.
|
||||||
|
- Zero additional experimental model calls during verification; no production/test changes.
|
||||||
|
- `Update report` placement/ownership: OPEN PRODUCT QUESTION (separate from the verified increment).
|
||||||
|
- Report lifecycle generation/reuse not re-established by this run.
|
||||||
|
- Evidence does not generalise to universally solved or production-complete.
|
||||||
|
|
||||||
|
**Next:** product-level review of Investigation Report experience.
|
||||||
|
|
||||||
#### Loop 2 (final semantic run — 2/2 valid calls)
|
#### Loop 2 (final semantic run — 2/2 valid calls)
|
||||||
|
|
||||||
Observed:
|
Observed:
|
||||||
|
|||||||
Reference in New Issue
Block a user