feat(confidence-engine): add serializeFocusedInvestigationEvidence evidence serializer and deterministic test

Apparatus-level pure serialization function that preserves all five
focused-deconstruction arrays (observations, uncertainties, assumptions,
relationships, possibleFollowUpQuestions) plus exact inputs and execution
metadata from the existing checkpoint — without modifying production code.

Deterministic verification: syntax check pass, 79/79 assertions pass
(including 28 new evidence-ser tests), zero live calls.

Experiment 4's missing structured output cannot be reconstructed (never
persisted by the apparatus). No new semantic evidence produced.
This commit is contained in:
2026-10-09 14:25:26 +01:00
parent 7fd05745e8
commit 7d7a9e8d29
4 changed files with 205 additions and 2 deletions
+19
View File
@@ -730,3 +730,22 @@ Tested canonical uncertainty description as initial display instead of generated
**Input provenance:** Synthetic answer deliberately crafted for this controlled experiment as FICTIONAL TEST DATA. Not a real user response. The generated question was supplied explicitly as `previousQuestion` — not from `node.openQuestion`. This does not imply that the production UI always supplies `node.openQuestion` as `previousQuestion`.
**Remaining limitations:** Single scenario shape only (recruitment). Does not establish consistency across domains or models. A single run does not prove generalisability or user comprehension. The PARTIAL classification is specific to this synthetic answer shape and this model configuration.
## Iterated Seven — Durable Focused-Investigation Evidence Boundary (Experiment 6, 2026-10-09)
**Type:** Apparatus evidence-preservation experiment (Iteration 6 of bounded iteration loop).
**Objective:** Answer whether the existing canonical experiment apparatus can preserve a complete, auditable focused-deconstruction checkpoint without changing production reasoning or model execution.
**Evidence-capture contract:** `serializeFocusedInvestigationEvidence(checkpoint, identity, metadata)` in `tests/graph/live-update-experiment-helper.cjs` (line ~524). Pure function accepting:
- `checkpoint` — object returned by `composeDeepInvestigationCheckpoint()`
- `identity` — `{ experimentId, scenarioId, targetNodeId, evidenceDesignation }`
- `metadata` — `{ modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? }`
Returns JSON-serializable record with: five deconstruction arrays (`observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`) preserved verbatim; exact inputs (`canonicalTargetDescription`, `actualQuestion`, `exactAnswer`); execution metadata (null when not exposed by apparatus).
**Deterministic verification:** Syntax check PASS. 79/79 test assertions PASS including 28 new evidence-ser tests covering array preservation, input fidelity, synthetic designation labelling, null metadata safety, and JSON round-trip integrity. Zero live LLM calls. Zero browser calls.
**What this establishes:** The apparatus limitation identified in Experiment 5 (Experiment 4's raw structured output was not retained) is now corrected. Future focused-investigation experiments can preserve their deconstruction checkpoint via `serializeFocusedInvestigationEvidence()`.
**What remains:** No new semantic evidence was produced. Experiment 4's missing structured output cannot be reconstructed — it was never persisted by the apparatus at that time. The PARTIAL semantic fidelity classification stands unchanged.
@@ -96,7 +96,8 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **Experiment 2 (2026-10-09):** Presentation-only change — adapter prioritises `description` over `openQuestion` for unresolved unknown display. Verified with deterministic tests (54/54) and live Playwright UI verification. Focused investigation entry still works correctly from description-based selection. `openQuestion` generation remains untouched. Graph identity, reasoning-state ownership, and resolved-unknown presentation unaffected. Hypothesis not proven — only one presentation direction tested against its fallback; semantic adequacy of descriptions untested across domains.
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
## Decision state: OPEN (after iteration 2 evidence).
## Decision state: OPEN (after iteration 6 evidence).
- **Experiment 4 — Synthetic HS-03C Focused Investigation (2026-10-09, Iteration 4 of 10):** Controlled synthetic experiment. Fixed scenario: launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. Target node `n2yzwlv`: "Timeline and feasibility of recruiting two specialists within the nine-month window." Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." The generated question was supplied explicitly as `previousQuestion` via `composeDeepInvestigationCheckpoint()`. Model: `qwen-claude:latest` (Ollama). Exactly one live LLM call. Result: **PARTIAL** semantic fidelity. The model correctly preserved the core uncertainty about whether recruitment can be completed within the nine-month window and correctly distinguished estimates from confirmed timing, possible candidates from secured hires, and expected onboarding from completed onboarding. However, it narrowed the investigation to hiring pipeline specifics ("What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods") rather than preserving the broader feasibility question of whether both specialists can become productive in time for the nine-month launch. The strongest counterevidence: the canonical description "Timeline and feasibility" was reduced to essentially a recruitment logistics investigation, losing the broader cross-area dependency and cascading-delay uncertainty captured in the original HS-03C graph (`unexplainedTransitions` with ID `utrans_dependency`). This is consistent with the HS-03C partial pattern from prior experiments. Input provenance: synthetic answer deliberately crafted for this controlled experiment, not a real user response. Graph constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`).
- **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments.