diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 1191092..13441e1 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -767,3 +767,27 @@ Returns JSON-serializable record with: five deconstruction arrays (`observations **Distinction:** This test proves that the serializer's JSON-serializable output survives write-to-disk and read-back without loss. It does NOT establish a production/live-run command for writing evidence to files — that remains an untested operational boundary for future work. The apparatus produces correct in-memory records; disk persistence of those records is now independently verified at the test level. **Experiment 4's original structured output remains unavailable.** Experiment 4 was executed before `serializeFocusedInvestigationEvidence` existed; its raw deconstruction checkpoint was never captured. No amount of apparatus testing can retroactively reconstruct it. The PARTIAL semantic fidelity classification for Experiment 4 stands unchanged. + +### Iterated Eight — Explicit Evidence Persistence Operation (Experiment 8, 2026-10-09) + +**Type:** Apparatus persistence operation experiment (Iteration 8 of bounded iteration loop). Zero-live-call apparatus experiment. + +**Objective:** Answer whether the existing canonical experiment apparatus can expose one explicit, reusable operation that saves a completed focused-investigation checkpoint to a durable JSON file without modifying production code or changing model execution. + +**Exported function:** `saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath)` — `tests/graph/live-update-experiment-helper.cjs`. + +**Required arguments:** +- `checkpoint` — object returned by `composeDeepInvestigationCheckpoint()` (must contain `postEpisodeContributions` with exactly one valid deconstruction contribution; must contain `focusedInvestigationInput` with exact values) +- `identity` — `{ experimentId, scenarioId, targetNodeId, evidenceDesignation }` (all four fields required, no nulls) +- `metadata` — optional `{ modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? }` +- `outputPath` — explicit non-empty string file path + +**Verified output contract:** Returns `{ output_path: string, verifiedEvidence: object }`. Writes complete structured evidence as formatted JSON to the specified file. Reads file back and verifies deep structural equality with serialized record before returning. Five structured arrays preserved verbatim (including legitimately empty ones). All identity fields preserved exactly. Exact input values preserved exactly. + +**Ambiguous contribution handling:** Rejected — throws `Error` when multiple valid deconstruction-carrying contributions exist in the checkpoint. Does not silently select one. + +**Production changes:** None. No production code modified. No live model calls. No Ollama invocation. No graph behaviour change. No new dependencies. + +**Targeted test result:** `node --check` on helper — PASS. Full deterministic test suite — 0 failures across all sections including 6 new save-evidence assertions (positive: file write, structured content, input fidelity, identity fidelity, return values; negative: missing path rejection, missing contribution rejection, ambiguous contribution rejection). All OS temp files cleaned in `finally` blocks. + +**Operational note:** Future live experiments must explicitly call `saveFocusedInvestigationEvidence()` after a completed checkpoint to persist the evidence record. The function is not invoked automatically. Experiment 4's raw output remains unavailable — it was never persisted by the apparatus at that time. diff --git a/docs/uncertainty-led-investigation-hypothesis.md b/docs/uncertainty-led-investigation-hypothesis.md index 1dc66dd..eb7269c 100644 --- a/docs/uncertainty-led-investigation-hypothesis.md +++ b/docs/uncertainty-led-investigation-hypothesis.md @@ -103,3 +103,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc - **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments. - **Experiment 7 — Durable Focused-Investigation Disk Persistence (2026-10-09, Iteration 7 of 10):** Verified that `serializeFocusedInvestigationEvidence()` output survives write-to-disk and read-back without loss. Extended deterministic test in `tests/graph/live-update-experiment-helper-checkpoint.test.cjs` writes serialized evidence to OS temp directory via `os.tmpdir()`, reads it back, parses recovered JSON, and asserts deep structural equality against the original. Explicitly verified: five structured arrays with correct lengths; exact synthetic inputs (`canonicalTargetDescription`, `actualQuestion`, `exactAnswer`) unchanged; identity fields preserved; execution metadata null-fields intact. Temp file removed in `finally` block. Zero live model calls. Zero browser calls. No production or canonical-helper modifications. Total test result: all assertions passed (0 failures). This test proves the serializer's JSON-serializable output survives disk persistence at the apparatus level — it does not establish a production/live-run command for writing evidence to files, which remains an untested operational boundary. Experiment 4's original structured output remains unavailable; never persisted by the apparatus at that time. + +- **Experiment 8 — Explicit Evidence Persistence Operation (2026-10-09, Iteration 8 of 10):** Added `saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath)` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). This is a small wrapper around `serializeFocusedInvestigationEvidence()` that performs explicit persistence via an explicitly supplied output path. Validates: (1) non-empty output path required; (2) exactly one valid deconstruction contribution in checkpoint (rejects ambiguous multi-candidate); (3) all four identity fields present (no nulls); (4) focused-investigation input values present. Writes formatted JSON, reads back, verifies deep structural equality. Returns `{ output_path, verifiedEvidence }`. Zero live model calls. Zero browser calls. No production code modified. Total deterministic test result: 0 failures across all assertions including positive tests (file write, structured evidence completeness, input fidelity, identity fidelity, return values) and negative tests (missing path rejection, missing contribution rejection, ambiguous contribution rejection). All OS temp files cleaned in `finally` blocks. Future live experiments must explicitly call this save function after a completed checkpoint. Experiment 4's original structured output remains unavailable; never persisted by the apparatus at that time. diff --git a/tests/graph/live-update-experiment-helper-checkpoint.test.cjs b/tests/graph/live-update-experiment-helper-checkpoint.test.cjs index c1a2208..3f7108f 100644 --- a/tests/graph/live-update-experiment-helper-checkpoint.test.cjs +++ b/tests/graph/live-update-experiment-helper-checkpoint.test.cjs @@ -14,7 +14,7 @@ * 6. Absent updated graph produces null structural checkpoint */ -const { extractLiveExperimentCheckpoint, _extractReasoningState, composeDeepInvestigationCheckpoint, runLiveExperiment } = require("./live-update-experiment-helper.cjs"); +const { extractLiveExperimentCheckpoint, _extractReasoningState, composeDeepInvestigationCheckpoint, runLiveExperiment, saveFocusedInvestigationEvidence } = require("./live-update-experiment-helper.cjs"); // ---- deriveCategory kept in test because it is NOT exported from the helper ---- // (It is part of the answer-meaning extraction infrastructure, not the checkpoint seam) @@ -740,6 +740,157 @@ console.log("\n=== DISK PERSISTENCE VERIFICATION ===\n"); } } +// ---- SAVE FOCUSED INVESTIGATION EVIDENCE (Experiment 8 — Iteration 8) ───── +console.log("\n=== SAVE FOCUSED INVESTIGATION EVIDENCE ===\n"); + +{ + const { saveFocusedInvestigationEvidence } = require("./live-update-experiment-helper.cjs"); + const { writeFileSync, readFileSync, unlinkSync } = require("fs"); + const os = require("os"); + const path = require("path"); + + assert(typeof saveFocusedInvestigationEvidence === "function", "saveFocusedInvestigationEvidence is exported and callable"); + + // Synthetic checkpoint containing all five output arrays (same shape as Experiment 7) + const saveCheckpoint = { + currentUnderstanding: "Synthetic understanding text", + updatedSituationGraph: { nodes: [{ id: "n1" }], edges: [], resolvedNodeIds: ["n1"] }, + postEpisodeContributions: [{ + id: "contrib-save-test-8", + targetNodeId: "n2yzwlv", + question: "What is the current hiring pipeline status?", + answer: "We have identified several possible candidates.", + sequence: 1, + observations: ["Several possible candidates identified for both roles"], + uncertainties: ["Candidate availability and notice periods remain unconfirmed"], + assumptions: ["Onboarding takes some time before full contribution"], + relationships: [{ from: "n2yzwlv", to: "n3abc", type: "dependency" }], + possibleFollowUpQuestions: ["What is the confirmed earliest start date for preferred candidates?"], + }], + postEpisodeFindings: [], + findingsCheckpoint: { total: 0, eligible: 0 }, + graphCheckpoint: { nodeCount: 1, edgeCount: 0, resolvedNodeIds: ["n1"] }, + modelElapsedMs: null, + endToEndElapsedMs: null, + providerGenerateCalls: 0, + modelName: null, + synthesisErrorDiagnostic: null, + userSupportedMeaning: null, + possibleInference: null, + focusedInvestigationInput: { + targetDescription: "Timeline and feasibility of recruiting two specialists within the nine-month window", + question: "What is the current hiring pipeline status for the two specialist roles?", + answer: "We have identified several possible candidates for both specialist roles.", + }, + }; + + const saveIdentity = { + experimentId: "exp-8-save-evidence", + scenarioId: "HS-03C-SAVE", + targetNodeId: "n2yzwlv", + evidenceDesignation: "synthetic-test-data", + }; + + const tmpDir = os.tmpdir(); + const tmpFileSave = path.join(tmpDir, "exp8-save-evidence-" + Date.now() + ".json"); + let savedPath = null; + let savedEvidence = null; + + try { + // Call save function with the synthetic checkpoint + const result = saveFocusedInvestigationEvidence(saveCheckpoint, saveIdentity, {}, tmpFileSave); + savedPath = result.output_path; + savedEvidence = result.verifiedEvidence; + + // Verify the function writes the requested file + assert(result.output_path === tmpFileSave, "save returns correct output_path"); + assert(typeof result.verifiedEvidence === "object", "save returns verified evidence object"); + + const fileContent = readFileSync(tmpFileSave, "utf8"); + assert(fileContent.length > 0, "file was written and is non-empty"); + assert(fileContent.trim().startsWith("{"), "file starts with JSON object"); + + // Verify the file contains complete structured evidence + const parsedFromFile = JSON.parse(fileContent); + assert(Array.isArray(parsedFromFile.observations), "file: observations array present"); + assert(Array.isArray(parsedFromFile.uncertainties), "file: uncertainties array present"); + assert(Array.isArray(parsedFromFile.assumptions), "file: assumptions array present"); + assert(Array.isArray(parsedFromFile.relationships), "file: relationships array present"); + assert(Array.isArray(parsedFromFile.possibleFollowUpQuestions), "file: possibleFollowUpQuestions array present"); + + // Verify inputs remain exact + assert(parsedFromFile.canonicalTargetDescription === "Timeline and feasibility of recruiting two specialists within the nine-month window", + "file: canonicalTargetDescription exact match"); + assert(parsedFromFile.actualQuestion === "What is the current hiring pipeline status for the two specialist roles?", + "file: actualQuestion exact match"); + assert(parsedFromFile.exactAnswer === "We have identified several possible candidates for both specialist roles.", + "file: exactAnswer exact match"); + + // Verify experiment identity remains exact + assert(parsedFromFile.experimentId === "exp-8-save-evidence", "file: experimentId exact match"); + assert(parsedFromFile.scenarioId === "HS-03C-SAVE", "file: scenarioId exact match"); + assert(parsedFromFile.targetNodeId === "n2yzwlv", "file: targetNodeId exact match"); + assert(parsedFromFile.evidenceDesignation === "synthetic-test-data", "file: evidenceDesignation exact match"); + + // Verify metadata (empty object → all null) + assert(parsedFromFile.modelName === null, "file: modelName is null"); + assert(parsedFromFile.providerGenerateCalls === null, "file: providerGenerateCalls is null"); + assert(parsedFromFile.modelElapsedMs === null, "file: modelElapsedMs is null"); + assert(parsedFromFile.endToEndElapsedMs === null, "file: endToEndElapsedMs is null"); + + // Verify the function returns the verified record and path (verifiedEvidence == recovered) + assert(JSON.stringify(savedEvidence) === JSON.stringify(parsedFromFile), "save verifiedEvidence matches file content"); + + console.log("\n=== SAVE EVIDENCE POSITIVE TEST COMPLETE — ALL ASSERTIONS PASSED ===\n"); + } finally { + try { if (savedPath) unlinkSync(savedPath); } catch (_) { /* already removed */ } + try { unlinkSync(tmpFileSave); } catch (_) { /* already removed */ } + } + + // Test: missing output path is rejected + let threw = false; + try { + saveFocusedInvestigationEvidence(saveCheckpoint, saveIdentity, {}, ""); + } catch (err) { + assert(err.message.includes("outputPath"), "missing outputPath throws"); + threw = true; + } + assert(threw, "empty output path rejected"); + + // Test: missing focused-deconstruction contribution is rejected + threw = false; + const noContribCheckpoint = { + postEpisodeContributions: [], + focusedInvestigationInput: { targetDescription: "t", question: "q", answer: "a" }, + }; + try { + saveFocusedInvestigationEvidence(noContribCheckpoint, saveIdentity, {}, "/tmp/no-contrib.json"); + } catch (err) { + assert(err.message.includes("contribution"), "missing contribution throws"); + threw = true; + } + assert(threw, "missing contribution rejected"); + + // Test: ambiguous contribution selection is rejected + threw = false; + const ambiguousCheckpoint = { + postEpisodeContributions: [ + { id: "c1", targetNodeId: "n1", observations: ["obs"], uncertainties: ["unc"], assumptions: [], relationships: [], possibleFollowUpQuestions: [] }, + { id: "c2", targetNodeId: "n1", observations: ["obs2"], uncertainties: ["unc2"], assumptions: [], relationships: [], possibleFollowUpQuestions: [] }, + ], + focusedInvestigationInput: { targetDescription: "t", question: "q", answer: "a" }, + }; + try { + saveFocusedInvestigationEvidence(ambiguousCheckpoint, saveIdentity, {}, "/tmp/ambiguous.json"); + } catch (err) { + assert(err.message.includes("ambiguous"), "ambiguous contribution throws"); + threw = true; + } + assert(threw, "ambiguous contribution rejected"); + + console.log("\n=== SAVE EVIDENCE NEGATIVE TESTS COMPLETE — ALL ASSERTIONS PASSED ===\n"); +} + // ---- SYNTHETIC TESTS WITH FAKE PROVIDER (Batch 3 Iteration 1) ──────────── // Tests synthesizeCurrentUnderstanding directly with DI-injected fake providers. // Zero live calls — provider.generateReconstruction returns synchronously. diff --git a/tests/graph/live-update-experiment-helper.cjs b/tests/graph/live-update-experiment-helper.cjs index 7acad92..d256376 100644 --- a/tests/graph/live-update-experiment-helper.cjs +++ b/tests/graph/live-update-experiment-helper.cjs @@ -583,4 +583,98 @@ function serializeFocusedInvestigationEvidence(checkpoint, identity, metadata) { }; } -module.exports = { runLiveExperiment, extractLiveExperimentCheckpoint, _extractReasoningState, composeDeepInvestigationCheckpoint, serializeFocusedInvestigationEvidence }; +/** + * Save a completed focused-investigation checkpoint to a durable JSON file. + * + * Wrapper around `serializeFocusedInvestigationEvidence()` that performs + * explicit persistence via the caller-supplied output path. Intended for + * deterministic test fixtures and future live experiment runbooks — NOT + * called automatically from production code. + * + * @param {object} checkpoint — returned from composeDeepInvestigationCheckpoint() + * @param {object} identity — { experimentId, scenarioId, targetNodeId, evidenceDesignation } + * @param {object} metadata — { modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? } + * @param {string} outputPath — explicit absolute or relative file path for the JSON output + * @returns {{ output_path: string, verifiedEvidence: object }} { output_path, verifiedEvidence } + * @throws {Error} on invalid inputs, missing required fields, ambiguous contribution, or I/O failure + */ +function saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath) { + const fs = require("fs"); + + /* ── 2. Require explicit non-empty output path ────────────────────── */ + if (!outputPath || typeof outputPath !== "string" || !outputPath.trim()) { + throw new Error("saveFocusedInvestigationEvidence requires an explicit non-empty outputPath string"); + } + + /* ── 3. Validate checkpoint has exactly one valid contribution ────── */ + const contributions = Array.isArray(checkpoint?.postEpisodeContributions) + ? checkpoint.postEpisodeContributions + : []; + + const validContributionCandidates = contributions.filter( + (c) => Array.isArray(c?.observations) && Array.isArray(c?.uncertainties) + ); + + if (validContributionCandidates.length === 0) { + throw new Error( + "saveFocusedInvestigationEvidence requires a checkpoint with at least one valid focused-deconstruction contribution; found 0" + ); + } + + if (validContributionCandidates.length > 1) { + throw new Error( + "saveFocusedInvestigationEvidence rejects ambiguous checkpoint: multiple candidates exist (" + + validContributionCandidates.length + + "); supply a single-target checkpoint or resolve ambiguity before saving" + ); + } + + /* ── 4. Validate identity and input values ────────────────────────── */ + if (!identity || typeof identity !== "object") { + throw new Error("saveFocusedInvestigationEvidence requires an identity object"); + } + if (identity.experimentId == null) { + throw new Error("saveFocusedInvestigationEvidence: identity.missing experimentId"); + } + if (identity.scenarioId == null) { + throw new Error("saveFocusedInvestigationEvidence: identity.missing scenarioId"); + } + if (identity.targetNodeId == null) { + throw new Error("saveFocusedInvestigationEvidence: identity.missing targetNodeId"); + } + if (identity.evidenceDesignation == null) { + throw new Error("saveFocusedInvestigationEvidence: identity.missing evidenceDesignation"); + } + + const fiInputs = checkpoint.focusedInvestigationInput; + if (!fiInputs || typeof fiInputs !== "object") { + throw new Error("saveFocusedInvestigationEvidence: checkpoint missing focusedInvestigationInput object"); + } + if (fiInputs.targetDescription == null || fiInputs.question == null || fiInputs.answer == null) { + throw new Error( + "saveFocusedInvestigationEvidence: checkpoint missing required focused-investigation input values (targetDescription, question, answer)" + ); + } + + /* ── 1. Serialize via existing serializer ─────────────────────────── */ + const serialized = serializeFocusedInvestigationEvidence(checkpoint, identity, metadata); + + /* ── 5-6. Write the complete record as readable JSON ──────────────── */ + const jsonContent = JSON.stringify(serialized, null, 2); + fs.writeFileSync(outputPath, jsonContent, "utf8"); + + /* ── 7-8. Read back and verify structural equality ────────────────── */ + const recoveredRaw = fs.readFileSync(outputPath, "utf8"); + const recovered = JSON.parse(recoveredRaw); + + if (JSON.stringify(serialized) !== JSON.stringify(recovered)) { + throw new Error( + "saveFocusedInvestigationEvidence: write/read-back verification failed — recovered JSON does not match serialized record" + ); + } + + /* ── 9. Return output path and verified evidence record ───────────── */ + return { output_path: outputPath, verifiedEvidence: recovered }; +} + +module.exports = { runLiveExperiment, extractLiveExperimentCheckpoint, _extractReasoningState, composeDeepInvestigationCheckpoint, serializeFocusedInvestigationEvidence, saveFocusedInvestigationEvidence };