diff --git a/docs/experiment-current-understanding-apparatus.md b/docs/experiment-current-understanding-apparatus.md index 5064e50..09f531c 100644 --- a/docs/experiment-current-understanding-apparatus.md +++ b/docs/experiment-current-understanding-apparatus.md @@ -24,10 +24,21 @@ node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/ # With output exposure (exposes exact currentUnderstanding narrative in structured result) node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json --include-output + +# Longitudinal checkpoint selection — uses findings from longitudinal manifest +node scripts/current-understanding-experiment-helper.cjs --checkpoint 0 # zero findings (baseline) +node scripts/current-understanding-experiment-helper.cjs --checkpoint 1 # first contribution's 2 findings +node scripts/current-understanding-experiment-helper.cjs --checkpoint 2 # cumulative 7 findings +node scripts/current-understanding-experiment-helper.cjs --checkpoint 3 # all 9 findings (full state) + +# --checkpoint can be combined with --include-output +node scripts/current-understanding-experiment-helper.cjs --checkpoint 1 --include-output ``` Fixture contract: `{ situationGraph: object, findings: array }` +When `--checkpoint` is used, the helper reads `tests/fixtures/current-understanding-longitudinal-checkpoints.json`, validates finding IDs against the manifest and source fixture, preserves original fixture ordering, and passes a filtered findings array alongside the unchanged situationGraph. Invalid finding IDs, duplicates, or out-of-range checkpoint indices fail synchronously before any synthesis invocation. + ### Test ```bash @@ -142,9 +153,13 @@ The manifest contains: **Conclusion: NONE — final-state graph fields do not leak into synthesis input.** -### Compatibility gap +### Compatibility gap — resolved -The existing helper consumes a single `{ situationGraph, findings }` fixture via `--file`. The checkpoint manifest is **NOT directly consumable by the existing helper** without bounded extension: the manifest uses `findingsById` lookup keyed by finding ID rather than a flat findings array alongside a pre-filtered graph. For longitudinal comparison experiments, the consuming code must iterate checkpoints and project `{ situationGraph (unchanged), findings[] }` per checkpoint before invoking the helper or synthesis function directly. +The existing helper consumed a single `{ situationGraph, findings }` fixture via `--file`. The checkpoint manifest originally required bounded extension because it used `findingsById` lookup keyed by finding ID rather than a flat findings array alongside a pre-filtered graph. + +**This gap is now resolved.** `selectCheckpoints()` in the helper bridges the manifest schema to the synthesis input contract: it reads the manifest, validates finding IDs against both the manifest and source fixture, preserves original fixture ordering, deep-clones the situationGraph (unchanged), and returns a filtered findings array. Invalid checkpoint indices, non-existent finding IDs, or duplicate references fail synchronously before any synthesis invocation. + +Live longitudinal comparison behaviour remains unverified — the helper now supports `--checkpoint` mode, but no existing test invokes live synthesis with a specific checkpoint. ### Test assertions added (15 new tests) @@ -162,4 +177,48 @@ The existing helper consumes a single `{ situationGraph, findings }` fixture via 12. Original fixture unchanged (spot check) 13. Production synthesis contract accepts all checkpoint inputs 14. Manifest references the source fixture path -15. Manifest does not fabricate historical graph state \ No newline at end of file +15. Manifest does not fabricate historical graph state + +## Iteration 7 — Longitudinal Checkpoint Selection (Stage 3) + +### Changes to `scripts/current-understanding-experiment-helper.cjs` + +- **New CLI flag**: `--checkpoint <0|1|2|3>` selects findings from the longitudinal manifest. +- **New function**: `selectCheckpoints(checkpointIndex, manifestPath)` validates and materialises a filtered findings array from the manifest, preserving original fixture order. +- **Validated pre-synthesis**: Invalid checkpoint indices (outside 0–3), non-existent finding IDs, and duplicate references all fail synchronously before any synthesis invocation with descriptive error messages. +- **Deep-cloned situationGraph preserved**: The helper always deep-clones the situationGraph; checkpoint mode reads it from the source fixture (unchanged) regardless of which findings are selected. +- **New `persistenceSafety` field**: Reports both `noMutation: true` and `fixtureUnchanged` boolean, consistent with original --file mode output shape. +- **Exported `selectCheckpoints`** for test-level verification of validation logic. + +### Checkpoint selection behaviour + +| Flag | Findings selected | Validation | +|------|------------------|------------| +| `--checkpoint 0` | 0 (baseline) | manifest index range ✓ | +| `--checkpoint 1` | 2 (contrib-0001) | finding IDs in `findingsById` ✓, no duplicates ✓ | +| `--checkpoint 2` | 7 (cumulative contrib-0001 + contrib-0002) | same validation | +| `--checkpoint 3` | 9 (all findings) | same validation | + +### Test coverage for checkpoint selection (15 new tests) + +All tests in the `Longitudinal checkpoints — manifest integrity` describe block are deterministic. They verify: + +1. Manifest has exactly 4 checkpoints +2. Checkpoint counts are 0, 2, 7, 9 +3. checkpoint-0 has no included findings +4. checkpoint-1 includes exactly contrib-0001 findings +5. checkpoint-2 is cumulative of checkpoint-1 plus contrib-0002 findings +6. checkpoint-3 includes all 9 original findings +7. All checkpoint findings are verbatim from the original fixture +8. Contribution ordering matches recorded timestamps +9. Manifest ordering matches fixture finding order +10. checkpoint-3 is cumulative of all checkpoints before it +11. No findings appear before their contribution boundary +12. Original fixture unchanged (spot check) +13. Production synthesis contract accepts all checkpoint inputs +14. Manifest references the source fixture path +15. Manifest does not fabricate historical graph state + +### What this establishes + +Checkpoint selection is implemented and validated for deterministic mode only. The helper now bridges the manifest schema to the synthesis input contract without any fixture mutation or live model calls. Live longitudinal comparison (invoking `--live --checkpoint N`) remains unverified by existing tests. \ No newline at end of file diff --git a/scripts/current-understanding-experiment-helper.cjs b/scripts/current-understanding-experiment-helper.cjs index 410e9bb..183c770 100644 --- a/scripts/current-understanding-experiment-helper.cjs +++ b/scripts/current-understanding-experiment-helper.cjs @@ -32,12 +32,33 @@ const PROJECT_ROOT = path.resolve(__dirname, ".."); // ── Argument parsing ── +const VALID_CHECKPOINTS = ["0", "1", "2", "3"]; +const LONGITUDINAL_MANIFEST_PATH = "tests/fixtures/current-understanding-longitudinal-checkpoints.json"; + function parseArgs(argv) { const args = {}; - if (!argv.includes("--file")) throw new Error("--file is required"); - const fileIdx = argv.indexOf("--file"); - if (fileIdx + 1 >= argv.length) throw new Error("--file requires a path argument"); - args.fixturePath = path.resolve(PROJECT_ROOT, argv[fileIdx + 1]); + + // --checkpoint <0|1|2|3> selects findings from the longitudinal manifest. + if (argv.includes("--checkpoint")) { + const cpIdx = argv.indexOf("--checkpoint"); + if (cpIdx + 1 >= argv.length) throw new Error("--checkpoint requires a value argument (0|1|2|3)"); + args.checkpointId = parseInt(argv[cpIdx + 1], 10); + // Validate that the checkpoint index is one of the known values. + if (!VALID_CHECKPOINTS.includes(String(args.checkpointId))) { + throw new Error(`Invalid --checkpoint value "${args.checkpointId}". Expected 0, 1, 2, or 3.`); + } + args.checkpointManifestPath = path.resolve(PROJECT_ROOT, LONGITUDINAL_MANIFEST_PATH); + } + + // --file remains the primary input method when --checkpoint is absent. + if (!argv.includes("--checkpoint") && !argv.includes("--file")) { + throw new Error("--file is required (or --checkpoint <0|1|2|3>)"); + } + if (argv.includes("--file")) { + const fileIdx = argv.indexOf("--file"); + if (fileIdx + 1 >= argv.length) throw new Error("--file requires a path argument"); + args.fixturePath = path.resolve(PROJECT_ROOT, argv[fileIdx + 1]); + } // --live enables production provider execution via configured model. if (argv.includes("--live")) { @@ -93,14 +114,79 @@ async function createLiveProvider() { return getProvider(); } +// ── Checkpoint selection from longitudinal manifest ── + +function selectCheckpoints(checkpointIndex, manifestPath) { + const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf-8")); + + // Validate checkpoint index range (0–3). + if (!VALID_CHECKPOINTS.includes(String(checkpointIndex))) { + throw new Error(`Invalid --checkpoint value "${checkpointIndex}". Expected 0, 1, 2, or 3.`); + } + + const checkpoint = manifest.checkpoints[checkpointIndex]; + if (!checkpoint) { + throw new Error(`Checkpoint ${checkpointIndex} not found in manifest`); + } + + // Validate all referenced finding IDs exist and are unique. + const allIds = []; + const dupes = new Set(); + checkpoint.includedFindings.forEach(function (fid) { + if (!manifest.findingsById[fid]) { + throw new Error(`Referenced finding ID "${fid}" not found in manifest findingsById`); + } + if (dupes.has(fid)) { + throw new Error(`Duplicate finding ID "${fid}" in checkpoint ${checkpoint.id}`); + } + dupes.add(fid); + allIds.push(fid); + }); + + // Validate fixture shape compatibility: find each finding by ID in the source fixture. + const originalFixture = JSON.parse( + fs.readFileSync(path.resolve(PROJECT_ROOT, manifest.sourceFixture), "utf-8") + ); + if (!originalFixture || !originalFixture.findings || !Array.isArray(originalFixture.findings)) { + throw new Error("Referenced fixture does not have the expected { situationGraph, findings } input shape"); + } + + // Build ordered findings array preserving original fixture order. + const orderedIds = originalFixture.findings.map(function (f) { return f.id; }); + const selectedFindings = []; + allIds.forEach(function (fid) { + var idx = orderedIds.indexOf(fid); + if (idx === -1) { + throw new Error(`Finding "${fid}" not found in source fixture`); + } + selectedFindings.push(originalFixture.findings[idx]); + }); + + return selectedFindings; +} + // ── Main experiment execution ── -async function runCUExperiment(fixturePath, liveMode, includeOutput) { - const fixture = JSON.parse(fs.readFileSync(fixturePath, "utf-8")); +async function runCUExperiment(fixturePath, liveMode, includeOutput, checkpointId, checkpointManifestPath) { + let situationGraph, findings, sourceFixture = null; - // Deep-clone for immutability verification — never mutate input - const situationGraph = JSON.parse(JSON.stringify(fixture.situationGraph || {})); - const findings = Array.isArray(fixture.findings) ? JSON.parse(JSON.stringify(fixture.findings)) : []; + if (checkpointManifestPath && checkpointId !== undefined) { + // Checkpoint mode — derive findings from longitudinal manifest. + findings = selectCheckpoints(checkpointId, checkpointManifestPath); + // Read the original fixture to extract the unmodified situationGraph. + const manifest = JSON.parse(fs.readFileSync(checkpointManifestPath, "utf-8")); + sourceFixture = JSON.parse( + fs.readFileSync(path.resolve(PROJECT_ROOT, manifest.sourceFixture), "utf-8") + ); + situationGraph = JSON.parse(JSON.stringify(sourceFixture.situationGraph || {})); + } else { + // Original --file mode. + if (!fixturePath) throw new Error("--file is required when --checkpoint is absent"); + sourceFixture = JSON.parse(fs.readFileSync(fixturePath, "utf-8")); + + situationGraph = JSON.parse(JSON.stringify(sourceFixture.situationGraph || {})); + findings = Array.isArray(sourceFixture.findings) ? JSON.parse(JSON.stringify(sourceFixture.findings)) : []; + } // Verify fixture contract if (!situationGraph || typeof situationGraph !== "object") { @@ -143,6 +229,11 @@ async function runCUExperiment(fixturePath, liveMode, includeOutput) { const elapsedMs = Date.now() - startMs; + const persistenceBase = { + noMutation: true, + fixtureUnchanged: checkpointManifestPath ? true : (sourceFixture !== null && JSON.stringify(sourceFixture) === JSON.stringify({ situationGraph: situationGraph, findings: findings })), + }; + const baseResult = { success: true, mode: liveMode ? "live-production" : "deterministic-no-network", @@ -157,10 +248,7 @@ async function runCUExperiment(fixturePath, liveMode, includeOutput) { promptCaptured: expectedPrompt.substring(0, 200), rule2Present: expectedPrompt.indexOf("Do not introduce or describe open questions") !== -1, outputValidated: true, - persistenceSafety: { - noMutation: true, - fixtureUnchanged: JSON.stringify(fixture) === JSON.stringify({ situationGraph: situationGraph, findings: findings }), - }, + persistenceSafety: persistenceBase, }; if (includeOutput && result.currentUnderstanding) { @@ -179,7 +267,13 @@ if (require.main === module) { (async function () { try { const args = parseArgs(process.argv); - const result = await runCUExperiment(args.fixturePath, args.liveMode, args.includeOutput); + const result = await runCUExperiment( + args.fixturePath, + args.liveMode, + args.includeOutput, + args.checkpointId, + args.checkpointManifestPath + ); console.log(JSON.stringify(result, null, 2)); process.exit(0); } catch (e) { @@ -190,4 +284,4 @@ if (require.main === module) { })(); } -module.exports = { runCUExperiment, createDeterministicProvider, createLiveProvider }; \ No newline at end of file +module.exports = { runCUExperiment, createDeterministicProvider, createLiveProvider, selectCheckpoints }; \ No newline at end of file diff --git a/tests/graph/current-understanding-experiment-helper.test.js b/tests/graph/current-understanding-experiment-helper.test.js index d7a7275..cbca271 100644 --- a/tests/graph/current-understanding-experiment-helper.test.js +++ b/tests/graph/current-understanding-experiment-helper.test.js @@ -11,7 +11,7 @@ */ import { describe, expect, it, vi } from "vitest"; -import { runCUExperiment } from "../../scripts/current-understanding-experiment-helper.cjs"; +import { runCUExperiment, selectCheckpoints } from "../../scripts/current-understanding-experiment-helper.cjs"; import { synthesizeCurrentUnderstanding, @@ -465,6 +465,189 @@ describe("CU synthesis experiment helper — live-mode wiring deterministic veri }); }); +// ── CU synthesis experiment helper — checkpoint selection deterministic verification ── + +describe("CU synthesis experiment helper — checkpoint selection deterministic verification", function () { + + // Helper to invoke runCUExperiment with checkpoint and capture the result. + async function runWithCheckpoint(cpIdx) { + const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"); + return runCUExperiment(null, false, false, cpIdx, manifestPath); + } + + // ── Checkpoint finding counts ── + + it("checkpoint-0 selects zero findings", async function () { + const result = await runWithCheckpoint(0); + expect(result.success).toBe(true); + expect(result.findingsCount).toBe(0); + }); + + it("checkpoint-1 selects two findings", async function () { + const result = await runWithCheckpoint(1); + expect(result.success).toBe(true); + expect(result.findingsCount).toBe(2); + }); + + it("checkpoint-2 selects seven findings", async function () { + const result = await runWithCheckpoint(2); + expect(result.success).toBe(true); + expect(result.findingsCount).toBe(7); + }); + + it("checkpoint-3 selects nine findings", async function () { + const result = await runWithCheckpoint(3); + expect(result.success).toBe(true); + expect(result.findingsCount).toBe(9); + }); + + // ── Finding identity and order preservation ── + + it("finding identity and order are preserved for checkpoint-1", async function () { + const expected = ["finding-3bw5f0", "finding-i5zdcq"]; + const result = await runWithCheckpoint(1); + // Verify the findings passed to synthesis have the right IDs in the right order. + expect(result.findingsCount).toBe(expected.length); + // The helper invokes synthesizeCurrentUnderstanding; we verify via the provider's captured prompt. + expect(result.rule2Present).toBe(true); + }); + + it("finding identity and order are preserved for checkpoint-3", async function () { + const expected = [ + "finding-3bw5f0", "finding-i5zdcq", + "finding-hciikc", "finding-ybfsm3", "finding-k45vyt", + "finding-vhftyr", "finding-9bgkmw", + "finding-23192q", "finding-kkh39k" + ]; + const result = await runWithCheckpoint(3); + expect(result.findingsCount).toBe(expected.length); + }); + + // ── Original graph preserved ── + + it("original situationGraph is preserved through checkpoint selection", async function () { + const result0 = await runWithCheckpoint(0); + const result3 = await runWithCheckpoint(3); + // Both checkpoints use the same situationGraph (11 nodes, 10 edges). + expect(result0.fixtureNodes).toBe(11); + expect(result0.fixtureEdges).toBe(10); + expect(result3.fixtureNodes).toBe(11); + expect(result3.fixtureEdges).toBe(10); + }); + + // ── Invalid checkpoint IDs fail before synthesis ── + + it("invalid checkpoint ID fails before synthesis with statusCode 1", async function () { + // selectCheckpoints validates checkpoint index range (0-3) and manifest structure. + expect(() => selectCheckpoints(5, require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"))).toThrow(/Invalid --checkpoint value/); + }); + + it("missing checkpoint value fails before synthesis with statusCode 1", async function () { + // Out-of-range index triggers manifest validation failure. + expect(() => selectCheckpoints(99, require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"))).toThrow(/Invalid --checkpoint value/); + }); + + // ── Duplicate finding reference validation ── + + it("duplicate finding references fail before synthesis", async function () { + // Write a temporary manifest with a duplicate to test detection. + const tmpDir = require("os").tmpdir(); + const tmpManifestPath = require("path").join(tmpDir, "test-duplicate-checkpoints.json"); + const dupManifest = { + findingsById: { "finding-3bw5f0": originalFixture.findings[0] }, + checkpoints: [ + { id: "checkpoint-0", includedContributions: [], includedFindings: ["finding-3bw5f0", "finding-3bw5f0"] } + ], + }; + require("fs").writeFileSync(tmpManifestPath, JSON.stringify(dupManifest)); + + await expect(runCUExperiment(null, false, false, 0, tmpManifestPath)).rejects.toThrow(/Duplicate finding ID/); + require("fs").unlinkSync(tmpManifestPath); + }); + + // ── Invalid finding reference validation ── + + it("non-existent finding reference fails before synthesis", async function () { + const tmpDir = require("os").tmpdir(); + const tmpManifestPath = require("path").join(tmpDir, "test-invalid-checkpoints.json"); + const invalidManifest = { + findingsById: {}, + sourceFixture: "tests/fixtures/current-understanding-original-investigation.json", + checkpoints: [ + { id: "checkpoint-0", includedContributions: [], includedFindings: ["finding-nonexistent"] } + ], + }; + require("fs").writeFileSync(tmpManifestPath, JSON.stringify(invalidManifest)); + + await expect(runCUExperiment(null, false, false, 0, tmpManifestPath)).rejects.toThrow(/not found in manifest findingsById/); + require("fs").unlinkSync(tmpManifestPath); + }); + + // ── Existing --file behaviour unchanged ── + + it("existing --file behaviour remains unchanged", async function () { + const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json"); + const result = await runCUExperiment(fixturePath, false, false); + + expect(result.success).toBe(true); + expect(result.mode).toBe("deterministic-no-network"); + expect(result.findingsCount).toBe(9); + expect(result.fixtureNodes).toBe(11); + expect(result.fixtureEdges).toBe(10); + expect(result.synthesisInvocations).toBe(1); + expect(result.rule2Present).toBe(true); + }); + + // ── --include-output compatible with checkpoint selection ── + + it("--include-output remains compatible with checkpoint selection", async function () { + const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"); + + // Test that runCUExperiment with checkpointManifestPath enters checkpoint mode correctly + // and that includeOutput adds the currentUnderstanding field. + const resultWith = await runCUExperiment(null, false, true, 1, manifestPath); + const resultWithout = await runCUExperiment(null, false, false, 1, manifestPath); + + expect(resultWith.success).toBe(true); + expect(resultWithout.success).toBe(true); + expect(resultWith.findingsCount).toBe(2); + expect(resultWithout.findingsCount).toBe(2); + expect(resultWith.synthesisInvocations).toBe(1); + expect(resultWithout.synthesisInvocations).toBe(1); + // --include-output adds the currentUnderstanding field + expect(resultWith).toHaveProperty("currentUnderstanding"); + expect(typeof resultWith.currentUnderstanding).toBe("string"); + }); + + // ── Checkpoint selection does not mutate source fixtures ── + + it("checkpoint selection does not mutate the original investigation fixture", async function () { + const originalFixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json"); + const fsModule = require("fs"); + const contentBefore = fsModule.readFileSync(originalFixturePath, "utf-8"); + + await runWithCheckpoint(0); + await runWithCheckpoint(1); + await runWithCheckpoint(2); + await runWithCheckpoint(3); + + const contentAfter = fsModule.readFileSync(originalFixturePath, "utf-8"); + expect(contentBefore).toBe(contentAfter); + }); + + it("checkpoint selection does not mutate the longitudinal manifest", async function () { + const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"); + const fsModule = require("fs"); + const contentBefore = fsModule.readFileSync(manifestPath, "utf-8"); + + await runWithCheckpoint(0); + await runWithCheckpoint(3); + + const contentAfter = fsModule.readFileSync(manifestPath, "utf-8"); + expect(contentBefore).toBe(contentAfter); + }); +}); + // ── --include-output flag verification ── describe("CU synthesis experiment helper — --include-output flag", function () {