robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
57c88b713a179697a8b869c52da49ea0d75435c0
confidence-engine/docs/experiment-current-understanding-apparatus.md
T

7.7 KiB
Raw Blame History

CU Synthesis Experiment Apparatus — Iteration 5

Purpose

Deterministic experiment helper for invoking the production synthesizeCurrentUnderstanding() function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.

Files

File Role
scripts/current-understanding-experiment-helper.cjs Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes
tests/graph/current-understanding-experiment-helper.test.js Deterministic verification (34 tests, 0 live calls)

Usage

Helper (CLI)

# Deterministic mode (default — zero live model calls)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json

# Live production mode (uses configured OLLAMA_MODEL from .env.local)
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json

# With output exposure (exposes exact currentUnderstanding narrative in structured result)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json --include-output

Fixture contract: { situationGraph: object, findings: array }

Test

npx vitest run tests/graph/current-understanding-experiment-helper.test.js --environment=node

Existing Mechanics Reused

  • Dynamic import() seam from CJS to ESM (proven in scripts/overview-experiment-helper.cjs line 64)
  • Environment loading via @next/env loadEnvConfig (proven in both existing helpers)
  • .env.local ownership of OLLAMA_BASE_URL and OLLAMA_MODEL preserved — helper reads from process.env, never writes
  • Deterministic inline test double provider pattern (mirrors makeFakeProvider in tests/graph/current-understanding-synthesis.test.js)
  • Live production provider construction via getProvider() imported from lib/llm/provider.js (canonical seam, first extraction at line 87-90 of experiment helper)

Production Entry Point

synthesizeCurrentUnderstanding() from lib/graph/current-understanding-synthesis.js

Input contract: { situationGraph: object, findings: array } Optional deps: { provider: object, modelName: string }

Output: { currentUnderstanding: string }

Verification Results

All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:

  • Fixed compatible input reaches synthesizeCurrentUnderstanding()
  • Expected structured output { currentUnderstanding } captured
  • Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
  • Exactly one synthesis invocation per run
  • Input fixture is not mutated (deep clone + compare)
  • No persistence/Supabase/DB operations
  • Live-mode wiring verified via deterministic injected mock provider
  • Deterministic mode behaviour preserved unchanged

Iteration 5 — Live Provider Seam

New seam: createLiveProvider() in scripts/current-understanding-experiment-helper.cjs (line 87-90)

Reuses the canonical provider construction from lib/llm/provider.js:

const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
return getProvider();

New CLI flag: --live

When --live is specified alongside --file, the helper uses the production provider instead of the test double. Requires OLLAMA_BASE_URL and OLLAMA_MODEL from .env.local.

Test coverage for live-mode (10 new tests)

All 10 tests use a deterministic injected mock provider (no network). They verify:

  1. Fixture passes through unchanged to synthesis function
  2. Provider seam invoked exactly once
  3. Evidence projection passed as prompt with correct headers
  4. Current Understanding text captured correctly
  5. Schema matches synthesisOutputSchema
  6. No persistence operations
  7. Deterministic mode still works (regression)
  8. Rule 2 boundary text present
  9. Zero I/O in tests (mock is pure function)
  10. Full apparatus output shape validated for live-mode path

Iteration 4 — Regression Fixture

Fixture path: tests/fixtures/current-understanding-original-investigation.json

Source: authenticated GET /api/investigations/6b080d97-033e-477b-9d41-7b2f4f04bbed (2026-10-08)

Contract: { situationGraph: object, findings: array }

Contents:

  • situationGraph: 11 nodes (1 state, 5 observation, 2 metric, 3 unknown), 10 edges, full reasoningState, resolvedNodeIds, centralStatement, activeUnknownNodeId
  • findings: 9 findings total; all 7 previously verified IDs present (hciikc, ybfsm3, k45vyt, vhftyr, 9bgkmw, 23192q, kkh39k)

Verified: helper parses without invoking live synthesis; fixture immutability confirmed via deep-clone roundtrip.

Iteration 6 — Longitudinal Checkpoints (Stage 2)

New files

File Role
tests/fixtures/current-understanding-longitudinal-checkpoints.json Compact checkpoint manifest referencing original fixture; lists included contribution IDs and finding IDs per checkpoint
tests/graph/current-understanding-experiment-helper.test.js (extended) 15 new longitudinal assertions in Longitudinal checkpoints — manifest integrity describe block

Checkpoint manifest structure

The manifest contains:

  • manifestVersion, description, sourceFixture, contaminationAssessment
  • contributions[] — ordered list with timestamps and ordering integers
  • findingsById — verbatim copy of each original finding (by ID)
  • checkpoints[] — four cumulative states: { id, includedContributions[], includedFindings[] }

Checkpoint state summary

Checkpoint Description Contribs Findings Count
checkpoint-0 Baseline (no findings) none none 0
checkpoint-1 contrib-0001 contrib-0001 3bw5f0, i5zdcq 2
checkpoint-2 contrib-0001 + contrib-0002 contrib-0001, contrib-0002 7 findings 7
checkpoint-3 All contributions all three all 9 original findings 9

Graph-state contamination analysis

synthesizeCurrentUnderstanding() and buildGraphEvidenceProjection() project only:

  • Nodes with status: "known" or "supported" (filtered)
  • centralStatement (framing only)
  • Findings passed as separate parameter

Explicitly excluded from prompt: resolvedNodeIds, activeUnknownNodeId, reasoningState, edges, currentSummary, nodes outside known/supported statuses.

Conclusion: NONE — final-state graph fields do not leak into synthesis input.

Compatibility gap

The existing helper consumes a single { situationGraph, findings } fixture via --file. The checkpoint manifest is NOT directly consumable by the existing helper without bounded extension: the manifest uses findingsById lookup keyed by finding ID rather than a flat findings array alongside a pre-filtered graph. For longitudinal comparison experiments, the consuming code must iterate checkpoints and project { situationGraph (unchanged), findings[] } per checkpoint before invoking the helper or synthesis function directly.

Test assertions added (15 new tests)

  1. Manifest has exactly 4 checkpoints
  2. Checkpoint counts are 0, 2, 7, 9
  3. checkpoint-0 has no included findings
  4. checkpoint-1 includes exactly contrib-0001 findings
  5. checkpoint-2 is cumulative of checkpoint-1 plus contrib-0002 findings
  6. checkpoint-3 includes all 9 original findings
  7. All checkpoint findings are verbatim from the original fixture
  8. Contribution ordering matches recorded timestamps
  9. Manifest ordering matches fixture finding order
  10. checkpoint-3 is cumulative of all checkpoints before it
  11. No findings appear before their contribution boundary
  12. Original fixture unchanged (spot check)
  13. Production synthesis contract accepts all checkpoint inputs
  14. Manifest references the source fixture path
  15. Manifest does not fabricate historical graph state
Reference in New Issue View Git Blame Copy Permalink