Files

T

robbondandClaude Code ce7df91a42 test(confidence-engine): enable live current understanding experiment execution

Extend the CU synthesis experiment helper with a --live flag that
uses the production provider (getProvider from lib/llm/provider.js)
for live model invocation. Preserves deterministic mode unchanged.

Adds 10 new deterministic tests verifying live-mode wiring via
injected mock provider: fixture pass-through, single invocation,
schema correctness, immutability, persistence safety, and regression
of existing deterministic behaviour. Zero network calls in all tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>

2026-10-08 09:59:03 +01:00

4.4 KiB

Raw Blame History

CU Synthesis Experiment Apparatus — Iteration 5

Purpose

Deterministic experiment helper for invoking the production synthesizeCurrentUnderstanding() function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.

Files

File	Role
`scripts/current-understanding-experiment-helper.cjs`	Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes
`tests/graph/current-understanding-experiment-helper.test.js`	Deterministic verification (24 tests, 0 live calls)

Usage

Helper (CLI)

# Deterministic mode (default — zero live model calls)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json

# Live production mode (uses configured OLLAMA_MODEL from .env.local)
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json

Fixture contract: { situationGraph: object, findings: array }

Test

npx vitest run tests/graph/current-understanding-experiment-helper.test.js --environment=node

Existing Mechanics Reused

Dynamic import() seam from CJS to ESM (proven in scripts/overview-experiment-helper.cjs line 64)
Environment loading via @next/env loadEnvConfig (proven in both existing helpers)
.env.local ownership of OLLAMA_BASE_URL and OLLAMA_MODEL preserved — helper reads from process.env, never writes
Deterministic inline test double provider pattern (mirrors makeFakeProvider in tests/graph/current-understanding-synthesis.test.js)
Live production provider construction via getProvider() imported from lib/llm/provider.js (canonical seam, first extraction at line 87-90 of experiment helper)

Production Entry Point

synthesizeCurrentUnderstanding() from lib/graph/current-understanding-synthesis.js

Input contract: { situationGraph: object, findings: array } Optional deps: { provider: object, modelName: string }

Output: { currentUnderstanding: string }

Verification Results

All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:

Fixed compatible input reaches synthesizeCurrentUnderstanding()
Expected structured output { currentUnderstanding } captured
Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
Exactly one synthesis invocation per run
Input fixture is not mutated (deep clone + compare)
No persistence/Supabase/DB operations
Live-mode wiring verified via deterministic injected mock provider
Deterministic mode behaviour preserved unchanged

Iteration 5 — Live Provider Seam

New seam: `createLiveProvider()` in `scripts/current-understanding-experiment-helper.cjs` (line 87-90)

Reuses the canonical provider construction from lib/llm/provider.js:

const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
return getProvider();

New CLI flag: `--live`

When --live is specified alongside --file, the helper uses the production provider instead of the test double. Requires OLLAMA_BASE_URL and OLLAMA_MODEL from .env.local.

Test coverage for live-mode (10 new tests)

All 10 tests use a deterministic injected mock provider (no network). They verify:

Fixture passes through unchanged to synthesis function
Provider seam invoked exactly once
Evidence projection passed as prompt with correct headers
Current Understanding text captured correctly
Schema matches synthesisOutputSchema
No persistence operations
Deterministic mode still works (regression)
Rule 2 boundary text present
Zero I/O in tests (mock is pure function)
Full apparatus output shape validated for live-mode path

Iteration 4 — Regression Fixture

Fixture path: tests/fixtures/current-understanding-original-investigation.json

Source: authenticated GET /api/investigations/6b080d97-033e-477b-9d41-7b2f4f04bbed (2026-10-08)

Contract: { situationGraph: object, findings: array }

Contents:

situationGraph: 11 nodes (1 state, 5 observation, 2 metric, 3 unknown), 10 edges, full reasoningState, resolvedNodeIds, centralStatement, activeUnknownNodeId
findings: 9 findings total; all 7 previously verified IDs present (hciikc, ybfsm3, k45vyt, vhftyr, 9bgkmw, 23192q, kkh39k)

Verified: helper parses without invoking live synthesis; fixture immutability confirmed via deep-clone roundtrip.

4.4 KiB Raw Blame History