test(confidence-engine): enable live current understanding experiment execution

Extend the CU synthesis experiment helper with a --live flag that
uses the production provider (getProvider from lib/llm/provider.js)
for live model invocation. Preserves deterministic mode unchanged.

Adds 10 new deterministic tests verifying live-mode wiring via
injected mock provider: fixture pass-through, single invocation,
schema correctness, immutability, persistence safety, and regression
of existing deterministic behaviour. Zero network calls in all tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
2026-10-08 09:59:03 +01:00
co-authored by Claude Code
parent 425e1e7805
commit ce7df91a42
3 changed files with 266 additions and 16 deletions
@@ -1,22 +1,26 @@
# CU Synthesis Experiment Apparatus — Iteration 3
# CU Synthesis Experiment Apparatus — Iteration 5
## Purpose
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, without live model calls.
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.
## Files
| File | Role |
|------|------|
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) |
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (14 tests, 0 live calls) |
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes |
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (24 tests, 0 live calls) |
## Usage
### Helper (CLI)
```bash
# Deterministic mode (default — zero live model calls)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json
# Live production mode (uses configured OLLAMA_MODEL from .env.local)
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json
```
Fixture contract: `{ situationGraph: object, findings: array }`
@@ -33,6 +37,7 @@ npx vitest run tests/graph/current-understanding-experiment-helper.test.js --env
- Environment loading via `@next/env` `loadEnvConfig` (proven in both existing helpers)
- `.env.local` ownership of `OLLAMA_BASE_URL` and `OLLAMA_MODEL` preserved — helper reads from `process.env`, never writes
- Deterministic inline test double provider pattern (mirrors `makeFakeProvider` in `tests/graph/current-understanding-synthesis.test.js`)
- Live production provider construction via `getProvider()` imported from `lib/llm/provider.js` (canonical seam, first extraction at line 87-90 of experiment helper)
## Production Entry Point
@@ -45,13 +50,43 @@ Output: `{ currentUnderstanding: string }`
## Verification Results
All 14 deterministic tests PASS. Zero live model calls. The following invariants verified:
All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:
- Fixed compatible input reaches `synthesizeCurrentUnderstanding()`
- Expected structured output `{ currentUnderstanding }` captured
- Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
- Exactly one synthesis invocation per run
- Input fixture is not mutated (deep clone + compare)
- No persistence/Supabase/DB operations
- Live-mode wiring verified via deterministic injected mock provider
- Deterministic mode behaviour preserved unchanged
## Iteration 5 — Live Provider Seam
### New seam: `createLiveProvider()` in `scripts/current-understanding-experiment-helper.cjs` (line 87-90)
Reuses the canonical provider construction from `lib/llm/provider.js`:
```js
const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
return getProvider();
```
### New CLI flag: `--live`
When `--live` is specified alongside `--file`, the helper uses the production provider instead of the test double. Requires `OLLAMA_BASE_URL` and `OLLAMA_MODEL` from `.env.local`.
### Test coverage for live-mode (10 new tests)
All 10 tests use a deterministic injected mock provider (no network). They verify:
1. Fixture passes through unchanged to synthesis function
2. Provider seam invoked exactly once
3. Evidence projection passed as prompt with correct headers
4. Current Understanding text captured correctly
5. Schema matches `synthesisOutputSchema`
6. No persistence operations
7. Deterministic mode still works (regression)
8. Rule 2 boundary text present
9. Zero I/O in tests (mock is pure function)
10. Full apparatus output shape validated for live-mode path
## Iteration 4 — Regression Fixture