test(confidence-engine): enable live current understanding experiment execution
Extend the CU synthesis experiment helper with a --live flag that uses the production provider (getProvider from lib/llm/provider.js) for live model invocation. Preserves deterministic mode unchanged. Adds 10 new deterministic tests verifying live-mode wiring via injected mock provider: fixture pass-through, single invocation, schema correctness, immutability, persistence safety, and regression of existing deterministic behaviour. Zero network calls in all tests. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -1,22 +1,26 @@
|
||||
# CU Synthesis Experiment Apparatus — Iteration 3
|
||||
# CU Synthesis Experiment Apparatus — Iteration 5
|
||||
|
||||
## Purpose
|
||||
|
||||
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, without live model calls.
|
||||
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) |
|
||||
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (14 tests, 0 live calls) |
|
||||
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes |
|
||||
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (24 tests, 0 live calls) |
|
||||
|
||||
## Usage
|
||||
|
||||
### Helper (CLI)
|
||||
|
||||
```bash
|
||||
# Deterministic mode (default — zero live model calls)
|
||||
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json
|
||||
|
||||
# Live production mode (uses configured OLLAMA_MODEL from .env.local)
|
||||
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json
|
||||
```
|
||||
|
||||
Fixture contract: `{ situationGraph: object, findings: array }`
|
||||
@@ -33,6 +37,7 @@ npx vitest run tests/graph/current-understanding-experiment-helper.test.js --env
|
||||
- Environment loading via `@next/env` `loadEnvConfig` (proven in both existing helpers)
|
||||
- `.env.local` ownership of `OLLAMA_BASE_URL` and `OLLAMA_MODEL` preserved — helper reads from `process.env`, never writes
|
||||
- Deterministic inline test double provider pattern (mirrors `makeFakeProvider` in `tests/graph/current-understanding-synthesis.test.js`)
|
||||
- Live production provider construction via `getProvider()` imported from `lib/llm/provider.js` (canonical seam, first extraction at line 87-90 of experiment helper)
|
||||
|
||||
## Production Entry Point
|
||||
|
||||
@@ -45,13 +50,43 @@ Output: `{ currentUnderstanding: string }`
|
||||
|
||||
## Verification Results
|
||||
|
||||
All 14 deterministic tests PASS. Zero live model calls. The following invariants verified:
|
||||
All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:
|
||||
- Fixed compatible input reaches `synthesizeCurrentUnderstanding()`
|
||||
- Expected structured output `{ currentUnderstanding }` captured
|
||||
- Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
|
||||
- Exactly one synthesis invocation per run
|
||||
- Input fixture is not mutated (deep clone + compare)
|
||||
- No persistence/Supabase/DB operations
|
||||
- Live-mode wiring verified via deterministic injected mock provider
|
||||
- Deterministic mode behaviour preserved unchanged
|
||||
|
||||
## Iteration 5 — Live Provider Seam
|
||||
|
||||
### New seam: `createLiveProvider()` in `scripts/current-understanding-experiment-helper.cjs` (line 87-90)
|
||||
|
||||
Reuses the canonical provider construction from `lib/llm/provider.js`:
|
||||
```js
|
||||
const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
|
||||
return getProvider();
|
||||
```
|
||||
|
||||
### New CLI flag: `--live`
|
||||
|
||||
When `--live` is specified alongside `--file`, the helper uses the production provider instead of the test double. Requires `OLLAMA_BASE_URL` and `OLLAMA_MODEL` from `.env.local`.
|
||||
|
||||
### Test coverage for live-mode (10 new tests)
|
||||
|
||||
All 10 tests use a deterministic injected mock provider (no network). They verify:
|
||||
1. Fixture passes through unchanged to synthesis function
|
||||
2. Provider seam invoked exactly once
|
||||
3. Evidence projection passed as prompt with correct headers
|
||||
4. Current Understanding text captured correctly
|
||||
5. Schema matches `synthesisOutputSchema`
|
||||
6. No persistence operations
|
||||
7. Deterministic mode still works (regression)
|
||||
8. Rule 2 boundary text present
|
||||
9. Zero I/O in tests (mock is pure function)
|
||||
10. Full apparatus output shape validated for live-mode path
|
||||
|
||||
## Iteration 4 — Regression Fixture
|
||||
|
||||
|
||||
Reference in New Issue
Block a user