106 lines
4.6 KiB
Markdown
106 lines
4.6 KiB
Markdown
# CU Synthesis Experiment Apparatus — Iteration 5
|
|
|
|
## Purpose
|
|
|
|
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.
|
|
|
|
## Files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes |
|
|
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (34 tests, 0 live calls) |
|
|
|
|
## Usage
|
|
|
|
### Helper (CLI)
|
|
|
|
```bash
|
|
# Deterministic mode (default — zero live model calls)
|
|
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json
|
|
|
|
# Live production mode (uses configured OLLAMA_MODEL from .env.local)
|
|
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json
|
|
|
|
# With output exposure (exposes exact currentUnderstanding narrative in structured result)
|
|
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json --include-output
|
|
```
|
|
|
|
Fixture contract: `{ situationGraph: object, findings: array }`
|
|
|
|
### Test
|
|
|
|
```bash
|
|
npx vitest run tests/graph/current-understanding-experiment-helper.test.js --environment=node
|
|
```
|
|
|
|
## Existing Mechanics Reused
|
|
|
|
- Dynamic `import()` seam from CJS to ESM (proven in `scripts/overview-experiment-helper.cjs` line 64)
|
|
- Environment loading via `@next/env` `loadEnvConfig` (proven in both existing helpers)
|
|
- `.env.local` ownership of `OLLAMA_BASE_URL` and `OLLAMA_MODEL` preserved — helper reads from `process.env`, never writes
|
|
- Deterministic inline test double provider pattern (mirrors `makeFakeProvider` in `tests/graph/current-understanding-synthesis.test.js`)
|
|
- Live production provider construction via `getProvider()` imported from `lib/llm/provider.js` (canonical seam, first extraction at line 87-90 of experiment helper)
|
|
|
|
## Production Entry Point
|
|
|
|
`synthesizeCurrentUnderstanding()` from `lib/graph/current-understanding-synthesis.js`
|
|
|
|
Input contract: `{ situationGraph: object, findings: array }`
|
|
Optional deps: `{ provider: object, modelName: string }`
|
|
|
|
Output: `{ currentUnderstanding: string }`
|
|
|
|
## Verification Results
|
|
|
|
All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:
|
|
- Fixed compatible input reaches `synthesizeCurrentUnderstanding()`
|
|
- Expected structured output `{ currentUnderstanding }` captured
|
|
- Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
|
|
- Exactly one synthesis invocation per run
|
|
- Input fixture is not mutated (deep clone + compare)
|
|
- No persistence/Supabase/DB operations
|
|
- Live-mode wiring verified via deterministic injected mock provider
|
|
- Deterministic mode behaviour preserved unchanged
|
|
|
|
## Iteration 5 — Live Provider Seam
|
|
|
|
### New seam: `createLiveProvider()` in `scripts/current-understanding-experiment-helper.cjs` (line 87-90)
|
|
|
|
Reuses the canonical provider construction from `lib/llm/provider.js`:
|
|
```js
|
|
const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
|
|
return getProvider();
|
|
```
|
|
|
|
### New CLI flag: `--live`
|
|
|
|
When `--live` is specified alongside `--file`, the helper uses the production provider instead of the test double. Requires `OLLAMA_BASE_URL` and `OLLAMA_MODEL` from `.env.local`.
|
|
|
|
### Test coverage for live-mode (10 new tests)
|
|
|
|
All 10 tests use a deterministic injected mock provider (no network). They verify:
|
|
1. Fixture passes through unchanged to synthesis function
|
|
2. Provider seam invoked exactly once
|
|
3. Evidence projection passed as prompt with correct headers
|
|
4. Current Understanding text captured correctly
|
|
5. Schema matches `synthesisOutputSchema`
|
|
6. No persistence operations
|
|
7. Deterministic mode still works (regression)
|
|
8. Rule 2 boundary text present
|
|
9. Zero I/O in tests (mock is pure function)
|
|
10. Full apparatus output shape validated for live-mode path
|
|
|
|
## Iteration 4 — Regression Fixture
|
|
|
|
Fixture path: `tests/fixtures/current-understanding-original-investigation.json`
|
|
|
|
Source: authenticated GET `/api/investigations/6b080d97-033e-477b-9d41-7b2f4f04bbed` (2026-10-08)
|
|
|
|
Contract: `{ situationGraph: object, findings: array }`
|
|
|
|
Contents:
|
|
- `situationGraph`: 11 nodes (1 state, 5 observation, 2 metric, 3 unknown), 10 edges, full `reasoningState`, `resolvedNodeIds`, `centralStatement`, `activeUnknownNodeId`
|
|
- `findings`: 9 findings total; all 7 previously verified IDs present (`hciikc`, `ybfsm3`, `k45vyt`, `vhftyr`, `9bgkmw`, `23192q`, `kkh39k`)
|
|
|
|
Verified: helper parses without invoking live synthesis; fixture immutability confirmed via deep-clone roundtrip. |