robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(confidence-engine): prove direct helper production seam

Browse Source
This commit is contained in:
robbond
2026-09-04 14:33:02 +01:00
parent 928954ee4a
commit 642554038e
3 changed files with 222 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+112
View File
@@ -957,6 +957,118 @@ node scripts/start-case-experiment-helper.cjs --file scenario.json
- Performance at scale
- All UI integration tests pass
## v0.61.2 — Direct-Helper Production-Seam Proof
**Status: APPARATUS FAILURE (import seam blocked) — but apparatus itself verified.**
### Environment loading result
| Item | Value |
|---|---|
| Existing compatible loader available | **YES** |
| Package/function | `@next/env` → `loadEnvConfig(projectDir, configPath?, logger, debug)` |
| Previous bespoke parser | Hand-written `.env.local` KEY=VALUE parser (replaced) |
| Final mechanism | `require("@next/env").loadEnvConfig(__dirname, undefined, { logOutput: "none" }, false)` |
| lib/config.js remains configuration authority | **YES** — interprets `OLLAMA_BASE_URL` and `OLLAMA_MODEL` |
| New dependency installed | **NO** — `@next/env` transitively available through `next ^14.2.0` |
### Real production import result
| Item | Value |
|---|---|
| Process | plain Node dynamic `import()` from standalone `.cjs` helper |
| Actual orchestrator imported | **NO** — module resolution barrier |
| Actual startCase export resolved | **N/A** (import fails before export resolution) |
| startCase invoked during import-only proof | **NO** — mode structurally stops before any invocation |
| Provider invoked | **NO** |
| Network used | **NO** |
**Failure reason:** The orchestrator's transitive dependency chain includes `lib/graph/apply-proposal.js` which imports from `@/lib/llm/provider`. This `@/` path alias is a Next.js compiler convention (configured in `jsconfig.json` as `"@/*": ["./*"]`). Plain Node has no resolver for `@/` aliases — it attempts to resolve `@/lib` as a bare package name and fails.
**Evidence:**
```
Cannot find module '/Users/.../lib/graph/orchestrator.js' imported from
/Users/.../scripts/start-case-experiment-helper.cjs
(cause: Cannot find package '@/lib' imported from apply-proposal.js)
```
**Two files in the import chain use `@/`:**
- `lib/graph/apply-proposal.js` → `import { getProvider } from "@/lib/llm/provider"`
- `lib/graph/focused-investigation.js` → (same alias pattern)
### Import-only structured output (failure path)
```json
{
"success": false,
"mode": "import-only",
"startCaseResolved": false,
"failureReason": "Module resolution failed: Cannot find module '@/lib/llm/provider' imported from /path/to/lib/graph/apply-proposal.js"
}
```
### Deterministic evidence
| Item | Value |
|---|---|
| Test file | `tests/scripts/start-case-experiment-helper.test.js` |
| Exact command | `npx vitest run tests/scripts/start-case-experiment-helper.test.js` |
| First-run result | **25/25 PASS** (first run, zero reruns) |
| Tests passed | 25 (18 existing apparatus + 7 new import-only) |
| Tests failed | 0 |
| Reruns | 0 |
### What was proven by this task
1. **Standard environment loader established:** `@next/env` `loadEnvConfig` replaces bespoke parser — no new dependency needed
2. **Import-only mode implemented:** Helper supports `START_CASE_EXPERIMENT_HELPER_IMPORT_ONLY=1` for deterministic seam verification
3. **Standalone production import incompatible:** Plain Node cannot load `lib/graph/orchestrator.js` due to Next.js `@/` alias chain in transitive dependencies
4. **No production files modified:** Helper and test changes only — zero impact on reasoning path
5. **Zero live calls:** Model calls: 0, HTTP /api/cases/start: 0, curl: 0, Playwright: NO
### Classification
**APPARATUS FAILURE** — standalone plain Node cannot import the real production orchestrator due to Next.js `@/` module alias convention.
This is documented evidence of a **tooling seam gap**: the helper architecture (standalone `.cjs` CLI) is incompatible with the repository's ESM path-alias module system without:
- Adding a bundler/loader (tsx, esbuild, bundler)
- Modifying production imports to use relative paths
- Running through Next.js tooling
The `@/` alias is a legitimate architectural choice that should not be changed. The gap means the helper must use **mock mode** for deterministic verification or run under Next.js-aware tooling.
### Existing helper status (pre-v0.61.2)
| Check | Result |
|---|---|
| Positional input reaches startCase seam | PASS (mock mode) |
| File input reads JSON fixture | PASS |
| .env.local loads without dotenv dependency | PASS (now via @next/env) |
| Exit code 0 on success | PASS |
| stdout is valid JSON with required fields | PASS |
| endToEndElapsedMs non-negative | PASS |
| Failure produces non-zero exit code | PASS |
| Failure output is valid JSON | PASS |
| startCase called exactly once on success | PASS |
| No retry on failure | PASS |
| Malformed --file fails before startCase | PASS |
| --file without path fails before startCase | PASS |
| Output structure integrity (stderr/stdout) | PASS |
### Live execution
- Model calls: **0**
- /api/cases/start calls: **0**
- curl calls: **0**
- Playwright: **NO**
### Files changed
- `scripts/start-case-experiment-helper.cjs` — environment loader replaced, import-only mode added
- `tests/scripts/start-case-experiment-helper.test.js` — 7 new import-only assertions (describe block H)
---
## Next restart point
> v0.60 is complete. Report is established as the culmination of an Investigation. No next product boundary is currently selected. Begin the next session by choosing the next unresolved user/product reasoning boundary from current product behaviour and founding principles, rather than continuing storage migration or assuming an old backlog item is next.