robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(confidence-engine): verify direct decomposition helper

Browse Source
This commit is contained in:
robbond
2026-09-04 14:02:36 +01:00
parent 41ea2cb6b9
commit 928954ee4a
3 changed files with 376 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+45
View File
@@ -912,6 +912,51 @@ node scripts/start-case-experiment-helper.cjs --file scenario.json
- **Playwright NOT default** for decomposition-only semantic experiments (apparatus reaches production reasoning path via direct import or HTTP)
- **Playwright REMAINS required** when the experiment concerns visible/browser behaviour, UI state transitions, or localStorage hydration
## v0.61.1 — Start-Case Experiment Helper Apparatus Verification
**Status: PASSED (18/18, first run, zero reruns)**
**Purpose:** Prove `scripts/start-case-experiment-helper.cjs` works as a standalone Node command with the same configured environment as production, without requiring live model calls or browser state.
### What was verified
| Check | Source | Result |
|---|---|---|
| A — Positional input reaches startCase seam | `tests/scripts/start-case-experiment-helper.test.js:76` | PASS |
| B — File input reads JSON fixture | `tests/scripts/start-case-experiment-helper.test.js:89` | PASS |
| B — Malformed file fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:224` | PASS |
| C — .env.local loads without dotenv dependency | `tests/scripts/start-case-experiment-helper.test.js:116` | PASS |
| D — Exit code 0 on success | `tests/scripts/start-case-experiment-helper.test.js:131` | PASS |
| D — stdout is valid JSON with required fields | `tests/scripts/start-case-experiment-helper.test.js:136,143` | PASS |
| D — endToEndElapsedMs non-negative | `tests/scripts/start-case-experiment-helper.test.js:152` | PASS |
| E — Failure produces non-zero exit code | `tests/scripts/start-case-experiment-helper.test.js:169` | PASS |
| E — Failure output is valid JSON | `tests/scripts/start-case-experiment-helper.test.js:180` | PASS |
| F — startCase called exactly once on success | `tests/scripts/start-case-experiment-helper.test.js:194` | PASS |
| F — No retry on failure | `tests/scripts/start-case-experiment-helper.test.js:205` | PASS |
| G — Malformed --file fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:224` | PASS |
| G — --file without path fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:235` | PASS |
| Integrity — no diagnostic leakage to stdout | `tests/scripts/start-case-experiment-helper.test.js:248` | PASS |
### Apparatus fixes applied during verification
1. **Inline `.env.local` parser** — replaced `require("dotenv")` (MODULE_NOT_FOUND) with built-in `fs` + `path` loader
2. **Mock injection seam** — `START_CASE_EXPERIMENT_HELPER_MOCK=1` enables deterministic test doubles without live calls
3. **Structured failure output** — plain text console.error → valid JSON on stdout
4. **execFile harness fix** — capture stdout/stderr regardless of execFile error state
### Zero-live-call verification
- Live model calls: ZERO
- Build required: NO (scripts/test only, no production code changes)
- Exact command: `npx vitest run tests/scripts/start-case-experiment-helper.test.js`
### NOT proven
- Actual semantic quality of decomposition output
- Behaviour with real LLM endpoints
- Performance at scale
- All UI integration tests pass
## Next restart point
> v0.60 is complete. Report is established as the culmination of an Investigation. No next product boundary is currently selected. Begin the next session by choosing the next unresolved user/product reasoning boundary from current product behaviour and founding principles, rather than continuing storage migration or assuming an old backlog item is next.