robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(confidence-engine): add CE-ULI-08 two-condition uncertainty comparison apparatus

Browse Source 
Adds experimental apparatus for testing whether removing the question field
from importantUnknowns in reconstruction output improves semantic fidelity.

New files:
- lib/reconstruction/schema-variant.js (conditionB schema + production passthrough)
- prompts/reconstruct-v0.5-b-noreq.md (alt prompt, identical line count to prod)

Changed files:
- lib/graph/orchestrator.js (pass generationSchema through dependencies)
- lib/analysis.js (accept opts.generationSchema fallback to original schema)
- scripts/start-case-experiment-helper.cjs (--condition B/A flag support)
- tests/scripts/start-case-experiment-helper.test.js (8 new deterministic tests)

43 targeted deterministic tests PASS. Production callers use original prompt/schema unchanged.
No live model calls. No full suite evidence available.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
robbond
2026-10-12 07:33:31 +01:00
co-authored by Claude Code
parent a54b36a2b2
commit 3b04d05355
7 changed files with 544 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/uncertainty-led-investigation-hypothesis.md
+2
View File
@@ -109,3 +109,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **Iteration 3 — Extract Focused Target Input Deterministic Test (Recovery):** Purified focused-investigation target-node extraction into a standalone pure function `extractFocusedTargetInput(situationGraph, targetNodeId)` within `tests/graph/live-update-experiment-helper.cjs`. The function uses `situationGraph.nodes.find(n => n.id === targetNodeId)` with `?? ""` empty-string fallback for unknown nodes. Reused inside the guarded focused-deconstruction path of `composeDeepInvestigationCheckpoint`. Deterministically tested with known-node exact-match assertions, unknown-node empty-fallback assertions, graph-immutable-side-effect assertion, and source-pattern integration assertion — all 5 assertions PASS. Zero live model calls.
- **Experiment 9 — Explicit Dependency Evidence and Focused Investigation (2026-10-09, Iteration 9 of 10):** Single-live-run semantic observation. Controlled scenario: synthetic answer explicitly raises three cross-area dependencies (recruitment → software integration dependency, integration → customer commitments dependency, cascading-delay risk). Model response: **PARTIAL** fidelity. Key finding — the model represented two explicit cross-area relationships and retained the overarching feasibility uncertainty ("Whether unconfirmed candidate notice periods and required onboarding duration leave enough time for software integration and customer commitments within nine months"), which was notably absent in Experiment 4's PARTIAL result. The follow-up question classifies as LEGITIMATE SPECIFICITY (narrows to notice periods but retains broader timeline context). Strongest counterevidence: overarching uncertainty combines multiple domains into one compound question, potentially obscuring the original recruitment-specific framing. Follow-up focuses on hiring logistics with cross-area context rather than independently investigating each dependency layer. Single-run limitation. Evidence file: `.evidence-temp/headless-semantic/HS-03C-focused-E9.json`.
- **CE-ULI-08 — Apparatus for Condition A/B Uncertainty Comparison (Iteration 3 of 10):** Bounded experimental apparatus implementing a two-condition treatment to test whether removing the `question` field from `importantUnknowns` in reconstruction output improves semantic fidelity. Changes: (1) `lib/reconstruction/schema-variant.js` — new `conditionBJsonSchema` using `itemSchemaV1` (id, description, confidence only; no question) and exports production `conditionAJsonSchema`; (2) `prompts/reconstruct-v0.5-b-noreq.md` — alt prompt with identical line count to production `reconstruct-v0.5.md`, Rule 7a rewritten to use description-only uncertainty representation instead of interrogative form, no question-generation language; (3) `lib/graph/orchestrator.js` — passes `generationSchema` through dependencies when present; (4) `lib/analysis.js` — accepts `opts.generationSchema ?? reconstructionGenerationJsonSchema` for experimental override; (5) `scripts/start-case-experiment-helper.cjs` — `--condition B/A` flag loads alt prompt/schema or falls back to production defaults; (6) `tests/scripts/start-case-experiment-helper.test.js` — 8 new tests verifying schema structure, prompt differences, and seam passthrough. **43 targeted deterministic tests PASS**. Full suite was interrupted and is not evidence of a complete suite pass. No live model calls. Production callers still use the original `reconstruct-v0.5.md` prompt and production schema unchanged (fallback path via `??`). Both conditions use the canonical provider invocation seam.