docs(confidence-engine): record CE-ULI-09 paired observation — question-free reconstruction valid through canonical path (single paired comparison)
This commit is contained in:
@@ -111,3 +111,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
|
||||
- **Experiment 9 — Explicit Dependency Evidence and Focused Investigation (2026-10-09, Iteration 9 of 10):** Single-live-run semantic observation. Controlled scenario: synthetic answer explicitly raises three cross-area dependencies (recruitment → software integration dependency, integration → customer commitments dependency, cascading-delay risk). Model response: **PARTIAL** fidelity. Key finding — the model represented two explicit cross-area relationships and retained the overarching feasibility uncertainty ("Whether unconfirmed candidate notice periods and required onboarding duration leave enough time for software integration and customer commitments within nine months"), which was notably absent in Experiment 4's PARTIAL result. The follow-up question classifies as LEGITIMATE SPECIFICITY (narrows to notice periods but retains broader timeline context). Strongest counterevidence: overarching uncertainty combines multiple domains into one compound question, potentially obscuring the original recruitment-specific framing. Follow-up focuses on hiring logistics with cross-area context rather than independently investigating each dependency layer. Single-run limitation. Evidence file: `.evidence-temp/headless-semantic/HS-03C-focused-E9.json`.
|
||||
|
||||
- **CE-ULI-08 — Apparatus for Condition A/B Uncertainty Comparison (Iteration 3 of 10):** Bounded experimental apparatus implementing a two-condition treatment to test whether removing the `question` field from `importantUnknowns` in reconstruction output improves semantic fidelity. Changes: (1) `lib/reconstruction/schema-variant.js` — new `conditionBJsonSchema` using `itemSchemaV1` (id, description, confidence only; no question) and exports production `conditionAJsonSchema`; (2) `prompts/reconstruct-v0.5-b-noreq.md` — alt prompt with identical line count to production `reconstruct-v0.5.md`, Rule 7a rewritten to use description-only uncertainty representation instead of interrogative form, no question-generation language; (3) `lib/graph/orchestrator.js` — passes `generationSchema` through dependencies when present; (4) `lib/analysis.js` — accepts `opts.generationSchema ?? reconstructionGenerationJsonSchema` for experimental override; (5) `scripts/start-case-experiment-helper.cjs` — `--condition B/A` flag loads alt prompt/schema or falls back to production defaults; (6) `tests/scripts/start-case-experiment-helper.test.js` — 8 new tests verifying schema structure, prompt differences, and seam passthrough. **43 targeted deterministic tests PASS**. Full suite was interrupted and is not evidence of a complete suite pass. No live model calls. Production callers still use the original `reconstruct-v0.5.md` prompt and production schema unchanged (fallback path via `??`). Both conditions use the canonical provider invocation seam.
|
||||
|
||||
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
|
||||
|
||||
Reference in New Issue
Block a user