feat(confidence-engine): record Experiment 4 synthetic HS-03C focused investigation results — PARTIAL fidelity, semantic narrowing observed

Controlled synthetic experiment (Iteration 4): fixed recruitment scenario with
synthetic answer showed partial preservation of the broader feasibility uncertainty.
The model correctly preserved timing estimates as observations but narrowed the
investigation to hiring pipeline logistics rather than cross-area launch feasibility.
This commit is contained in:
2026-10-09 14:12:35 +01:00
parent d12615075d
commit 7fd05745e8
2 changed files with 33 additions and 0 deletions
+30
View File
@@ -700,3 +700,33 @@ Tested canonical uncertainty description as initial display instead of generated
**Branch:** `experiment/uncertainty-led-investigation`
**Commit:** pending verification complete.
### Iterated Six — Synthetic HS-03C Focused Investigation (Experiment 4, 2026-10-09)
**Type:** Controlled synthetic experiment (Iteration 4 of bounded iteration loop).
**Fixed inputs:**
- Scenario: HS-03C — launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments.
- Target node: `n2yzwlv` — "Timeline and feasibility of recruiting two specialists within the nine-month window"
- `previousQuestion`: "What is the current hiring pipeline status for the two specialist roles, and can onboarding be completed within the nine-month window?" (supplied explicitly as `previousQuestion`, NOT from `node.openQuestion`)
- Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch."
- Graph: constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `composeDeepInvestigationCheckpoint()` from `tests/graph/live-update-experiment-helper.cjs`, calling production `focusedDeconstruct()` in `lib/graph/focused-investigation.js`
- Live calls: exactly 1
**Hypothesis:** Focused deconstruction can use the canonical description to preserve the broader recruitment uncertainty while treating the synthetic answer as evidence containing both observations and unresolved assumptions. The model should not treat a speculative recruitment estimate as a confirmed delivery schedule.
**Result: PARTIAL semantic fidelity.**
- **Recruitment timing — PRESERVED:** The model correctly extracted "estimated to take between three and six months" as an observation (not commitment), retained uncertainty about "candidate availability and notice periods remain unconfirmed," and produced the correct overarching uncertainty: "Whether the actual hiring duration and confirmed candidate start dates will allow both specialists to begin contributing within the nine-month window."
- **Recruitment feasibility — PARTIALLY PRESERVED:** The model captured whether recruitment can be completed within the nine-month window but did not explicitly retain uncertainty about whether both specialists can become *productive* in time, or whether cross-area cascading delays could prevent it.
- **Evidence integrity — FAITHFUL:** Correctly distinguished possible candidates from secured hires ("Several possible candidates" vs "neither position has been filled"), estimated duration from confirmed timing ("estimated" not "confirmed"), expected onboarding from completed onboarding, plausible feasibility from established feasibility.
- **Semantic narrowing — PARTIAL:** The model narrowed the investigation from "Timeline and feasibility of recruiting two specialists within the nine-month window" to essentially a hiring pipeline status question. The follow-up question focused on notice periods for preferred candidates rather than the broader feasibility concern: "What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods and necessary onboarding time?"
- **Facilitation — PARTIAL:** The follow-up question helps investigate unresolved matters but frames them as a hiring logistics exercise. It does not surface the risk of cascading delays across the three interdependent requirements (hiring, software integration, customer commitments) that the original graph captured via `unexplainedTransitions` node `utrans_dependency`.
**Strongest counterevidence to FAITHFUL classification:** The canonical description contained "feasibility" — a broader question about whether two specialists can be recruited AND onboarded within the nine-month window. The model treated this as a hiring pipeline question, producing an observation of "estimated 3-6 months" (which is correct but does not capture feasibility) and losing the cross-area dependency risk explicitly present in the HS-03C graph. This reproduces the HS-03C partial transformation pattern documented in prior hypothesis analysis: moving from underlying feasibility/timing/dependency uncertainties to present evidence, schedules, or bottlenecks.
**Input provenance:** Synthetic answer deliberately crafted for this controlled experiment as FICTIONAL TEST DATA. Not a real user response. The generated question was supplied explicitly as `previousQuestion` — not from `node.openQuestion`. This does not imply that the production UI always supplies `node.openQuestion` as `previousQuestion`.
**Remaining limitations:** Single scenario shape only (recruitment). Does not establish consistency across domains or models. A single run does not prove generalisability or user comprehension. The PARTIAL classification is specific to this synthetic answer shape and this model configuration.
@@ -97,3 +97,6 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
## Decision state: OPEN (after iteration 2 evidence).
- **Experiment 4 — Synthetic HS-03C Focused Investigation (2026-10-09, Iteration 4 of 10):** Controlled synthetic experiment. Fixed scenario: launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. Target node `n2yzwlv`: "Timeline and feasibility of recruiting two specialists within the nine-month window." Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." The generated question was supplied explicitly as `previousQuestion` via `composeDeepInvestigationCheckpoint()`. Model: `qwen-claude:latest` (Ollama). Exactly one live LLM call. Result: **PARTIAL** semantic fidelity. The model correctly preserved the core uncertainty about whether recruitment can be completed within the nine-month window and correctly distinguished estimates from confirmed timing, possible candidates from secured hires, and expected onboarding from completed onboarding. However, it narrowed the investigation to hiring pipeline specifics ("What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods") rather than preserving the broader feasibility question of whether both specialists can become productive in time for the nine-month launch. The strongest counterevidence: the canonical description "Timeline and feasibility" was reduced to essentially a recruitment logistics investigation, losing the broader cross-area dependency and cascading-delay uncertainty captured in the original HS-03C graph (`unexplainedTransitions` with ID `utrans_dependency`). This is consistent with the HS-03C partial pattern from prior experiments. Input provenance: synthetic answer deliberately crafted for this controlled experiment, not a real user response. Graph constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`).