From 7fd05745e8e4420cad65e1c950d8f9296f6abef3 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 9 Oct 2026 14:12:35 +0100 Subject: [PATCH] =?UTF-8?q?feat(confidence-engine):=20record=20Experiment?= =?UTF-8?q?=204=20synthetic=20HS-03C=20focused=20investigation=20results?= =?UTF-8?q?=20=E2=80=94=20PARTIAL=20fidelity,=20semantic=20narrowing=20obs?= =?UTF-8?q?erved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Controlled synthetic experiment (Iteration 4): fixed recruitment scenario with synthetic answer showed partial preservation of the broader feasibility uncertainty. The model correctly preserved timing estimates as observations but narrowed the investigation to hiring pipeline logistics rather than cross-area launch feasibility. --- docs/current-handoff.md | 30 +++++++++++++++++++ ...ncertainty-led-investigation-hypothesis.md | 3 ++ 2 files changed, 33 insertions(+) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index b83ea9d..9a201d5 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -700,3 +700,33 @@ Tested canonical uncertainty description as initial display instead of generated **Branch:** `experiment/uncertainty-led-investigation` **Commit:** pending verification complete. + +### Iterated Six — Synthetic HS-03C Focused Investigation (Experiment 4, 2026-10-09) + +**Type:** Controlled synthetic experiment (Iteration 4 of bounded iteration loop). + +**Fixed inputs:** +- Scenario: HS-03C — launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. +- Target node: `n2yzwlv` — "Timeline and feasibility of recruiting two specialists within the nine-month window" +- `previousQuestion`: "What is the current hiring pipeline status for the two specialist roles, and can onboarding be completed within the nine-month window?" (supplied explicitly as `previousQuestion`, NOT from `node.openQuestion`) +- Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." +- Graph: constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`) +- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`) +- Apparatus: `composeDeepInvestigationCheckpoint()` from `tests/graph/live-update-experiment-helper.cjs`, calling production `focusedDeconstruct()` in `lib/graph/focused-investigation.js` +- Live calls: exactly 1 + +**Hypothesis:** Focused deconstruction can use the canonical description to preserve the broader recruitment uncertainty while treating the synthetic answer as evidence containing both observations and unresolved assumptions. The model should not treat a speculative recruitment estimate as a confirmed delivery schedule. + +**Result: PARTIAL semantic fidelity.** + +- **Recruitment timing — PRESERVED:** The model correctly extracted "estimated to take between three and six months" as an observation (not commitment), retained uncertainty about "candidate availability and notice periods remain unconfirmed," and produced the correct overarching uncertainty: "Whether the actual hiring duration and confirmed candidate start dates will allow both specialists to begin contributing within the nine-month window." +- **Recruitment feasibility — PARTIALLY PRESERVED:** The model captured whether recruitment can be completed within the nine-month window but did not explicitly retain uncertainty about whether both specialists can become *productive* in time, or whether cross-area cascading delays could prevent it. +- **Evidence integrity — FAITHFUL:** Correctly distinguished possible candidates from secured hires ("Several possible candidates" vs "neither position has been filled"), estimated duration from confirmed timing ("estimated" not "confirmed"), expected onboarding from completed onboarding, plausible feasibility from established feasibility. +- **Semantic narrowing — PARTIAL:** The model narrowed the investigation from "Timeline and feasibility of recruiting two specialists within the nine-month window" to essentially a hiring pipeline status question. The follow-up question focused on notice periods for preferred candidates rather than the broader feasibility concern: "What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods and necessary onboarding time?" +- **Facilitation — PARTIAL:** The follow-up question helps investigate unresolved matters but frames them as a hiring logistics exercise. It does not surface the risk of cascading delays across the three interdependent requirements (hiring, software integration, customer commitments) that the original graph captured via `unexplainedTransitions` node `utrans_dependency`. + +**Strongest counterevidence to FAITHFUL classification:** The canonical description contained "feasibility" — a broader question about whether two specialists can be recruited AND onboarded within the nine-month window. The model treated this as a hiring pipeline question, producing an observation of "estimated 3-6 months" (which is correct but does not capture feasibility) and losing the cross-area dependency risk explicitly present in the HS-03C graph. This reproduces the HS-03C partial transformation pattern documented in prior hypothesis analysis: moving from underlying feasibility/timing/dependency uncertainties to present evidence, schedules, or bottlenecks. + +**Input provenance:** Synthetic answer deliberately crafted for this controlled experiment as FICTIONAL TEST DATA. Not a real user response. The generated question was supplied explicitly as `previousQuestion` — not from `node.openQuestion`. This does not imply that the production UI always supplies `node.openQuestion` as `previousQuestion`. + +**Remaining limitations:** Single scenario shape only (recruitment). Does not establish consistency across domains or models. A single run does not prove generalisability or user comprehension. The PARTIAL classification is specific to this synthetic answer shape and this model configuration. diff --git a/docs/uncertainty-led-investigation-hypothesis.md b/docs/uncertainty-led-investigation-hypothesis.md index e55774c..b59ad21 100644 --- a/docs/uncertainty-led-investigation-hypothesis.md +++ b/docs/uncertainty-led-investigation-hypothesis.md @@ -97,3 +97,6 @@ Preserve existing graph relationships, stable identities, provenance, persistenc - **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified. ## Decision state: OPEN (after iteration 2 evidence). + +- **Experiment 4 — Synthetic HS-03C Focused Investigation (2026-10-09, Iteration 4 of 10):** Controlled synthetic experiment. Fixed scenario: launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. Target node `n2yzwlv`: "Timeline and feasibility of recruiting two specialists within the nine-month window." Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." The generated question was supplied explicitly as `previousQuestion` via `composeDeepInvestigationCheckpoint()`. Model: `qwen-claude:latest` (Ollama). Exactly one live LLM call. Result: **PARTIAL** semantic fidelity. The model correctly preserved the core uncertainty about whether recruitment can be completed within the nine-month window and correctly distinguished estimates from confirmed timing, possible candidates from secured hires, and expected onboarding from completed onboarding. However, it narrowed the investigation to hiring pipeline specifics ("What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods") rather than preserving the broader feasibility question of whether both specialists can become productive in time for the nine-month launch. The strongest counterevidence: the canonical description "Timeline and feasibility" was reduced to essentially a recruitment logistics investigation, losing the broader cross-area dependency and cascading-delay uncertainty captured in the original HS-03C graph (`unexplainedTransitions` with ID `utrans_dependency`). This is consistent with the HS-03C partial pattern from prior experiments. Input provenance: synthetic answer deliberately crafted for this controlled experiment, not a real user response. Graph constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`). +