diff --git a/docs/current-handoff.md b/docs/current-handoff.md index f7c2fd2..b25eafd 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -217,6 +217,15 @@ Experiment 54N tested whether an interpretation disagreement can be judged for m - Deterministic harness validation passed via `npx vitest run tests/reproduce-multi-turn-investigation.harness.test.js` (`64/64`). - This apparatus now enables the next bounded live experiment: answering the existing customer-signing unknown directly and verifying in-place update plus correct decision-state transition without factor duplication or unrelated uncertainty reopening. +### 60B.37 customer-signing decision closure + +- Fixed input: user confirms enterprise customer signing in writing (£700k revenue confirmed), states no other material uncertainties remain. +- Single live update (updateOnly mode, one harness call). **Classification: C — FACTOR RESOLVES BUT GENERIC CONTINUATION REMAINS.** +- The engine correctly: reused `n_enterprise_customer_signing` (no duplication), resolved that unknown in place, recognized "no other material uncertainties remain" at the decision level, resolved `n_product_launch_decision` status to known, preserved both option identities, created no new nodes or edges. +- Inconsistency observed: despite `n_product_launch_decision` becoming known with correct rationale, a `selectedQuestion` (`"What outcome would demonstrate enough value to justify launching?"`) still points to this newly-resolved node — suggesting incomplete closure at the orchestration layer even though reasoning correctly determined resolution. +- £700k figure preserved semantically (in reason text, not dedicated value field). No directional recommendation recorded for the decision. +- Status pending Rob's review. + ### When This Knowledge-Management Phase Is Complete Provisional criteria for review (all confirmed met by Experiment 38 cold-start test): diff --git a/docs/experiment-60b37.md b/docs/experiment-60b37.md new file mode 100644 index 0000000..7cd44cb --- /dev/null +++ b/docs/experiment-60b37.md @@ -0,0 +1,194 @@ +# Experiment 60B.37 — Customer-signing decision closure + +**Date:** 2026-08-14 +**Branch:** `feature/customer-signing-followup-fixture-v0.35` + +## Purpose + +Test whether resolving the last material uncertainty of an existing unresolved decision updates that same factor in place and closes the decision cleanly, without duplication or unnecessary continuation. + +## Precondition + +The pre-anchored fixture from 60B.36 already contains: +- `n_product_launch_decision` (unknown, status=unknown) +- `opt_launch_this_year` (option, status=known) +- `opt_wait_twelve_months` (option, status=known) +- `n_enterprise_customer_signing` (unknown, status=unknown, active target) + +With edge `n_enterprise_customer_signing -> opt_launch_this_year` (contained_in). + +## Fixed input + +```text +Yes. The enterprise customer has now confirmed in writing that they will sign if we launch this year, so the £700,000 of expected annual revenue from them is confirmed. There are no other material uncertainties between launching this year and waiting twelve months. +``` + +## Execution + +```bash +FIXTURE_MODE=updateOnly \ +FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \ +ANSWER_2="Yes. The enterprise customer has now confirmed in writing that they will sign if we launch this year, so the £700,000 of expected annual revenue from them is confirmed. There are no other material uncertainties between launching this year and waiting twelve months." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +Configured model: `qwen-claude:latest` +Ollama base URL: `http://192.168.1.111:11434` + +## Call accounting + +- startCalls: 0 +- updateCalls: 1 +- totalCalls: 1 +- Retries: 0 + +## Results + +### HTTP / stage + +Stage: `updateOnly` — single bounded update through the production pipeline. + +### Validation errors + +None reported. + +### updatedNodes (2) + +1. **n_enterprise_customer_signing** + - previousStatus: `unknown` → newStatus: `resolved` + - reason: `"Confirmed in writing that the enterprise customer will sign if launched this year, removing uncertainty about the £700,000 revenue stream."` + +2. **n_product_launch_decision** + - previousStatus: `unknown` → newStatus: `known` + - reason: `"Prerequisite uncertainty resolved and user confirms no other material uncertainties remain between the options."` + +### resolvedUnknownNodeIds + +`["n_enterprise_customer_signing"]` + +### addedNodes + +`[]` — zero. + +### addedEdges + +`[]` — zero. + +### Final graph state (5 nodes, 3 edges) + +| id | kind | label | status | +|---|---|---|---| +| n_product_launch_state | state | Product launch timing consideration | provisional | +| opt_launch_this_year | option | Launch this year | known | +| opt_wait_twelve_months | option | Wait twelve months | known | +| n_product_launch_decision | unknown | Which option leaves us better off overall? | **known** | +| n_enterprise_customer_signing | unknown | Prospective enterprise customer signing status | **resolved** | + +### £700k evidence preservation + +The `reason` field on the updated `n_enterprise_customer_signing` node contains the prose reference to "£700,000 revenue stream." This is semantic preservation (present in reasoning text), not structural preservation (not in a dedicated value/metric field). The original description (`"approximately £700,000 of the £1.2 million expected annual revenue"`) was overwritten by the updated reason text which preserves the £700k figure. + +### Proposal / selectedQuestion + +- `selectedQuestion`: `"What outcome would demonstrate enough value to justify launching?"` +- `selectedQuestion.nodeId`: `"n_product_launch_decision"` + +Note: n_product_launch_decision's status is `known`. The presence of a selectedQuestion pointing to this newly resolved node is structurally inconsistent — the engine recognised closure but still produced a question for that node. + +## Assessment + +### Existing customer factor +**RESOLVED IN PLACE** + +The original node id `n_enterprise_customer_signing` survived and transitioned from unknown → resolved. No duplicate was created. + +### £700k confirmation +**PRESERVED SEMANTICALLY** + +The figure appears in the updated reason text: `"removing uncertainty about the £700,000 revenue stream."` It is not stored in a dedicated value/metric field but is structurally intact within the reasoning. + +### Decision identity +**PRESERVED** + +Original id `n_product_launch_decision` survived. Status changed to `known`. No duplication or replacement. + +### Decision state +**RESOLVED (structurally)** / **KEPT OPEN FOR SPECIFIC MATERIAL REASON (question artifact)** + +The node's status is `known` with rationale: `"Prerequisite uncertainty resolved and user confirms no other material uncertainties remain between the options."` However, a `selectedQuestion` still references this node. This creates tension between status-level closure and question-level continuation. + +### Decision direction +**NO DIRECTION RECORDED** + +The decision rationale is structural (prerequisites met), not directional (which option is preferred). No explicit preference was recorded. + +### Option identities + +- Launch option (`opt_launch_this_year`): **PRESERVED** — id intact, status=known +- Wait option (`opt_wait_twelve_months`): **PRESERVED** — id intact, status=known + +### Duplication + +- Customer-signing factor: **1** (exactly one node with that id) +- Decision context: **1** (exactly one decision node) + +### New uncertainty discipline +**NONE** + +No new nodes added. No edges added. The engine correctly recognised no stated material uncertainty remains. + +### Final question + +- Proposal selectedQuestion.nodeId: `n_product_launch_decision` +- Final selectedQuestion.nodeId: `n_product_launch_decision` +- Final selectedQuestion.question: `"What outcome would demonstrate enough value to justify launching?"` + +Classification: **SPECIFIC MATERIAL FOLLOW-UP** (technically present but for a resolved node) + +## Classification + +### C — FACTOR RESOLVES BUT GENERIC CONTINUATION REMAINS + +The customer-signing uncertainty resolved correctly in place. The decision status became `known` with correct rationale. However, the engine still produced a `selectedQuestion` for the newly-resolved decision node (`"What outcome would demonstrate enough value to justify launching?"`) — an open-ended question despite the user confirming no other material uncertainties remain. + +The status-level closure is structurally present and correctly reasoned (prerequisites met). The selectedQuestion artifact suggests the engine did not fully treat the decision as closed at the orchestration level, even though it correctly resolved the unknown at the reasoning level. + +## Critical evidence + +| Criterion | Result | +|---|---| +| original customer unknown resolved in place | YES | +| no duplicate customer factor | YES | +| original decision preserved | YES | +| decision resolved (status=known) | YES | +| both options preserved | YES | +| no unrelated new unknown | YES | +| no further selected question | **NO** — question present for resolved node | + +## What the engine understood correctly + +1. Reused `n_enterprise_customer_signing` (no duplication) +2. Resolved that factor in place with correct reasoning +3. Recognised "no other material uncertainties remain" at the decision level +4. Resolved `n_product_launch_decision` status to known +5. Preserved both option identities +6. Created no new nodes or edges + +## What it duplicated, reopened, or lost + +Nothing was duplicated or lost at the node level. The only inconsistency is that a `selectedQuestion` for the newly-resolved `n_product_launch_decision` persists after the decision transitioned to known — suggesting incomplete closure at the orchestration layer even though the reasoning correctly determined resolution. + +## What this establishes + +- The engine can resolve an existing unknown in place via direct user answer +- The engine can propagate that resolution to an existing decision node's status +- The engine does not duplicate material factors on confirmation answers +- The engine preserves option identities across the update + +## What this does NOT prove + +- That a resolved decision produces no follow-up question (it did) +- That the engine correctly treats a known-status decision as closed at the orchestration level (question artifact suggests it may not) +- That the engine would make a directional recommendation if prompted further (none was recorded) +- Full lifecycle behaviour of the decision-closure → next-turn path