diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 3bfd23d..3dec5d7 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2841,3 +2841,9 @@ Experiment 60B.14 performed read-only design analysis on whether a newly-created --- Experiment 60B.15 addressed the unresolved boundary question from 60B.14: **what exact structural relationship is strong enough to count as "embedded in the active context" without becoming so permissive that genuine pattern transitions are hidden?** Analysis of all available signals confirmed: (1) SituationRelationship enum defines 11 edge types with distinct semantic force, (2) node-level parentId/childIds provide unambiguous hierarchy, (3) parent-chain-only is too narrow (misses edge-connected nodes), (4) any-graph-path is too permissive, (5) Candidate E — PARENT OR DECISION-OPTION PATH was selected as the winning predicate. The rule: a new unknown is embedded in active context Y if EITHER its parentId/ancestor chain reaches a node with pattern Y, OR it attaches via may_cause/causes/affects edge to an option contained_in a decision-unknown. This uses two independent routes with clear boundaries — neither alone sufficient, together covering all common embedding patterns. The genuine-transition test at apply-proposal.test.js:2486 (comparison child under decision parent) is preserved because comparison IS already in the ALLOWED list for decision context — structural embedding only affects the normalization step when intrinsic inference produces an incompatible result, which indicates likely wording drift. No new schema required. Implementation readiness: A — ready for bounded implementation. + +--- + +Experiment 60B.20 was the live verification of bounded structural context admission from 60B.19 (commit d871a8c). **Classification: A — FULL LIVE CHAIN CONFIRMED.** The exact client-retention case that failed at result_validation in 60B.12 now passes through all stages end-to-end. The model produced `n_client_retention_risk` (kind=unknown, status=unknown) with a `may_cause` edge from opt_relocate. Proposal applied successfully. Decision remains unresolved. Final selectedQuestion targets n_client_retention_risk specifically with text "will our largest client leave if we relocate?". Client risk clearly attributed to Relocate. No fabrication, no over-closure, no unrelated uncertainty invented. + +--- diff --git a/docs/experiment-60b20.md b/docs/experiment-60b20.md new file mode 100644 index 0000000..e102a0d --- /dev/null +++ b/docs/experiment-60b20.md @@ -0,0 +1,202 @@ +# Experiment 60B.20 — Live Verification of Bounded Structural Context Admission + +**Branch:** `feature/reasoning-context-compatibility-v0.28` +**Starting HEAD:** `a7ca8d7` (feature/reasoning-context-compatibility-v0.28) +**Date:** 2026-08-13 +**Status:** COMPLETE +**Type:** LIVE RUN — Single-call live regression of 60B.19 bounded structural context admission + +## Objective + +Rerun the exact client-retention case that was blocked in 60B.12: + +> Does the engine now accept the client-retention unknown through reasoning-pattern validation, preserve it as material unresolved, and target it as the final question? + +## Following + +Experiment 60B.19 — Bounded structural context admission (commit d871a8c) +Experiment 60B.12 — Previous blocked live case (rejection at result_validation) + +This is the live regression that 60B.19 explicitly identified as unproven: +> "the precise 60B.12 client-retention continuation case in a live end-to-end path where the model introduces the factor and the final selected target preserves that same ready material unknown." + +## Configured Model + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 + +## Fixed Starting Graph + +**Fixture:** `tests/fixtures/pre-anchored-decision-options.json` + +| Node | Kind | Status | Label | +|------|------|--------|-------| +| n_relocation_state | state | provisional | Engineering team relocation consideration | +| opt_relocate | option | known | Relocate to Manchester | +| opt_stay_put | option | known | Stay in London (Status Quo) | +| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? | + +## Fixed Answer (verbatim, exact) + +> We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year. + +## Execution + +Exactly one update call through `reproduce-multi-turn-investigation.mjs` in `updateOnly` mode. + +## Call Accounting + +``` +startCalls: 0 +updateCalls: 1 +totalCalls: 1 +Retries: 0 +Second live invocation: NO +``` + +## UPDATE — Raw Output + +**HTTP:** 200 (no rejection) +**Stage:** All stages passed (result_validation did not reject) +**Validation errors:** None (structuralActionRequired = null) +**Proposal applied:** YES + +### Structural Data + +- **updatedNodes:** `[]` +- **resolvedUnknownNodeIds:** `[]` +- **addedNodes:** + ```json + { + "id": "n_client_retention_risk", + "label": "Will our largest client leave if we relocate?", + "description": "Uncertainty regarding whether our largest client would depart following a relocation to Manchester; matters because their departure would cost approximately £5M per year, potentially outweighing the relocation savings and necessitating a revised net-value comparison.", + "kind": "unknown", + "status": "unknown", + "confidence": "medium", + "dependsOn": ["opt_relocate"], + "parentId": null + } + ``` +- **addedEdges:** + ```json + { + "id": "e-opt-rel-to-client-risk", + "fromNodeId": "opt_relocate", + "toNodeId": "n_client_retention_risk", + "relationship": "may_cause" + } + ``` + +### Question Targeting + +- **selectedQuestion.nodeId:** `n_client_retention_risk` +- **selectedQuestion.question:** `"will our largest client leave if we relocate?"` + +### Resulting Graph State + +``` +Nodes (5): + n_relocation_state | state | provisional + opt_relocate | option | known + opt_stay_put | option | known + n_relocation_decision | unknown | unknown ← still unresolved + n_client_retention_risk | unknown | unknown ← newly added, unresolved + +Edges (3): + opt_relocate → n_relocation_decision (contained_in) + opt_stay_put → n_relocation_decision (contained_in) + opt_relocate → n_client_retention_risk (may_cause) +``` + +## Assessment + +### Structural-context admission: PASSED + +The former `result_validation` rejection ("Active unknown violates reasoning pattern consistency") is gone. The model produced `kind=unknown` for the client-retention node, which is compatible with the active decision pattern. No validation errors occurred. + +### Client-retention representation: FIRST-CLASS UNKNOWN + +Node created as `kind=unknown`, `status=unknown`, with explicit label "Will our largest client leave if we relocate?" and description carrying the £5M/year material context. Not text-only, not lost. + +### Reasoning-pattern treatment + +- **Intrinsic/client node pattern:** `unknown` (intrinsic inference yields unknown; compatible with active decision pattern) +- **Active pattern:** `decision` +- **Diagnosis→decision mismatch accepted through bounded context admission:** YES + +The structural action fallback from 60B.19 runs at the pre-mutation proposal boundary for newly-added unresolved unknowns reaching the active decision context. No rejection occurred because kind=unknown is inherently compatible with active pattern "decision". + +### Materiality behaviour: PRESERVED + +- Decision (`n_relocation_decision`) remains unresolved (status=unknown) +- Client-retention issue is the material reason for continued investigation +- No unrelated uncertainty invented +- £5M/year context preserved in node description + +### Client-risk ownership: CLEARLY OWNED BY RELOCATE + +Edge `opt_relocate → n_client_retention_risk` with relationship `may_cause` directly attributes client departure risk to relocation. Node's `dependsOn: ["opt_relocate"]` reinforces this linkage. + +### Preferred-target behaviour + +- **Proposal selectedQuestion.nodeId:** `n_client_retention_risk` +- **Final selectedQuestion.nodeId:** `n_client_retention_risk` (via model-selected nodeId, honored as preferred target per 60B.8+60B.11) + +**Classification: MODEL MATERIAL TARGET PRESERVED** + +The model's selected question targets the exact newly-created client-retention unknown node. The deterministic preference-aware targeting preserves this selection because it is structurally valid (kind=unknown, status=unknown, unresolved). + +### Question text: SPECIFIC TO CLIENT RETENTION + +Text: `"will our largest client leave if we relocate?"` — directly addresses the client-retention uncertainty with no generic framing. + +## Comparison with 60B.12 + +| Field | 60B.12 | 60B.20 | +|-------|--------|--------| +| Validation stage | result_validation (rejected) | All stages passed (200) | +| Proposal applied | NO | YES | +| Client-retention node | never created (rejected before application) | `n_client_retention_risk` (kind=unknown, status=unknown) | +| Final selectedQuestion.nodeId | UNAVAILABLE | `n_client_retention_risk` | +| Final question text | NONE | "will our largest client leave if we relocate?" | + +## Classification: A — FULL LIVE CHAIN CONFIRMED + +All criteria met: + +- [x] Former reasoning-pattern rejection removed +- [x] Proposal applied successfully +- [x] Client-retention factor survives as unresolved unknown +- [x] Decision remains unresolved +- [x] Client risk belongs to Relocate (may_cause edge) +- [x] Final selectedQuestion.nodeId targets the client-retention unknown + +## Critical evidence + +1. **Validation pass:** No result_validation rejection. The structural context admission fix in d871a8c allows newly-added `kind=unknown` nodes reaching the active decision context through the bounded same-turn path. +2. **Materiality preserved:** Decision unknown status unchanged; client-retention node is the material unresolved factor with £5M/year context intact. +3. **Targeting alignment:** Final selected question targets exactly `n_client_retention_risk` — the same node that was just created and is tied to Relocate via may_cause. +4. **No fabrication:** No unrelated uncertainty invented; no over-closure of any existing node. + +## What improved relative to 60B.12 + +- The bounded structural context admission in d871a8c (fix from 60B.19) removed the validation rejection that previously blocked the entire proposal +- The model's kind=unknown output for the client-retention node is now admitted through both intrinsic compatibility and the same-turn fallback path +- Final question targeting correctly aligns with the material factor, enabled by the 60B.8/60B.11 preferred-target mechanism + +## What remains weak or unproven + +- The model still produced `n_client_retention_risk` (not `n_client_retention` as in 60B.6) — naming is stochastically different but functionally equivalent +- Whether the structural context admission also helps if the model produces kind=diagnosis remains untested by this case (this run exercised kind=unknown which was always theoretically compatible) +- No Vitest regression suite rerun against the new boundary; only the live case + +## Production code changed: NO +## Prompt changed during experiment: NO +## Validator changed: NO +## Schema changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls: 1 +## Direct API calls: 0 +## Dev server disturbed: NO