experiment: choose reasoning-pattern inheritance boundary
This commit is contained in:
@@ -2821,3 +2821,7 @@ Experiment 60B.12 reran the exact 60B.6 live case to verify prerequisite-aware t
|
||||
---
|
||||
|
||||
Experiment 60B.13 performed read-only architectural diagnosis of the 60B.12 kind mismatch. **Classification: A + D — Prompt kind ambiguity + missing normalisation path.** Analysis confirmed: (1) `kind=diagnosis` is NOT a valid SituationKind — it exists only as a reasoning pattern and as the default fallback in `selectReasoningPattern`; (2) the validator at line 3927/3998 of apply-proposal.js correctly rejects diagnosis under active decision pattern — semantically, diagnosis and decision are distinct reasoning types; (3) the model likely produced kind=unknown with diagnostic-inferred text analysis, not kind=diagnosis directly (which would fail zod immediately); (4) `hasDecisionContext`'s keyword list (`whether to|build|launch|continue|proceed|invest|commercially justified|viability`) does not include "relocate"/"relocation", so material factors about relocation decisions get inferred as diagnosis; (5) the prompt's kind rules cover decision questions and candidate options but have no rule for material unresolved factors within a decision. Minimum corrective boundary: B — one clarifying rule in Prompt Proposal Rules section stating that new material factors affecting a decision outcome use kind=unknown, with reasoning pattern determined by graph context. Implementation readiness: A.
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.14 performed read-only design analysis on whether a newly-created unresolved factor inside an active decision should inherit the decision's reasoning pattern rather than being classified mainly from its wording. **Classification: D — COMPATIBILITY FALLBACK.** The current architecture separates *active pattern determination* (which DOES use parent-chain traversal via `determineActiveReasoningPattern` and correctly yields "decision" for the 60B.12 node) from *node-intrinsic pattern inference* (which does NOT use that context — it runs standalone text analysis on the node's label/description only). This separation is the root cause: the active pattern correctly walks up to find "decision" in the decision unknown, but the compatibility check re-runs standalone inference on the new node and gets "diagnosis" from its diagnostic-style wording ("will our largest client leave"). The smallest correct fix preserves intrinsic text analysis as primary signal but adds a normalization fallback: when inferred node pattern is incompatible with active pattern AND the node's graph position (parentId, edges) places it structurally within that active context, reinterpret using the active pattern rather than rejecting. This requires no new schema, no new keywords, and preserves genuine pattern transitions (the intrinsic inference still returns diagnosis; only the compatibility decision changes).
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
# Experiment 60B.14 — Reasoning-Pattern Inheritance Boundary
|
||||
|
||||
**Branch:** `feature/question-target-alignment-v0.0.27`
|
||||
**Starting HEAD:** (current)
|
||||
**Date:** 2026-08-13
|
||||
**Status:** COMPLETE (read-only design analysis)
|
||||
**Type:** ARCHITECTURAL DESIGN — Determine where reasoning-pattern ownership should live: node wording or investigation context
|
||||
|
||||
## Objective
|
||||
|
||||
Answer: **When a newly-created unresolved factor is structurally part of an active decision, should its reasoning pattern be inherited from that decision context rather than inferred mainly from its wording?**
|
||||
|
||||
No implementation. Design comparison only.
|
||||
|
||||
## Context Route Summary
|
||||
|
||||
### Source files examined
|
||||
|
||||
| File | Key functions | Lines read |
|
||||
|------|--------------|------------|
|
||||
| `lib/graph/question-formulator.js` | `selectReasoningPattern` | 1030–1101 (72 lines) |
|
||||
| `lib/graph/question-formulator.js` | `hasDecisionContext` | 901–921 (21 lines) |
|
||||
| `lib/graph/question-formulator.js` | `buildParentChain` | 888–899 (12 lines) |
|
||||
| `lib/graph/question-formulator.js` | `collectRelatedNodes` | 25–45 (21 lines) |
|
||||
| `lib/graph/apply-proposal.js` | `determineActiveReasoningPattern` | 1804–1832 (29 lines) |
|
||||
| `lib/graph/apply-proposal.js` | `inferIntrinsicNodePattern` | 1834–1881 (48 lines) |
|
||||
| `lib/graph/apply-proposal.js` | `assessReasoningPatternCompatibility` | 1883–1908 (26 lines) |
|
||||
| `lib/graph/apply-proposal.js` | `ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN` | 1794–1802 (9 lines) |
|
||||
|
||||
### Focused test coverage found
|
||||
|
||||
- **Decision pattern:** `reasoning-pattern-selection.test.js` — definition, comparison scenarios
|
||||
- **Diagnosis pattern:** `reasoning-pattern-validation.test.js` — explanation, definition fixtures
|
||||
- **Parent/inherited pattern:** No dedicated tests for `determineActiveReasoningPattern`. Coverage exists only through integration in apply-proposal tests.
|
||||
- **Active-pattern compatibility:** `apply-proposal.test.js` line 2486 — "does not allow a decision-mode active unknown to remain a comparison child"; no test for new-node-inheritance during proposal application
|
||||
|
||||
---
|
||||
|
||||
## CHECKPOINT 1 — Current Precedence
|
||||
|
||||
### Actual pattern-selection order today
|
||||
|
||||
When a **new unresolved unknown** is created inside an active decision and then validated:
|
||||
|
||||
```
|
||||
1. determineActiveReasoningPattern(newNode, graph)
|
||||
→ walks up parent chain via buildParentChain()
|
||||
→ calls selectReasoningPattern() on each ancestor
|
||||
→ returns first non-"definition" pattern found
|
||||
→ for 60B.12 case: returns "decision" from the decision-unknown ancestor
|
||||
|
||||
2. assessReasoningPatternCompatibility({node, graph, activePattern})
|
||||
→ calls inferIntrinsicNodePattern(node, graph) on the NEW node ONLY
|
||||
(not ancestors — standalone text analysis of label/description)
|
||||
→ checks regex keyword lists against node's own label + description
|
||||
→ falls through to selectReasoningPattern() which uses hasDecisionContext()
|
||||
→ for 60B.12 case: returns "diagnosis" (default fallback)
|
||||
|
||||
3. Compatibility check:
|
||||
→ nodePattern="diagnosis" NOT in ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN["decision"] = ["decision", "definition"]
|
||||
→ incompatible → WHOLE PROPOSAL REJECTED
|
||||
```
|
||||
|
||||
### Answers
|
||||
|
||||
**Does active decision context currently influence the new node's inferred pattern?**
|
||||
NO — `determineActiveReasoningPattern` correctly finds "decision" in ancestor chain, but `inferIntrinsicNodePattern` runs independently without that signal. The compatibility check compares two different things: parent-derived active pattern vs standalone-inferred node pattern.
|
||||
|
||||
**Does option/decision topology influence it?**
|
||||
PARTIAL — Topology is used for `determineActiveReasoningPattern` (parent chain walk) but NOT for `inferIntrinsicNodePattern`. The new node's inferred pattern ignores its structural position entirely.
|
||||
|
||||
**Can wording override/invalidate surrounding context?**
|
||||
YES — The new node's label/description keywords determine its intrinsic pattern independently of surrounding decision context. If the wording lacks decision-context phrases, it defaults to "diagnosis" regardless of being structurally nested inside a decision tree.
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE A — PROMPT FRAMING ONLY
|
||||
|
||||
Keep deterministic inference unchanged. Prompt instructs the model to phrase material decision unknowns with explicit decision-context wording so existing keyword inference returns `decision`.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Semantic robustness:** MEDIUM
|
||||
The prompt can guide phrasing but cannot guarantee it. The model may correctly identify a concept as decision-relevant while choosing natural diagnostic language ("will X happen?") that lacks decision keywords.
|
||||
|
||||
**Dependence on wording:** HIGH
|
||||
Inherits the current architecture's reliance on keyword matching. If the model chooses synonyms not in the keyword list, inference fails again.
|
||||
|
||||
**Provider robustness:** MEDIUM
|
||||
Depends on model following prompt guidance consistently across providers. Some models may prioritize semantic correctness over keyword compliance.
|
||||
|
||||
**New deterministic logic:** NONE
|
||||
|
||||
**Principal risk:** The model treats "will our largest client leave" as a legitimate question phrasing (it is grammatically natural). Forcing decision-context keywords into diagnostic-structure questions produces unnatural text and creates a fragile dependency on the exact keyword list.
|
||||
|
||||
### 60B.12 inferred pattern: diagnosis
|
||||
(The prompt framing changes what the *model writes*, not what the *validator computes*. With the current keyword list, "Will our largest client leave if we relocate?" still lacks keywords.)
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE B — STRUCTURAL DECISION-CONTEXT INHERITANCE
|
||||
|
||||
Rule concept:
|
||||
> If a new unresolved unknown is structurally attached to an option that belongs to an active decision context, treat its reasoning pattern as decision unless there is explicit structural evidence of a genuine pattern transition.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Existing topology sufficient:** PARTIAL
|
||||
The existing `parentId`, edges, and `buildParentChain` provide the necessary graph structure. However, "explicit structural evidence of a genuine pattern transition" has no defined mechanism — what would constitute such evidence without adding new schema or rules?
|
||||
|
||||
**Semantic robustness:** HIGH
|
||||
Structurally attached unknowns in decision trees are almost certainly decision factors by definition of their position.
|
||||
|
||||
**Could mask genuine diagnosis transition:** PARTIAL
|
||||
If the model genuinely needs to shift from decision reasoning to diagnosis reasoning (e.g., discovering an unexpected causal mechanism), this rule would override it unless we define what counts as "explicit structural evidence." No existing transition mechanism covers this.
|
||||
|
||||
**New schema required:** NO — uses existing parentId, edges, and node-kind fields.
|
||||
|
||||
**Principal risk:** Defining the boundary between "genuine pattern transition" and "harmless wording drift" without new schema or keywords requires additional rule expansion that approaches the complexity of Candidate C.
|
||||
|
||||
### 60B.12 inferred pattern: decision
|
||||
(The node's parentId points to an option, whose ancestor is a decision unknown. The rule matches.)
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE C — GENERAL ACTIVE-PATTERN INHERITANCE
|
||||
|
||||
Rule concept:
|
||||
> New unresolved unknown inherits the current active reasoning pattern by default. Intrinsic wording may only change pattern when explicit transition evidence exists.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Semantic robustness:** HIGH
|
||||
Within any active investigation, newly created unknowns are naturally part of that investigation's reasoning mode. The active pattern represents the investigation's current direction.
|
||||
|
||||
**Risk of over-inheritance:** HIGH — This is the critical trade-off. It could mask genuine transitions where a new unknown should start a *different* investigation track (e.g., discovering a regulatory compliance issue inside a commercial decision). Without clear "explicit transition evidence" criteria, everything inherits.
|
||||
|
||||
**Existing transition mechanism sufficient:** NO PARTIAL
|
||||
No existing mechanism defines what counts as "explicit transition evidence." The model's wording would need to be the signal, but Candidate C only allows wording changes with "explicit" evidence — circular without new rules.
|
||||
|
||||
**New schema required:** NO
|
||||
Uses existing active pattern tracking and intrinsic inference.
|
||||
|
||||
**Principal risk:** Over-inheritance creates a reasoning monoculture where all newly created unknowns share one pattern regardless of their actual investigative needs. The architecture currently handles transitions by allowing the model to create nodes with different wording that naturally trigger different patterns — Candidate C would suppress that mechanism.
|
||||
|
||||
### 60B.12 inferred pattern: decision
|
||||
(Inherits active pattern directly.)
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE D — COMPATIBILITY FALLBACK
|
||||
|
||||
Keep intrinsic inference first. If intrinsic pattern is incompatible with active pattern BUT node is structurally embedded in that active context, reinterpret it using the active pattern rather than rejecting.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Semantic robustness:** HIGH
|
||||
Intrinsic text analysis provides the primary signal (preserving genuine transitions where wording strongly indicates a different mode). The fallback only activates when there's BOTH incompatibility AND structural embedding — two independent signals converging on "this is likely a harmless drift, not a real transition."
|
||||
|
||||
**Acts as normalization rather than inference:** YES
|
||||
It preserves the intrinsic inference ("diagnosis") for transparency but normalizes the compatibility decision to "compatible because structurally embedded." The node's pattern label stays "diagnosis" — only the acceptance/rejection changes.
|
||||
|
||||
**Could hide genuine incompatible reasoning:** PARTIAL
|
||||
If wording strongly signals diagnosis (e.g., "what is the root cause?") inside a decision context, this approach would still normalize it. However, the normalization includes both incompatibility AND structural embedding as criteria — requiring BOTH signals to activate reduces false positives significantly compared to simple inheritance.
|
||||
|
||||
**New schema required:** NO
|
||||
Uses existing `inferIntrinsicNodePattern`, `hasDecisionContext`, and graph topology fields (parentId, edges).
|
||||
|
||||
**Principal risk:** The "structural embedding" check for the fallback needs clear definition: what structural relationship qualifies? Using parentId chain (same as `determineActiveReasoningPattern`) is sufficient and already implemented. This is the minimal additional criterion beyond what's already in the compatibility function.
|
||||
|
||||
### 60B.12 inferred pattern: decision
|
||||
(Intrinsic inference returns "diagnosis" but normalization to active context "decision" applies because node is structurally embedded via parentId chain.)
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL DISTINCTION
|
||||
|
||||
> Is reasoning pattern primarily a property of a node's wording, or a property of the investigation context in which that node participates?
|
||||
|
||||
**Answer: HYBRID**
|
||||
|
||||
**Why (based on current architecture):**
|
||||
|
||||
1. **Not purely node-intrinsic:** `selectReasoningPattern` already uses graph-wide context (`collectRelatedNodes`, `hasDecisionContext` with ancestors, central statement). It is not pure text analysis — it combines node text with structural signals. The function itself is hybrid.
|
||||
|
||||
2. **Not purely contextual:** The architecture distinguishes between node pattern (what the specific node needs) and active pattern (the investigation's current mode). Node-level inference must still read intrinsic signals because different nodes within one investigation may legitimately need different patterns (e.g., a definition unknown inside an explanation investigation is explicitly allowed).
|
||||
|
||||
3. **The actual separation:** The problem in 60B.12 arises because `determineActiveReasoningPattern` and `inferIntrinsicNodePattern` operate on *different scopes*:
|
||||
- Active pattern: whole ancestor chain (correctly finds "decision")
|
||||
- Node inference: standalone text analysis only (misses the decision context)
|
||||
|
||||
Both are necessary pieces of a hybrid model. The gap is that the compatibility check doesn't bridge them — it compares parent-derived active against child-derived intrinsic without asking whether structural position explains the mismatch.
|
||||
|
||||
---
|
||||
|
||||
## 60B.12 WALKTHROUGH
|
||||
|
||||
Scenario:
|
||||
```text
|
||||
Active decision context: "Which option leaves us better off overall?"
|
||||
Option: "Relocate" (parent of new unknown)
|
||||
New unresolved factor: "Will our largest client leave if we relocate?" (kind=unknown)
|
||||
Potential consequence: ~£5M/year loss
|
||||
```
|
||||
|
||||
| Candidate | Inferred pattern | Mechanism |
|
||||
|-----------|-----------------|-----------|
|
||||
| A — PROMPT FRAMING | diagnosis | Wording lacks decision keywords; model may still phrase naturally as diagnostic question |
|
||||
| B — STRUCTURAL DECISION INHERITANCE | decision | parentId → option → decision ancestry matches structural rule |
|
||||
| C — GENERAL ACTIVE-PATTERN INHERITANCE | decision | Inherits active pattern directly |
|
||||
| D — COMPATIBILITY FALLBACK | diagnosis (intrinsic) → **decision** (normalized) | Intrinsic text returns diagnosis; structural embedding normalizes to active context |
|
||||
|
||||
---
|
||||
|
||||
## DECISION CRITERIA EVALUATION
|
||||
|
||||
| Criterion | A | B | C | D |
|
||||
|-----------|---|---|---|---|
|
||||
| 1. Prevents valid decision factors rejected due only to wording | PARTIAL (depends on model following prompt) | YES | YES | YES |
|
||||
| 2. Does not require domain-specific keyword expansion | YES | YES | YES | YES |
|
||||
| 3. Preserves genuine reasoning-pattern transitions | YES | PARTIAL (needs "transition evidence" definition) | NO (suppresses all transitions) | PARTIAL (intrinsic signal preserved, compatibility decision changes) |
|
||||
| 4. Uses existing graph structure where possible | YES | YES | YES | YES |
|
||||
| 5. Adds no schema unless unavoidable | YES | YES | YES | YES |
|
||||
| 6. Remains provider-agnostic | YES | YES | PARTIAL (model must follow "explicit transition" rule) | YES |
|
||||
|
||||
---
|
||||
|
||||
## FINAL CHOICE
|
||||
|
||||
### D — COMPATIBILITY FALLBACK
|
||||
|
||||
**Why:**
|
||||
|
||||
1. **Minimal change with maximum coverage.** The compatibility function already computes both active pattern (from parent chain) and node pattern (from text). Adding a normalization step when BOTH conditions hold — incompatible intrinsic pattern AND structural embedding in the active context — fixes 60B.12 without over-correction.
|
||||
|
||||
2. **Preserves genuine transitions.** If a new unknown genuinely signals a different reasoning mode through its wording, `inferIntrinsicNodePattern` still returns that pattern. The proposal is not silently coerced — the intrinsic inference result is preserved for transparency. Only the compatibility decision changes when structural evidence outweighs text-based drift.
|
||||
|
||||
3. **No schema, no keywords, no new rules.** Uses existing fields: `parentId`, edges (for structural embedding), and existing `inferIntrinsicNodePattern`/`selectReasoningPattern` outputs. The only change is in `assessReasoningPatternCompatibility`'s return logic.
|
||||
|
||||
4. **Acts as normalization, not inference.** This is the right level of intervention. We're not saying "this node IS a decision factor" — we're saying "this node's diagnostic-style wording is structurally compatible with its surrounding decision context, so accept it." The distinction matters for debugging and traceability.
|
||||
|
||||
5. **Smallest implementation boundary:** One conditional branch in `assessReasoningPatternCompatibility` (apply-proposal.js line ~1897):
|
||||
```javascript
|
||||
if (!compatible && isStructurallyEmbeddedInActiveContext(node, graph, activePattern)) {
|
||||
return { compatible: true, nodePattern, activePattern,
|
||||
reason: "Node reinterpreted as compatible via structural embedding in active context." };
|
||||
}
|
||||
```
|
||||
|
||||
### Smallest implementation boundary:
|
||||
Single conditional in `assessReasoningPatternCompatibility` using existing graph topology (`buildParentChain` / `hasDecisionContext`) to determine structural embedding. No schema changes. No keyword expansion. No prompt changes.
|
||||
|
||||
---
|
||||
|
||||
## IMPLEMENTATION READINESS
|
||||
|
||||
**A — READY FOR BOUNDED IMPLEMENTATION**
|
||||
|
||||
No unresolved design questions. The "structural embedding" criterion is already defined by the existing parent-chain traversal in `determineActiveReasoningPattern` and `hasDecisionContext`.
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed: NO
|
||||
Validator changed: NO (read-only analysis only — change would be a single conditional)
|
||||
Schema changed: NO
|
||||
Tests changed: NO
|
||||
Ollama calls: 0
|
||||
Live API calls: 0
|
||||
Vitest run: NO
|
||||
Documentation updated: YES
|
||||
|
||||
Git status: clean (documentation commit pending)
|
||||
Reference in New Issue
Block a user