diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 1b45332..14a025b 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2092,3 +2092,45 @@ Full record in `docs/experiment-59b2.md`. Configured Ollama: qwen-claude:latest ### Experiment 59B.4 — Explicit Two-Option Decision Structure (full record in docs/experiment-59b4.md) +### Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Type:** READ-ONLY ARCHITECTURE DIAGNOSIS +**Full record:** `docs/experiment-60a1.md` + +**Context route:** This diagnosis follows experiments 59B.2–59B.4 which showed the persistent pattern: engine understood trade-offs semantically but could not preserve two competing options as independently recoverable structural entities in the graph. + +**Objective:** Answer whether the model fails to use decision structure that already exists, or whether the current graph vocabulary lacks an adequate first-class representation for alternatives and decisions. + +**Methodology:** Read-only analysis of `lib/graph/schema.js`, `lib/graph/prompt-builder.js`, `lib/graph/utils.js`, `lib/graph/apply-proposal.js`, and experiment history. No production code changes, no API calls, no test runs. + +**Key findings from 8-section diagnosis:** + +1. **Vocabulary inventory (Section 1):** SituationKind has 9 node kinds (observation, reported_claim, metric, state, transition, relationship, assumption, unknown, conclusion) — zero kinds represent choices, options, or decisions. SituationRelationship has 10 edge types including `compares_with` but with no alternative-specific semantics. + +2. **Representational adequacy test (Section 2):** The schema cannot express a "decision" node as a first-class entity. The schema cannot express "these two options are alternatives for the same decision." Consequences can attach only if option nodes exist first — a chicken-and-egg problem. The do-nothing baseline has no dedicated representation. + +3. **Concept mapping for 59B.4 (Section 3):** User input had 8 distinct structured components (2 options, 6 consequences, comparison intent). Schema mapped 0 of these as structure. All 8 collapsed into one undifferentiated unknown node's prose description. + +4. **Prompt-vs-schema diagnosis (Section 4):** The schema lacks the primitives AND the prompt lacks rules to use them — but root cause is schema-level. No existing primitive semantically means "an available choice." All four prompt-level reuse candidates (`state`, `compares_with`, `contradicts`, `transition`) fail because no rule references them for alternatives. + +5. **Reuse strategy evaluation (Section 5):** Four candidate strategies tested (`state` as options, `observation` for consequences, repurpose `compares_with`, `transition` for outcomes). All are either semantically incorrect or incomplete without schema-level support. No reuse strategy is viable without adding new primitives. + +6. **Minimum architectural distinction (Section 6):** Three missing primitives identified: DECISION node kind, OPTION node kind, and ALTERNATIVE-OF edge type. These three additions would enable full representation of the 59B.4 decision scenario. Smallest improved graph: ~25 lines total. + +7. **Smallest improved graph (Section 7):** Concrete proposal — add `option` and `decision` to SituationKind, add `alternative_of` and `contains_option` to SituationRelationship, add 4 prompt rules for option creation. No breaking changes. Existing contract unchanged. ~25 lines of code. + +8. **Implementation readiness (Section 8):** Schema additions are low-risk trivial enum extensions. Prompt rule additions are straightforward. Recommendation: next experiment should implement the three-primitive addition and re-run the 59B.4 scenario to verify structural option representation. + +**Classification: DIAGNOSIS COMPLETE — SCHEMA-LEVEL ROOT CAUSE IDENTIFIED** + +The current graph vocabulary lacks first-class primitives for alternatives and decisions. The model cannot represent what the schema does not define. Schema-level additions are necessary; prompt-only changes would not resolve the root cause. + +Production code changed: NO +Prompt changed during experiment: NO +Validator changed during experiment: NO +Vitest run: NO +Ollama calls: 0 +Dev server disturbed: NO +Read-only diagnosis: YES diff --git a/docs/experiment-60a1.md b/docs/experiment-60a1.md new file mode 100644 index 0000000..deb5d00 --- /dev/null +++ b/docs/experiment-60a1.md @@ -0,0 +1,527 @@ +# Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Type:** READ-ONLY ARCHITECTURE DIAGNOSIS — No production code changes, no API calls, no test runs. + +**Following experiments:** 59B.2–59B.4 demonstrated a persistent structural pattern: the engine understood trade-offs semantically but could not preserve two competing options (relocate vs stay-put) as independently recoverable structural entities in the graph. + +## Objective + +Answer this architectural question with evidence from schema, prompt rules, and the apply-proposal orchestrator: + +> Is the model failing to use decision structure that already exists, or does the current graph vocabulary lack an adequate first-class representation for alternatives and decisions? + +This diagnosis is read-only. It analyzes whether the issue is **prompt-level** (the vocabulary exists but rules don't instruct the model) or **schema-level** (the vocabulary itself lacks the concepts needed). + +--- + +## 1. Inventory of Current Graph Vocabulary + +### 1a. Node Kinds (`SituationKind`) + +| Kind | Semantic Domain | First-Class Option/Decision Support? | +|------|----------------|-------------------------------------| +| `observation` | Factual claim about a state of the world | No — asserts existence, not choice | +| `reported_claim` | Third-party assertion | No | +| `metric` | Numerical measure | No | +| `state` | World condition / status | No — describes "what is", not "what could be" | +| `transition` | Change from one state to another | Partial — can describe a change event, but has no option-anchoring semantics | +| `relationship` | Connection between concepts | No | +| `assumption` | Taken-for-granted premise | No | +| `unknown` | Unresolved question / uncertainty | **Partial** — the only node kind that *could* host a decision-related unknown, but has no sub-structure distinguishing "option A vs option B" from "what is X?" | +| `conclusion` | Derived answer to an unknown | No — captures outcome, not process of choosing | + +**Total: 9 distinct node kinds. Zero node kinds have semantics for choices, alternatives, or decision structures.** + +### 1b. Node Statuses (`SituationStatus`) + +| Status | Meaning | Option Relevance | +|--------|---------|-----------------| +| `known` | Established fact | Irrelevant to options | +| `unknown` | Unresolved | Could host "which option?" but has no structure | +| `provisional` | Partially supported | Could be a status for an unconfirmed option | +| `supported` | Evidence-backed claim | No structural option meaning | +| `weakened` | Undermined claim | No option-specific semantics | +| `contradicted` | Conflicts with evidence | Could represent a rejected option (conceptually) | +| `resolved` | Question answered | No option-specific semantics | + +### 1c. Edge Relationships (`SituationRelationship`) + +| Relationship | Meaning | Option-Alternative Support? | +|--------------|---------|---------------------------| +| `supports` | Evidence strengthens a node | No | +| `weakens` | Evidence undermines a node | Could represent negative consequence of an option (if options existed) | +| `contradicts` | Two nodes are mutually exclusive | **Potentially relevant** — mutual exclusivity is related to alternatives, but this expresses contradiction between *claims*, not choice between *options* | +| `depends_on` | One thing requires another | Could express prerequisite relationship within a decision branch | +| `causes` | Direct causal relationship | Could express option → consequence (if options existed as nodes) | +| `may_cause` | Probabilistic causal | Same as above, with uncertainty | +| `measures` | Metric tracks a concept | No | +| `compares_with` | Two things are compared | **Existing but underspecified** — has no documented semantics for mutual-exclusive alternatives; used generically | +| `updates` | One node updates another's value/status | No | +| `other` | Unclassified edge type | No semantic meaning | + +### 1d. Key Finding + +The vocabulary contains the **building blocks** (nodes, edges, statuses) but lacks a **decision-specific primitive**. There is no: + +- **Decision node kind**: No way to represent "the system has a decision to make" as a first-class entity +- **Option/alternative node kind**: No way to represent "relocate" and "stay put" as independently queryable alternatives +- **"Alternative-of" edge relationship**: No way to say "this option belongs to this decision" +- **Explicit mutual-exclusivity semantics**: `compares_with` exists but is semantically underspecified for options + +--- + +## 2. Test: Representational Adequacy for the Relocate vs Stay-Put Decision (Using Only Current Schema) + +### 2a. Scenario Specification + +Using only the schema defined in `lib/graph/schema.js`: + +- User says: "Option 1 is relocate (save £2M/year, lose 2 engineers, delay 2 months). Option 2 is stay put (keep engineers, avoid disruption, continue paying £2M/year)." +- The user's intent: compare these two alternatives to decide which leaves them better off. + +### 2b. Can the schema express a "decision" node? + +**No.** No kind in `SituationKind` semantically means "a decision point requiring choice between alternatives." The closest candidates are: + +| Candidate | Why it's inadequate | +|-----------|-------------------| +| `observation` | An observation asserts what *is*, not what *might be chosen* | +| `state` | A state describes a condition, not a choice about conditions | +| `unknown` | Represents uncertainty about a question, not the alternatives themselves | +| `relationship` | Can link things but cannot contain structured content like "I must choose between A and B" | +| `transition` | Describes a change event, not a decision about which path to take | + +### 2c. Can the schema express "these two options are alternatives for the same decision"? + +**Partially, but with no structural guarantee.** The `compares_with` edge type exists and could theoretically connect two nodes as "comparable." However: + +1. **No defined semantics** for what it means when both endpoints are *options* (as opposed to two observations being compared). +2. **No parent-of-decision relationship**: No way to say "these options belong to this decision node." +3. **No mutual-exclusivity constraint**: `compares_with` does not express that choosing one precludes the other. +4. **Not a structural alternative representation**: Without a rule explicitly instructing the model to use it for alternatives, and without schema-level semantics, the model treats it as a generic "other" bucket. + +### 2d. Can consequences attach to options structurally? + +**Theoretically yes, but only if options exist as nodes first.** If Option A and Option B were both represented as nodes (what kind?), then consequences could attach via `causes` or `may_cause`. But without option nodes, there is nothing for the causal edges to attach to. This is a **chicken-and-egg problem**: you need option nodes before you can represent their consequences structurally. + +### 2e. Can the do-nothing baseline be represented? + +**No dedicated representation exists.** The stay-put alternative in 59B.4 was entirely absent from graph structure because: +- There's no kind for "the current state without any action" +- `state` nodes describe conditions, not baseline alternatives +- Without a decision/option primitive, there's no structural anchor for the baseline + +**Verdict: The schema is structurally inadequate for representing competing alternatives as first-class entities.** + +--- + +## 3. Concept Mapping: What Maps to What in Experiment 59B.4? + +### 3a. User Input Components vs Schema Elements + +| User Concept | Attempted Schema Mapping | Result | +|-------------|------------------------|--------| +| **Option 1 — relocate** | No dedicated kind → forced into `unknown` description prose | Collapsed into single unknown node's text | +| **Option 2 — stay put** | No dedicated kind → lost entirely from graph | Zero structural representation | +| **"I am deciding which..."** (the decision itself) | No kind for decision/choice point | Ignored structurally | +| **£2M/year saving** (relocate benefit) | Could be `metric` or `state`, but no anchor node for the option | Embedded in unknown's description | +| **Two engineers leave** (relocate cost) | Same as above | Embedded in unknown's description | +| **Two-month delay** (relocate cost) | Same as above | Embedded in unknown's description | +| **Keep both engineers** (stay-put benefit) | No anchor node for the option | Lost from graph entirely | +| **Avoid delivery disruption** (stay-put benefit) | Same as above | Lost from graph entirely | +| **Continue paying £2M/year** (stay-put cost) | Same as above | Lost from graph entirely | +| **Two options are alternatives for the same decision** | `compares_with` edge type exists but has no alternative semantics | No edges created between alternatives | + +### 3b. The Core Mapping Failure + +The user's input structure is: + +``` +DECISION (which option?) +├── Option A: relocate +│ ├── Benefit: save £2M/year +│ ├── Cost: lose 2 engineers +│ └── Cost: delay 2 months +└── Option B: stay put + ├── Benefit: keep both engineers + ├── Benefit: avoid delivery disruption + └── Cost: continue paying £2M/year +``` + +The graph schema can represent **none** of the above as structure because it lacks: `DECISION`, `OPTION`, and `ALTERNATIVE-OF` primitives. What the user intended as a **structured decision tree** was forced into the closest available primitive — `unknown` — producing a single undifferentiated node whose description contained both options as prose. + +--- + +## 4. Prompt-vs-Schema Diagnosis: Where Is the Failure? + +### 4a. Testing the "Existing Structure" Hypothesis + +If the problem were **prompt-level** (model fails to use existing vocabulary), we would expect: +- The schema contains a kind/relationship that *could* express alternatives +- The prompt rules instruct the model to use it +- The model ignores the instruction + +Let's check each candidate: + +**Candidate 1: Use `state` for option descriptions** +- Schema allows it ✓ +- Prompt rule instructs it? **No.** No rule references using `state` nodes for "what happens if we choose X" | +- Result: Model doesn't do this (no instruction) + +**Candidate 2: Use `compares_with` edges between options** +- Schema allows it ✓ +- Prompt rule defines semantics for alternatives? **No.** No rule gives `compares_with` alternative-specific meaning. | +- Result: Model treats it generically (same as always) + +**Candidate 3: Use `contradicts` edges between mutually exclusive options** +- Schema allows it ✓ +- But `contradicts` expresses factual contradiction, not choice — using it for options would be semantically wrong +- **No rule instructs its use for alternatives** | +- Result: Not applicable + +**Candidate 4: Use `transition` for option outcomes** +- Schema allows it ✓ (a transition is "change from one state to another") +- But a transition represents an actual change event, not a hypothetical option's consequences +- **No rule instructs its use for options** | +- Result: Not used; no instruction + +### 4b. Testing the "Missing Vocabulary" Hypothesis + +If the problem is **schema-level** (vocabulary lacks needed concepts), we would expect: +- The schema has no kind/relationship that correctly represents alternatives or decisions +- Adding prompt rules without adding schema primitives wouldn't help +- The model produces prose because it's the only remaining option + +This matches our evidence exactly. Every analysis above shows that: +1. No existing kind semantically means "an available choice" or "a decision point" +2. No existing edge type has alternative-specific semantics +3. All four prompt-level candidates fail for the same reason: **no instruction exists** because there is no schema concept to instruct about + +### 4c. The Prompt Rules Analysis (Rule #7 and Others) + +Looking at `prompt-builder.js` rule #7: + +> "Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case." + +This rule says "decision" in the sense of *an unknown about a decision*, not *a decision object containing options*. It's a **quantity constraint** (when to create unknowns), not a **structure instruction** (how to represent options within an unknown). The word "decision" here means "the model should recognize this answer introduces a new decision-related uncertainty," not "model should represent the decision structure itself." + +**No rule in the entire prompt (rules 1–32, plus additional guidance) instructs the model to:** +- Create separate nodes for competing alternatives +- Use any specific node kind for options +- Connect alternatives with a specific edge type +- Represent a do-nothing baseline structurally +- Distinguish "what happens if we choose X" from "what happens if we do nothing" + +### 4d. The Prompt-Builder's Role in the Collapse + +The prompt does instruct the model to produce structural mutation (rule #6, additional guidance), and it *does* do this — but only with the primitives available. Since no primitive exists for alternatives, the model: +1. Identifies a relevant unknown ("uncertainty about operational impact") +2. Creates it as a single `unknown` node +3. Embeds both options in its description prose +4. Does not (and cannot) create option structure because none exists + +**Verdict: Both — the schema lacks the primitives AND the prompt lacks the rules to use them.** But the root cause is schema-level; adding prompt rules without schema changes would produce inconsistent results (the model might guess which primitive to repurpose, inconsistently). + +--- + +## 5. Reuse Strategy Evaluation: Could Existing Primitives Be Repurposed? + +### 5a. Strategy: Treat `state` nodes as option descriptions + +**Mechanism:** Model creates `state` nodes for "relocate state" and "stay-put state," linked by `compares_with`. + +**Pros:** +- Schema allows it (no validation error) +- Minimal schema change needed + +**Cons:** +- `state` semantically means "a condition that holds true." Options are *conditional futures*, not actual states. This is a category error. +- Prompt rules have no guidance for this repurposing. +- Future reasoning about these nodes would treat them as known facts, not hypotheticals. +- The do-nothing baseline (`state`) would be indistinguishable from an active option's outcome state. + +**Verdict: Semantically incorrect. Would cause reasoning errors downstream.** + +### 5b. Strategy: Treat `observation` nodes for option consequences + +**Mechanism:** Each consequence (save £2M, lose engineers) becomes its own `observation` node attached to the option via `supports`. + +**Pros:** +- Schema allows it +- `supports` edges are well-defined + +**Cons:** +- The *option* itself still has no structural representation. +- Observations assert what *is*, not what *would be if chosen*. +- Without a parent option node, consequences float without context. +- No way to say "these observations all belong to Option A." + +**Verdict: Incomplete. Captures consequences but not the option structure that binds them.** + +### 5c. Strategy: Add `compares_with` semantics for alternatives + +**Mechanism:** Define `compares_with` edge type as "these two nodes represent competing alternatives for the same decision," and add a prompt rule instructing the model to use it. + +**Pros:** +- Schema already has the edge type (no schema change needed) +- If semantic definition is clear, the model can follow an explicit instruction + +**Cons:** +- `compares_with` semantically should mean "these two things share comparable properties" not "these are mutually exclusive options for one decision." These are fundamentally different concepts. +- Risk of edge-type confusion when the same relationship type is used for both comparison and alternatives. +- Still doesn't solve the missing **decision node** or **option node** problem — you'd have standalone option nodes without a parent decision context. + +**Verdict: Partially viable as an interim solution, but semantically contaminated. Better to add dedicated types.** + +### 5d. Strategy: Use `unknown` sub-structure via metadata (not supported) + +**Mechanism:** Add a `decision_type` or `option_category` field to the existing `SituationNode` schema. + +**Cons:** +- Requires schema change (adds a field) +- Still doesn't solve "how does the model know when to create option nodes vs standard unknowns?" +- Adds complexity to an already dense node schema. + +**Verdict: Fragile. Requires both schema and prompt changes with uncertain ROI.** + +### 5e. Strategy: Use `transition` for option outcomes + +**Mechanism:** A `transition` node represents "what happens if we choose X" (a change from baseline). + +**Pros:** +- Semantically closer than `state` — a transition *is* a change, and choosing an option causes a change. +- Schema already has the kind. + +**Cons:** +- `transition` semantically means "a change event that occurred or is occurring," not "a hypothetical future state contingent on a choice." +- The do-nothing baseline has no transition (it's stasis), so it would still lack structural representation. +- Same problem as above: no parent decision node to group transitions under. + +**Verdict: Conceptually closer than `state`, but still a category error for hypothetical option outcomes.** + +### 5f. Summary of Reuse Strategies + +| Strategy | Schema Change Needed | Semantic Fit | Prompt Rule Needed | Viability | +|----------|-------------------|-------------|-------------------|-----------| +| `state` as options | No | Poor (assertion vs hypothesis) | Yes | ❌ Not viable | +| `observation` for consequences | Partial (need option nodes) | Poor (is vs would-be) | Yes | ❌ Incomplete | +| Repurpose `compares_with` | No | Contaminated (comparison ≠ alternatives) | Yes | ⚠️ Interim only | +| Add `decision_type` field to nodes | Yes | N/A (structural fix on existing type) | Maybe | ⚠️ Fragile | +| `transition` for outcomes | No | Moderate (change event) | Yes | ⚠️ Partial | + +**None of the reuse strategies are satisfactory without schema-level changes.** All either represent category errors or produce incomplete structures that lose information. + +--- + +## 6. Minimum Architectural Distinction: What Exactly Is Missing? + +### 6a. The Structural Gap as a Hierarchy of Primitives + +To properly represent the 59B.4 decision scenario, the graph needs (from most general to most specific): + +``` +1. DECISION — "there is a choice to make here" (parent context) + ↓ (contains) +2. OPTION — "one available path within this decision" (branch entity) + ↓ (has consequence) +3. CONSEQUENCE — "an outcome of choosing this option" (leaf detail) + +Plus: +4. ALTERNATIVE-OF — "these two options compete for the same decision" (option ↔ option relationship) +5. BASELINE — "the state if no option is chosen" (implicit default option) +``` + +Currently available in schema: +``` +❌ DECISION — none exists +❌ OPTION — none exists +✓ CONSEQUENCE — can use `state` or `observation` (semantically imperfect but usable) +❌ ALTERNATIVE-OF — no dedicated edge type +❌ BASELINE — no dedicated primitive +``` + +### 6b. The Minimum Viable Addition + +To solve the 59B.4 pattern, the graph needs at minimum: + +1. **A new node kind `option`** (or `alternative`) that represents "a choice available within a decision context." +2. **A new edge relationship `alternative_of`** (or belongs_to) that says "this option is one of the choices for this decision." +3. **A prompt rule** instructing the model to create option nodes when the answer explicitly presents competing alternatives. + +That's it — three additions. Everything else (consequences, comparisons, baseline) can be built on top of these primitives with existing edge types. + +### 6c. Smallest Improved Graph for 59B.4 (Conceptual) + +With the three new primitives above, the 59B.4 answer would produce: + +``` +[DECISION: which option leaves us better off overall?] +├── [OPTION A: relocate] — alternative_of → DECISION +│ ├── [CONSEQUENCE: save £2M/year] — may_cause → OPTION A +│ ├── [CONSEQUENCE: lose 2 senior engineers] — may_cause → OPTION A +│ └── [CONSEQUENCE: delay up to 2 months] — may_cause → OPTION A +├── [OPTION B: stay put] — alternative_of → DECISION +│ ├── [CONSEQUENCE: retain both engineers] — causes → OPTION B +│ ├── [CONSEQUENCE: avoid delivery disruption] — causes → OPTION B +│ └── [CONSEQUENCE: continue paying £2M/year] — may_cause → OPTION B +└── [ALTERNATIVE-OF between OPTION A and OPTION B] +``` + +Without those primitives, the best the current schema can do is the 59B.4 result: a single unknown node whose description text contains everything. + +--- + +## 7. Smallest Improved Graph Schema (Concrete Proposal) + +### 7a. New Enum Values + +In `SituationKind`: +```javascript +option: "option" // A choice available within a decision context +decision: "decision" // A decision point requiring selection among alternatives +``` + +In `SituationRelationship`: +```javascript +alternative_of: "alternative_of" // This option is one of the choices for a decision +contains_option: "contains_option" // This decision contains this option +``` + +### 7b. Minimal Node Additions to `situationNodeSchema` (Optional) + +Option nodes could carry an additional field: +```javascript +is_baseline: z.boolean().optional() // Is this the do-nothing / current-state default? +``` + +This is optional — the semantics can be conveyed through `description` text if preferred. + +### 7c. Prompt Rule Additions Needed (One Sentence Each) + +1. "When the answer presents two or more competing alternatives for a decision, create one node of kind 'option' for each alternative." +2. "Connect each option node to its parent decision node using the relationship 'contains_option'." +3. "Connect competing option nodes to each other using the relationship 'alternative_of'." +4. "If the answer explicitly or implicitly references a do-nothing baseline, represent it as an option node with is_baseline = true." + +### 7d. What Changes in the Schema Files + +| File | Change | Lines Affected | +|------|--------|---------------| +| `schema.js` — SituationKind enum | Add `option` and `decision` | ~5 lines | +| `schema.js` — SituationRelationship enum | Add `alternative_of` and `contains_option` | ~3 lines | +| `schema.js` — situationNodeSchema | Add optional `is_baseline` to option nodes | ~2 lines (optional) | +| `prompt-builder.js` — rules | Add 4 new rules or extend existing rules | ~15 lines | +| `utils.js` — applyGraphUpdate | No changes needed (new kinds are just more enum values) | 0 | +| `apply-proposal.js` — validation | No mandatory changes; pattern compatibility logic may optionally extend to support decision reasoning patterns | 0 | + +**Total: ~25 lines of schema + prompt changes.** + +### 7e. What Does NOT Change + +- Existing node kinds, statuses, and edge types remain unchanged. +- The GraphUpdate contract (addedNodes, updatedNodes, etc.) remains unchanged. +- No existing nodes need migration or restructuring. +- No propagation logic needs modification (the decision/option structure sits at the same level as the existing unknown hierarchy). + +--- + +## 8. Implementation Readiness Assessment + +### 8a. What Has Already Been Established by Prior Experiments + +| Finding | Experiment | Implication for 60A.1 | +|---------|-----------|----------------------| +| Explicit dual-option framing does not produce structural alternatives | 59B.4 | Confirms the vocabulary gap is active, not theoretical | +| Do-nothing baseline remains invisible as structure | 59B.3 | Baseline needs explicit representation, not implicit inference | +| Engine extracts facts from structured prose correctly | 59B.4 (what it did right) | Model can extract option details; the gap is structural anchoring | +| No resolution of `savings_realism` despite confirmatory language | 59B.4 | Resolution logic needs to recognize option-based confirmation patterns | +| Single unknown node with multi-option description prose | 59B.2, 59B.3, 59B.4 | Confirmed persistent pattern across multiple inputs | + +### 8b. What Is Ready to Implement (Low Risk) + +1. **Schema additions** (`option`, `decision` kinds; `alternative_of`, `contains_option` edges): Trivial — new enum values. No breaking changes. No validation logic changes needed (new enum values are valid per Zod). +2. **Prompt rule additions**: Straightforward — 4–5 sentences of explicit instruction for the model. Low risk, high clarity. +3. **Deterministic decomposition templates** for option-based decisions: Can be added to `buildDecompositionTemplates()` in `apply-proposal.js` when a decision node is active. + +### 8c. What Would Benefit from a Follow-Up Experiment (Medium Risk) + +1. **Option-based reasoning pattern**: The current reasoning pattern system (`decision`, `explanation`, `contradiction`, etc.) could benefit from an explicit `option_comparison` pattern that governs how the engine reasons over option nodes. +2. **Consequence propagation through option structure**: How should resolving one option's unknown propagate? If we resolve "the two engineers won't leave" for Option A, does that affect the comparison with Option B? This needs design. +3. **Baseline visibility in reasoning patterns**: The `prioritisation` pattern could be extended to explicitly consider do-nothing baselines when an active reasoning context involves decisions. + +### 8d. What Is Not Ready / Needs More Investigation (High Risk) + +1. **Interim strategy via `compares_with` repurposing**: While technically possible, giving it alternative semantics risks confusion with the comparison semantics that already exist for observation comparisons (e.g., "compare two months' sales data"). A dedicated edge type is strongly preferred. +2. **Automatic baseline detection**: Whether the model can infer a do-nothing baseline without explicit user framing needs testing. Some answers imply it; others don't. The prompt would need precise trigger conditions. +3. **Multi-option decisions (> 2 options)**: How should N competing alternatives be represented? Linear chains of `alternative_of` edges, or a star topology centered on the decision node? This needs design. + +### 8e. Recommended Next Experiment + +**Experiment 60A.2 (proposed): Implement the three-primitive addition and verify 59B.4 structure.** + +- Add `option`, `decision` kinds; `alternative_of`, `contains_option` edges to schema. +- Add 4 prompt rules for option creation. +- Run 59B.4 scenario against updated engine. +- Measure: do both options appear as structural entities? Can the do-nothing baseline be represented? Is the comparison question structurally grounded? + +--- + +## Diagnosis Conclusion + +### The Answer to the Research Question + +**The current graph vocabulary lacks an adequate first-class representation for alternatives and decisions.** It is not primarily a prompt problem — it is a schema problem. The model cannot represent what the schema does not define. Without `option` or `decision` node kinds and without `alternative_of` edge semantics, any dual-option input will always collapse into undifferentiated unknown prose. + +### Evidence Chain + +1. **Schema analysis**: Zero node kinds in SituationKind semantically represent choices or alternatives. The 9 available kinds cover observations, states, metrics, relationships, assumptions, unknowns, and conclusions — but no "option" or "decision." +2. **Edge analysis**: `compares_with` exists but has no alternative-specific semantics. No dedicated "this option is one of the choices for this decision" relationship type exists. +3. **Prompt analysis**: None of the 32 rules instruct the model to create structural options. The word "decision" in rule #7 refers to *a question about a decision*, not *a structural representation of the decision*. +4. **Empirical evidence**: Experiments 59B.2–59B.4 consistently showed the same pattern — explicit dual-option input collapsed into a single undifferentiated unknown node, regardless of how strongly the user framed the comparison. +5. **Reuse analysis**: All four candidate reuse strategies (`state`, `observation`, repurposed `compares_with`, `transition`) are either semantically incorrect or incomplete without schema-level support. + +### Why Adding Prompt Rules Without Schema Changes Would Not Help + +The model follows instructions precisely. If no instruction references a concept that doesn't exist in the vocabulary, the model cannot invent it. Telling the model to "create option nodes" without a valid `kind` value would cause validation errors. Repurposing existing kinds requires both schema changes (new enum values) and prompt rules anyway — so the schema change is unavoidable regardless of approach. + +### Recommendation + +Add two node kinds (`option`, `decision`) and two edge relationships (`alternative_of`, `contains_option`) to the schema. Add four prompt rules for option creation. Total impact: ~25 lines of code. This addresses the root cause rather than treating symptoms. + +--- + +## Summary Tables + +### Key Findings Matrix + +| Finding | Evidence | Confidence | +|---------|----------|-----------| +| Schema lacks option/decision primitives | schema.js SituationKind has 9 kinds, none for options | HIGH — direct code analysis | +| `compares_with` has no alternative semantics | No documented semantics; used generically | HIGH — code + experiment history | +| Prompt has no option creation rules | prompt-builder.js rules 1–32, no mention of options | HIGH — direct code analysis | +| Empirical pattern persists across experiments | 59B.2, 59B.3, 59B.4 all same failure mode | HIGH — observed results | +| Schema-only fix is ~25 lines | Two kinds + two relationships + four rules | HIGH — direct enumeration | + +### Vocabulary Gap Summary + +| Needed Primitive | Exists? | If not: What to Add | +|-----------------|---------|-------------------| +| Decision point representation | ❌ No | `decision` node kind | +| Option / alternative entity | ❌ No | `option` node kind | +| "This option belongs to this decision" link | ❌ No | `contains_option` edge type | +| "These options compete" link | ❌ No (partial: `compares_with` exists but wrong semantics) | `alternative_of` edge type | +| Do-nothing baseline representation | ❌ No | `is_baseline` flag on option nodes | +| Option consequence attachment | ⚠️ Partially (via existing edges, if options existed) | N/A (works once options exist) | + +--- + +Production code changed: NO +Prompt changed during experiment: NO +Validator changed during experiment: NO +Vitest run: NO +Ollama calls: 0 +Dev server disturbed: NO +Read-only diagnosis: YES