exp 60A.1: read-only vocabulary adequacy diagnosis for alternatives and decisions
Diagnoses the root cause of the persistent pattern from 59B.2-59B.4 where explicit dual-option input collapsed into a single undifferentiated unknown node. Concludes the graph vocabulary lacks first-class primitives for options/decisions (not primarily a prompt issue). Identifies three missing primitives: option node kind, decision node kind, alternative_of edge type. Recommends ~25-line schema addition for 60A.2 implementation.
This commit is contained in:
@@ -2092,3 +2092,45 @@ Full record in `docs/experiment-59b2.md`. Configured Ollama: qwen-claude:latest
|
||||
|
||||
### Experiment 59B.4 — Explicit Two-Option Decision Structure (full record in docs/experiment-59b4.md)
|
||||
|
||||
### Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Type:** READ-ONLY ARCHITECTURE DIAGNOSIS
|
||||
**Full record:** `docs/experiment-60a1.md`
|
||||
|
||||
**Context route:** This diagnosis follows experiments 59B.2–59B.4 which showed the persistent pattern: engine understood trade-offs semantically but could not preserve two competing options as independently recoverable structural entities in the graph.
|
||||
|
||||
**Objective:** Answer whether the model fails to use decision structure that already exists, or whether the current graph vocabulary lacks an adequate first-class representation for alternatives and decisions.
|
||||
|
||||
**Methodology:** Read-only analysis of `lib/graph/schema.js`, `lib/graph/prompt-builder.js`, `lib/graph/utils.js`, `lib/graph/apply-proposal.js`, and experiment history. No production code changes, no API calls, no test runs.
|
||||
|
||||
**Key findings from 8-section diagnosis:**
|
||||
|
||||
1. **Vocabulary inventory (Section 1):** SituationKind has 9 node kinds (observation, reported_claim, metric, state, transition, relationship, assumption, unknown, conclusion) — zero kinds represent choices, options, or decisions. SituationRelationship has 10 edge types including `compares_with` but with no alternative-specific semantics.
|
||||
|
||||
2. **Representational adequacy test (Section 2):** The schema cannot express a "decision" node as a first-class entity. The schema cannot express "these two options are alternatives for the same decision." Consequences can attach only if option nodes exist first — a chicken-and-egg problem. The do-nothing baseline has no dedicated representation.
|
||||
|
||||
3. **Concept mapping for 59B.4 (Section 3):** User input had 8 distinct structured components (2 options, 6 consequences, comparison intent). Schema mapped 0 of these as structure. All 8 collapsed into one undifferentiated unknown node's prose description.
|
||||
|
||||
4. **Prompt-vs-schema diagnosis (Section 4):** The schema lacks the primitives AND the prompt lacks rules to use them — but root cause is schema-level. No existing primitive semantically means "an available choice." All four prompt-level reuse candidates (`state`, `compares_with`, `contradicts`, `transition`) fail because no rule references them for alternatives.
|
||||
|
||||
5. **Reuse strategy evaluation (Section 5):** Four candidate strategies tested (`state` as options, `observation` for consequences, repurpose `compares_with`, `transition` for outcomes). All are either semantically incorrect or incomplete without schema-level support. No reuse strategy is viable without adding new primitives.
|
||||
|
||||
6. **Minimum architectural distinction (Section 6):** Three missing primitives identified: DECISION node kind, OPTION node kind, and ALTERNATIVE-OF edge type. These three additions would enable full representation of the 59B.4 decision scenario. Smallest improved graph: ~25 lines total.
|
||||
|
||||
7. **Smallest improved graph (Section 7):** Concrete proposal — add `option` and `decision` to SituationKind, add `alternative_of` and `contains_option` to SituationRelationship, add 4 prompt rules for option creation. No breaking changes. Existing contract unchanged. ~25 lines of code.
|
||||
|
||||
8. **Implementation readiness (Section 8):** Schema additions are low-risk trivial enum extensions. Prompt rule additions are straightforward. Recommendation: next experiment should implement the three-primitive addition and re-run the 59B.4 scenario to verify structural option representation.
|
||||
|
||||
**Classification: DIAGNOSIS COMPLETE — SCHEMA-LEVEL ROOT CAUSE IDENTIFIED**
|
||||
|
||||
The current graph vocabulary lacks first-class primitives for alternatives and decisions. The model cannot represent what the schema does not define. Schema-level additions are necessary; prompt-only changes would not resolve the root cause.
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed during experiment: NO
|
||||
Validator changed during experiment: NO
|
||||
Vitest run: NO
|
||||
Ollama calls: 0
|
||||
Dev server disturbed: NO
|
||||
Read-only diagnosis: YES
|
||||
|
||||
@@ -0,0 +1,527 @@
|
||||
# Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Type:** READ-ONLY ARCHITECTURE DIAGNOSIS — No production code changes, no API calls, no test runs.
|
||||
|
||||
**Following experiments:** 59B.2–59B.4 demonstrated a persistent structural pattern: the engine understood trade-offs semantically but could not preserve two competing options (relocate vs stay-put) as independently recoverable structural entities in the graph.
|
||||
|
||||
## Objective
|
||||
|
||||
Answer this architectural question with evidence from schema, prompt rules, and the apply-proposal orchestrator:
|
||||
|
||||
> Is the model failing to use decision structure that already exists, or does the current graph vocabulary lack an adequate first-class representation for alternatives and decisions?
|
||||
|
||||
This diagnosis is read-only. It analyzes whether the issue is **prompt-level** (the vocabulary exists but rules don't instruct the model) or **schema-level** (the vocabulary itself lacks the concepts needed).
|
||||
|
||||
---
|
||||
|
||||
## 1. Inventory of Current Graph Vocabulary
|
||||
|
||||
### 1a. Node Kinds (`SituationKind`)
|
||||
|
||||
| Kind | Semantic Domain | First-Class Option/Decision Support? |
|
||||
|------|----------------|-------------------------------------|
|
||||
| `observation` | Factual claim about a state of the world | No — asserts existence, not choice |
|
||||
| `reported_claim` | Third-party assertion | No |
|
||||
| `metric` | Numerical measure | No |
|
||||
| `state` | World condition / status | No — describes "what is", not "what could be" |
|
||||
| `transition` | Change from one state to another | Partial — can describe a change event, but has no option-anchoring semantics |
|
||||
| `relationship` | Connection between concepts | No |
|
||||
| `assumption` | Taken-for-granted premise | No |
|
||||
| `unknown` | Unresolved question / uncertainty | **Partial** — the only node kind that *could* host a decision-related unknown, but has no sub-structure distinguishing "option A vs option B" from "what is X?" |
|
||||
| `conclusion` | Derived answer to an unknown | No — captures outcome, not process of choosing |
|
||||
|
||||
**Total: 9 distinct node kinds. Zero node kinds have semantics for choices, alternatives, or decision structures.**
|
||||
|
||||
### 1b. Node Statuses (`SituationStatus`)
|
||||
|
||||
| Status | Meaning | Option Relevance |
|
||||
|--------|---------|-----------------|
|
||||
| `known` | Established fact | Irrelevant to options |
|
||||
| `unknown` | Unresolved | Could host "which option?" but has no structure |
|
||||
| `provisional` | Partially supported | Could be a status for an unconfirmed option |
|
||||
| `supported` | Evidence-backed claim | No structural option meaning |
|
||||
| `weakened` | Undermined claim | No option-specific semantics |
|
||||
| `contradicted` | Conflicts with evidence | Could represent a rejected option (conceptually) |
|
||||
| `resolved` | Question answered | No option-specific semantics |
|
||||
|
||||
### 1c. Edge Relationships (`SituationRelationship`)
|
||||
|
||||
| Relationship | Meaning | Option-Alternative Support? |
|
||||
|--------------|---------|---------------------------|
|
||||
| `supports` | Evidence strengthens a node | No |
|
||||
| `weakens` | Evidence undermines a node | Could represent negative consequence of an option (if options existed) |
|
||||
| `contradicts` | Two nodes are mutually exclusive | **Potentially relevant** — mutual exclusivity is related to alternatives, but this expresses contradiction between *claims*, not choice between *options* |
|
||||
| `depends_on` | One thing requires another | Could express prerequisite relationship within a decision branch |
|
||||
| `causes` | Direct causal relationship | Could express option → consequence (if options existed as nodes) |
|
||||
| `may_cause` | Probabilistic causal | Same as above, with uncertainty |
|
||||
| `measures` | Metric tracks a concept | No |
|
||||
| `compares_with` | Two things are compared | **Existing but underspecified** — has no documented semantics for mutual-exclusive alternatives; used generically |
|
||||
| `updates` | One node updates another's value/status | No |
|
||||
| `other` | Unclassified edge type | No semantic meaning |
|
||||
|
||||
### 1d. Key Finding
|
||||
|
||||
The vocabulary contains the **building blocks** (nodes, edges, statuses) but lacks a **decision-specific primitive**. There is no:
|
||||
|
||||
- **Decision node kind**: No way to represent "the system has a decision to make" as a first-class entity
|
||||
- **Option/alternative node kind**: No way to represent "relocate" and "stay put" as independently queryable alternatives
|
||||
- **"Alternative-of" edge relationship**: No way to say "this option belongs to this decision"
|
||||
- **Explicit mutual-exclusivity semantics**: `compares_with` exists but is semantically underspecified for options
|
||||
|
||||
---
|
||||
|
||||
## 2. Test: Representational Adequacy for the Relocate vs Stay-Put Decision (Using Only Current Schema)
|
||||
|
||||
### 2a. Scenario Specification
|
||||
|
||||
Using only the schema defined in `lib/graph/schema.js`:
|
||||
|
||||
- User says: "Option 1 is relocate (save £2M/year, lose 2 engineers, delay 2 months). Option 2 is stay put (keep engineers, avoid disruption, continue paying £2M/year)."
|
||||
- The user's intent: compare these two alternatives to decide which leaves them better off.
|
||||
|
||||
### 2b. Can the schema express a "decision" node?
|
||||
|
||||
**No.** No kind in `SituationKind` semantically means "a decision point requiring choice between alternatives." The closest candidates are:
|
||||
|
||||
| Candidate | Why it's inadequate |
|
||||
|-----------|-------------------|
|
||||
| `observation` | An observation asserts what *is*, not what *might be chosen* |
|
||||
| `state` | A state describes a condition, not a choice about conditions |
|
||||
| `unknown` | Represents uncertainty about a question, not the alternatives themselves |
|
||||
| `relationship` | Can link things but cannot contain structured content like "I must choose between A and B" |
|
||||
| `transition` | Describes a change event, not a decision about which path to take |
|
||||
|
||||
### 2c. Can the schema express "these two options are alternatives for the same decision"?
|
||||
|
||||
**Partially, but with no structural guarantee.** The `compares_with` edge type exists and could theoretically connect two nodes as "comparable." However:
|
||||
|
||||
1. **No defined semantics** for what it means when both endpoints are *options* (as opposed to two observations being compared).
|
||||
2. **No parent-of-decision relationship**: No way to say "these options belong to this decision node."
|
||||
3. **No mutual-exclusivity constraint**: `compares_with` does not express that choosing one precludes the other.
|
||||
4. **Not a structural alternative representation**: Without a rule explicitly instructing the model to use it for alternatives, and without schema-level semantics, the model treats it as a generic "other" bucket.
|
||||
|
||||
### 2d. Can consequences attach to options structurally?
|
||||
|
||||
**Theoretically yes, but only if options exist as nodes first.** If Option A and Option B were both represented as nodes (what kind?), then consequences could attach via `causes` or `may_cause`. But without option nodes, there is nothing for the causal edges to attach to. This is a **chicken-and-egg problem**: you need option nodes before you can represent their consequences structurally.
|
||||
|
||||
### 2e. Can the do-nothing baseline be represented?
|
||||
|
||||
**No dedicated representation exists.** The stay-put alternative in 59B.4 was entirely absent from graph structure because:
|
||||
- There's no kind for "the current state without any action"
|
||||
- `state` nodes describe conditions, not baseline alternatives
|
||||
- Without a decision/option primitive, there's no structural anchor for the baseline
|
||||
|
||||
**Verdict: The schema is structurally inadequate for representing competing alternatives as first-class entities.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Concept Mapping: What Maps to What in Experiment 59B.4?
|
||||
|
||||
### 3a. User Input Components vs Schema Elements
|
||||
|
||||
| User Concept | Attempted Schema Mapping | Result |
|
||||
|-------------|------------------------|--------|
|
||||
| **Option 1 — relocate** | No dedicated kind → forced into `unknown` description prose | Collapsed into single unknown node's text |
|
||||
| **Option 2 — stay put** | No dedicated kind → lost entirely from graph | Zero structural representation |
|
||||
| **"I am deciding which..."** (the decision itself) | No kind for decision/choice point | Ignored structurally |
|
||||
| **£2M/year saving** (relocate benefit) | Could be `metric` or `state`, but no anchor node for the option | Embedded in unknown's description |
|
||||
| **Two engineers leave** (relocate cost) | Same as above | Embedded in unknown's description |
|
||||
| **Two-month delay** (relocate cost) | Same as above | Embedded in unknown's description |
|
||||
| **Keep both engineers** (stay-put benefit) | No anchor node for the option | Lost from graph entirely |
|
||||
| **Avoid delivery disruption** (stay-put benefit) | Same as above | Lost from graph entirely |
|
||||
| **Continue paying £2M/year** (stay-put cost) | Same as above | Lost from graph entirely |
|
||||
| **Two options are alternatives for the same decision** | `compares_with` edge type exists but has no alternative semantics | No edges created between alternatives |
|
||||
|
||||
### 3b. The Core Mapping Failure
|
||||
|
||||
The user's input structure is:
|
||||
|
||||
```
|
||||
DECISION (which option?)
|
||||
├── Option A: relocate
|
||||
│ ├── Benefit: save £2M/year
|
||||
│ ├── Cost: lose 2 engineers
|
||||
│ └── Cost: delay 2 months
|
||||
└── Option B: stay put
|
||||
├── Benefit: keep both engineers
|
||||
├── Benefit: avoid delivery disruption
|
||||
└── Cost: continue paying £2M/year
|
||||
```
|
||||
|
||||
The graph schema can represent **none** of the above as structure because it lacks: `DECISION`, `OPTION`, and `ALTERNATIVE-OF` primitives. What the user intended as a **structured decision tree** was forced into the closest available primitive — `unknown` — producing a single undifferentiated node whose description contained both options as prose.
|
||||
|
||||
---
|
||||
|
||||
## 4. Prompt-vs-Schema Diagnosis: Where Is the Failure?
|
||||
|
||||
### 4a. Testing the "Existing Structure" Hypothesis
|
||||
|
||||
If the problem were **prompt-level** (model fails to use existing vocabulary), we would expect:
|
||||
- The schema contains a kind/relationship that *could* express alternatives
|
||||
- The prompt rules instruct the model to use it
|
||||
- The model ignores the instruction
|
||||
|
||||
Let's check each candidate:
|
||||
|
||||
**Candidate 1: Use `state` for option descriptions**
|
||||
- Schema allows it ✓
|
||||
- Prompt rule instructs it? **No.** No rule references using `state` nodes for "what happens if we choose X" |
|
||||
- Result: Model doesn't do this (no instruction)
|
||||
|
||||
**Candidate 2: Use `compares_with` edges between options**
|
||||
- Schema allows it ✓
|
||||
- Prompt rule defines semantics for alternatives? **No.** No rule gives `compares_with` alternative-specific meaning. |
|
||||
- Result: Model treats it generically (same as always)
|
||||
|
||||
**Candidate 3: Use `contradicts` edges between mutually exclusive options**
|
||||
- Schema allows it ✓
|
||||
- But `contradicts` expresses factual contradiction, not choice — using it for options would be semantically wrong
|
||||
- **No rule instructs its use for alternatives** |
|
||||
- Result: Not applicable
|
||||
|
||||
**Candidate 4: Use `transition` for option outcomes**
|
||||
- Schema allows it ✓ (a transition is "change from one state to another")
|
||||
- But a transition represents an actual change event, not a hypothetical option's consequences
|
||||
- **No rule instructs its use for options** |
|
||||
- Result: Not used; no instruction
|
||||
|
||||
### 4b. Testing the "Missing Vocabulary" Hypothesis
|
||||
|
||||
If the problem is **schema-level** (vocabulary lacks needed concepts), we would expect:
|
||||
- The schema has no kind/relationship that correctly represents alternatives or decisions
|
||||
- Adding prompt rules without adding schema primitives wouldn't help
|
||||
- The model produces prose because it's the only remaining option
|
||||
|
||||
This matches our evidence exactly. Every analysis above shows that:
|
||||
1. No existing kind semantically means "an available choice" or "a decision point"
|
||||
2. No existing edge type has alternative-specific semantics
|
||||
3. All four prompt-level candidates fail for the same reason: **no instruction exists** because there is no schema concept to instruct about
|
||||
|
||||
### 4c. The Prompt Rules Analysis (Rule #7 and Others)
|
||||
|
||||
Looking at `prompt-builder.js` rule #7:
|
||||
|
||||
> "Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case."
|
||||
|
||||
This rule says "decision" in the sense of *an unknown about a decision*, not *a decision object containing options*. It's a **quantity constraint** (when to create unknowns), not a **structure instruction** (how to represent options within an unknown). The word "decision" here means "the model should recognize this answer introduces a new decision-related uncertainty," not "model should represent the decision structure itself."
|
||||
|
||||
**No rule in the entire prompt (rules 1–32, plus additional guidance) instructs the model to:**
|
||||
- Create separate nodes for competing alternatives
|
||||
- Use any specific node kind for options
|
||||
- Connect alternatives with a specific edge type
|
||||
- Represent a do-nothing baseline structurally
|
||||
- Distinguish "what happens if we choose X" from "what happens if we do nothing"
|
||||
|
||||
### 4d. The Prompt-Builder's Role in the Collapse
|
||||
|
||||
The prompt does instruct the model to produce structural mutation (rule #6, additional guidance), and it *does* do this — but only with the primitives available. Since no primitive exists for alternatives, the model:
|
||||
1. Identifies a relevant unknown ("uncertainty about operational impact")
|
||||
2. Creates it as a single `unknown` node
|
||||
3. Embeds both options in its description prose
|
||||
4. Does not (and cannot) create option structure because none exists
|
||||
|
||||
**Verdict: Both — the schema lacks the primitives AND the prompt lacks the rules to use them.** But the root cause is schema-level; adding prompt rules without schema changes would produce inconsistent results (the model might guess which primitive to repurpose, inconsistently).
|
||||
|
||||
---
|
||||
|
||||
## 5. Reuse Strategy Evaluation: Could Existing Primitives Be Repurposed?
|
||||
|
||||
### 5a. Strategy: Treat `state` nodes as option descriptions
|
||||
|
||||
**Mechanism:** Model creates `state` nodes for "relocate state" and "stay-put state," linked by `compares_with`.
|
||||
|
||||
**Pros:**
|
||||
- Schema allows it (no validation error)
|
||||
- Minimal schema change needed
|
||||
|
||||
**Cons:**
|
||||
- `state` semantically means "a condition that holds true." Options are *conditional futures*, not actual states. This is a category error.
|
||||
- Prompt rules have no guidance for this repurposing.
|
||||
- Future reasoning about these nodes would treat them as known facts, not hypotheticals.
|
||||
- The do-nothing baseline (`state`) would be indistinguishable from an active option's outcome state.
|
||||
|
||||
**Verdict: Semantically incorrect. Would cause reasoning errors downstream.**
|
||||
|
||||
### 5b. Strategy: Treat `observation` nodes for option consequences
|
||||
|
||||
**Mechanism:** Each consequence (save £2M, lose engineers) becomes its own `observation` node attached to the option via `supports`.
|
||||
|
||||
**Pros:**
|
||||
- Schema allows it
|
||||
- `supports` edges are well-defined
|
||||
|
||||
**Cons:**
|
||||
- The *option* itself still has no structural representation.
|
||||
- Observations assert what *is*, not what *would be if chosen*.
|
||||
- Without a parent option node, consequences float without context.
|
||||
- No way to say "these observations all belong to Option A."
|
||||
|
||||
**Verdict: Incomplete. Captures consequences but not the option structure that binds them.**
|
||||
|
||||
### 5c. Strategy: Add `compares_with` semantics for alternatives
|
||||
|
||||
**Mechanism:** Define `compares_with` edge type as "these two nodes represent competing alternatives for the same decision," and add a prompt rule instructing the model to use it.
|
||||
|
||||
**Pros:**
|
||||
- Schema already has the edge type (no schema change needed)
|
||||
- If semantic definition is clear, the model can follow an explicit instruction
|
||||
|
||||
**Cons:**
|
||||
- `compares_with` semantically should mean "these two things share comparable properties" not "these are mutually exclusive options for one decision." These are fundamentally different concepts.
|
||||
- Risk of edge-type confusion when the same relationship type is used for both comparison and alternatives.
|
||||
- Still doesn't solve the missing **decision node** or **option node** problem — you'd have standalone option nodes without a parent decision context.
|
||||
|
||||
**Verdict: Partially viable as an interim solution, but semantically contaminated. Better to add dedicated types.**
|
||||
|
||||
### 5d. Strategy: Use `unknown` sub-structure via metadata (not supported)
|
||||
|
||||
**Mechanism:** Add a `decision_type` or `option_category` field to the existing `SituationNode` schema.
|
||||
|
||||
**Cons:**
|
||||
- Requires schema change (adds a field)
|
||||
- Still doesn't solve "how does the model know when to create option nodes vs standard unknowns?"
|
||||
- Adds complexity to an already dense node schema.
|
||||
|
||||
**Verdict: Fragile. Requires both schema and prompt changes with uncertain ROI.**
|
||||
|
||||
### 5e. Strategy: Use `transition` for option outcomes
|
||||
|
||||
**Mechanism:** A `transition` node represents "what happens if we choose X" (a change from baseline).
|
||||
|
||||
**Pros:**
|
||||
- Semantically closer than `state` — a transition *is* a change, and choosing an option causes a change.
|
||||
- Schema already has the kind.
|
||||
|
||||
**Cons:**
|
||||
- `transition` semantically means "a change event that occurred or is occurring," not "a hypothetical future state contingent on a choice."
|
||||
- The do-nothing baseline has no transition (it's stasis), so it would still lack structural representation.
|
||||
- Same problem as above: no parent decision node to group transitions under.
|
||||
|
||||
**Verdict: Conceptually closer than `state`, but still a category error for hypothetical option outcomes.**
|
||||
|
||||
### 5f. Summary of Reuse Strategies
|
||||
|
||||
| Strategy | Schema Change Needed | Semantic Fit | Prompt Rule Needed | Viability |
|
||||
|----------|-------------------|-------------|-------------------|-----------|
|
||||
| `state` as options | No | Poor (assertion vs hypothesis) | Yes | ❌ Not viable |
|
||||
| `observation` for consequences | Partial (need option nodes) | Poor (is vs would-be) | Yes | ❌ Incomplete |
|
||||
| Repurpose `compares_with` | No | Contaminated (comparison ≠ alternatives) | Yes | ⚠️ Interim only |
|
||||
| Add `decision_type` field to nodes | Yes | N/A (structural fix on existing type) | Maybe | ⚠️ Fragile |
|
||||
| `transition` for outcomes | No | Moderate (change event) | Yes | ⚠️ Partial |
|
||||
|
||||
**None of the reuse strategies are satisfactory without schema-level changes.** All either represent category errors or produce incomplete structures that lose information.
|
||||
|
||||
---
|
||||
|
||||
## 6. Minimum Architectural Distinction: What Exactly Is Missing?
|
||||
|
||||
### 6a. The Structural Gap as a Hierarchy of Primitives
|
||||
|
||||
To properly represent the 59B.4 decision scenario, the graph needs (from most general to most specific):
|
||||
|
||||
```
|
||||
1. DECISION — "there is a choice to make here" (parent context)
|
||||
↓ (contains)
|
||||
2. OPTION — "one available path within this decision" (branch entity)
|
||||
↓ (has consequence)
|
||||
3. CONSEQUENCE — "an outcome of choosing this option" (leaf detail)
|
||||
|
||||
Plus:
|
||||
4. ALTERNATIVE-OF — "these two options compete for the same decision" (option ↔ option relationship)
|
||||
5. BASELINE — "the state if no option is chosen" (implicit default option)
|
||||
```
|
||||
|
||||
Currently available in schema:
|
||||
```
|
||||
❌ DECISION — none exists
|
||||
❌ OPTION — none exists
|
||||
✓ CONSEQUENCE — can use `state` or `observation` (semantically imperfect but usable)
|
||||
❌ ALTERNATIVE-OF — no dedicated edge type
|
||||
❌ BASELINE — no dedicated primitive
|
||||
```
|
||||
|
||||
### 6b. The Minimum Viable Addition
|
||||
|
||||
To solve the 59B.4 pattern, the graph needs at minimum:
|
||||
|
||||
1. **A new node kind `option`** (or `alternative`) that represents "a choice available within a decision context."
|
||||
2. **A new edge relationship `alternative_of`** (or belongs_to) that says "this option is one of the choices for this decision."
|
||||
3. **A prompt rule** instructing the model to create option nodes when the answer explicitly presents competing alternatives.
|
||||
|
||||
That's it — three additions. Everything else (consequences, comparisons, baseline) can be built on top of these primitives with existing edge types.
|
||||
|
||||
### 6c. Smallest Improved Graph for 59B.4 (Conceptual)
|
||||
|
||||
With the three new primitives above, the 59B.4 answer would produce:
|
||||
|
||||
```
|
||||
[DECISION: which option leaves us better off overall?]
|
||||
├── [OPTION A: relocate] — alternative_of → DECISION
|
||||
│ ├── [CONSEQUENCE: save £2M/year] — may_cause → OPTION A
|
||||
│ ├── [CONSEQUENCE: lose 2 senior engineers] — may_cause → OPTION A
|
||||
│ └── [CONSEQUENCE: delay up to 2 months] — may_cause → OPTION A
|
||||
├── [OPTION B: stay put] — alternative_of → DECISION
|
||||
│ ├── [CONSEQUENCE: retain both engineers] — causes → OPTION B
|
||||
│ ├── [CONSEQUENCE: avoid delivery disruption] — causes → OPTION B
|
||||
│ └── [CONSEQUENCE: continue paying £2M/year] — may_cause → OPTION B
|
||||
└── [ALTERNATIVE-OF between OPTION A and OPTION B]
|
||||
```
|
||||
|
||||
Without those primitives, the best the current schema can do is the 59B.4 result: a single unknown node whose description text contains everything.
|
||||
|
||||
---
|
||||
|
||||
## 7. Smallest Improved Graph Schema (Concrete Proposal)
|
||||
|
||||
### 7a. New Enum Values
|
||||
|
||||
In `SituationKind`:
|
||||
```javascript
|
||||
option: "option" // A choice available within a decision context
|
||||
decision: "decision" // A decision point requiring selection among alternatives
|
||||
```
|
||||
|
||||
In `SituationRelationship`:
|
||||
```javascript
|
||||
alternative_of: "alternative_of" // This option is one of the choices for a decision
|
||||
contains_option: "contains_option" // This decision contains this option
|
||||
```
|
||||
|
||||
### 7b. Minimal Node Additions to `situationNodeSchema` (Optional)
|
||||
|
||||
Option nodes could carry an additional field:
|
||||
```javascript
|
||||
is_baseline: z.boolean().optional() // Is this the do-nothing / current-state default?
|
||||
```
|
||||
|
||||
This is optional — the semantics can be conveyed through `description` text if preferred.
|
||||
|
||||
### 7c. Prompt Rule Additions Needed (One Sentence Each)
|
||||
|
||||
1. "When the answer presents two or more competing alternatives for a decision, create one node of kind 'option' for each alternative."
|
||||
2. "Connect each option node to its parent decision node using the relationship 'contains_option'."
|
||||
3. "Connect competing option nodes to each other using the relationship 'alternative_of'."
|
||||
4. "If the answer explicitly or implicitly references a do-nothing baseline, represent it as an option node with is_baseline = true."
|
||||
|
||||
### 7d. What Changes in the Schema Files
|
||||
|
||||
| File | Change | Lines Affected |
|
||||
|------|--------|---------------|
|
||||
| `schema.js` — SituationKind enum | Add `option` and `decision` | ~5 lines |
|
||||
| `schema.js` — SituationRelationship enum | Add `alternative_of` and `contains_option` | ~3 lines |
|
||||
| `schema.js` — situationNodeSchema | Add optional `is_baseline` to option nodes | ~2 lines (optional) |
|
||||
| `prompt-builder.js` — rules | Add 4 new rules or extend existing rules | ~15 lines |
|
||||
| `utils.js` — applyGraphUpdate | No changes needed (new kinds are just more enum values) | 0 |
|
||||
| `apply-proposal.js` — validation | No mandatory changes; pattern compatibility logic may optionally extend to support decision reasoning patterns | 0 |
|
||||
|
||||
**Total: ~25 lines of schema + prompt changes.**
|
||||
|
||||
### 7e. What Does NOT Change
|
||||
|
||||
- Existing node kinds, statuses, and edge types remain unchanged.
|
||||
- The GraphUpdate contract (addedNodes, updatedNodes, etc.) remains unchanged.
|
||||
- No existing nodes need migration or restructuring.
|
||||
- No propagation logic needs modification (the decision/option structure sits at the same level as the existing unknown hierarchy).
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Readiness Assessment
|
||||
|
||||
### 8a. What Has Already Been Established by Prior Experiments
|
||||
|
||||
| Finding | Experiment | Implication for 60A.1 |
|
||||
|---------|-----------|----------------------|
|
||||
| Explicit dual-option framing does not produce structural alternatives | 59B.4 | Confirms the vocabulary gap is active, not theoretical |
|
||||
| Do-nothing baseline remains invisible as structure | 59B.3 | Baseline needs explicit representation, not implicit inference |
|
||||
| Engine extracts facts from structured prose correctly | 59B.4 (what it did right) | Model can extract option details; the gap is structural anchoring |
|
||||
| No resolution of `savings_realism` despite confirmatory language | 59B.4 | Resolution logic needs to recognize option-based confirmation patterns |
|
||||
| Single unknown node with multi-option description prose | 59B.2, 59B.3, 59B.4 | Confirmed persistent pattern across multiple inputs |
|
||||
|
||||
### 8b. What Is Ready to Implement (Low Risk)
|
||||
|
||||
1. **Schema additions** (`option`, `decision` kinds; `alternative_of`, `contains_option` edges): Trivial — new enum values. No breaking changes. No validation logic changes needed (new enum values are valid per Zod).
|
||||
2. **Prompt rule additions**: Straightforward — 4–5 sentences of explicit instruction for the model. Low risk, high clarity.
|
||||
3. **Deterministic decomposition templates** for option-based decisions: Can be added to `buildDecompositionTemplates()` in `apply-proposal.js` when a decision node is active.
|
||||
|
||||
### 8c. What Would Benefit from a Follow-Up Experiment (Medium Risk)
|
||||
|
||||
1. **Option-based reasoning pattern**: The current reasoning pattern system (`decision`, `explanation`, `contradiction`, etc.) could benefit from an explicit `option_comparison` pattern that governs how the engine reasons over option nodes.
|
||||
2. **Consequence propagation through option structure**: How should resolving one option's unknown propagate? If we resolve "the two engineers won't leave" for Option A, does that affect the comparison with Option B? This needs design.
|
||||
3. **Baseline visibility in reasoning patterns**: The `prioritisation` pattern could be extended to explicitly consider do-nothing baselines when an active reasoning context involves decisions.
|
||||
|
||||
### 8d. What Is Not Ready / Needs More Investigation (High Risk)
|
||||
|
||||
1. **Interim strategy via `compares_with` repurposing**: While technically possible, giving it alternative semantics risks confusion with the comparison semantics that already exist for observation comparisons (e.g., "compare two months' sales data"). A dedicated edge type is strongly preferred.
|
||||
2. **Automatic baseline detection**: Whether the model can infer a do-nothing baseline without explicit user framing needs testing. Some answers imply it; others don't. The prompt would need precise trigger conditions.
|
||||
3. **Multi-option decisions (> 2 options)**: How should N competing alternatives be represented? Linear chains of `alternative_of` edges, or a star topology centered on the decision node? This needs design.
|
||||
|
||||
### 8e. Recommended Next Experiment
|
||||
|
||||
**Experiment 60A.2 (proposed): Implement the three-primitive addition and verify 59B.4 structure.**
|
||||
|
||||
- Add `option`, `decision` kinds; `alternative_of`, `contains_option` edges to schema.
|
||||
- Add 4 prompt rules for option creation.
|
||||
- Run 59B.4 scenario against updated engine.
|
||||
- Measure: do both options appear as structural entities? Can the do-nothing baseline be represented? Is the comparison question structurally grounded?
|
||||
|
||||
---
|
||||
|
||||
## Diagnosis Conclusion
|
||||
|
||||
### The Answer to the Research Question
|
||||
|
||||
**The current graph vocabulary lacks an adequate first-class representation for alternatives and decisions.** It is not primarily a prompt problem — it is a schema problem. The model cannot represent what the schema does not define. Without `option` or `decision` node kinds and without `alternative_of` edge semantics, any dual-option input will always collapse into undifferentiated unknown prose.
|
||||
|
||||
### Evidence Chain
|
||||
|
||||
1. **Schema analysis**: Zero node kinds in SituationKind semantically represent choices or alternatives. The 9 available kinds cover observations, states, metrics, relationships, assumptions, unknowns, and conclusions — but no "option" or "decision."
|
||||
2. **Edge analysis**: `compares_with` exists but has no alternative-specific semantics. No dedicated "this option is one of the choices for this decision" relationship type exists.
|
||||
3. **Prompt analysis**: None of the 32 rules instruct the model to create structural options. The word "decision" in rule #7 refers to *a question about a decision*, not *a structural representation of the decision*.
|
||||
4. **Empirical evidence**: Experiments 59B.2–59B.4 consistently showed the same pattern — explicit dual-option input collapsed into a single undifferentiated unknown node, regardless of how strongly the user framed the comparison.
|
||||
5. **Reuse analysis**: All four candidate reuse strategies (`state`, `observation`, repurposed `compares_with`, `transition`) are either semantically incorrect or incomplete without schema-level support.
|
||||
|
||||
### Why Adding Prompt Rules Without Schema Changes Would Not Help
|
||||
|
||||
The model follows instructions precisely. If no instruction references a concept that doesn't exist in the vocabulary, the model cannot invent it. Telling the model to "create option nodes" without a valid `kind` value would cause validation errors. Repurposing existing kinds requires both schema changes (new enum values) and prompt rules anyway — so the schema change is unavoidable regardless of approach.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Add two node kinds (`option`, `decision`) and two edge relationships (`alternative_of`, `contains_option`) to the schema. Add four prompt rules for option creation. Total impact: ~25 lines of code. This addresses the root cause rather than treating symptoms.
|
||||
|
||||
---
|
||||
|
||||
## Summary Tables
|
||||
|
||||
### Key Findings Matrix
|
||||
|
||||
| Finding | Evidence | Confidence |
|
||||
|---------|----------|-----------|
|
||||
| Schema lacks option/decision primitives | schema.js SituationKind has 9 kinds, none for options | HIGH — direct code analysis |
|
||||
| `compares_with` has no alternative semantics | No documented semantics; used generically | HIGH — code + experiment history |
|
||||
| Prompt has no option creation rules | prompt-builder.js rules 1–32, no mention of options | HIGH — direct code analysis |
|
||||
| Empirical pattern persists across experiments | 59B.2, 59B.3, 59B.4 all same failure mode | HIGH — observed results |
|
||||
| Schema-only fix is ~25 lines | Two kinds + two relationships + four rules | HIGH — direct enumeration |
|
||||
|
||||
### Vocabulary Gap Summary
|
||||
|
||||
| Needed Primitive | Exists? | If not: What to Add |
|
||||
|-----------------|---------|-------------------|
|
||||
| Decision point representation | ❌ No | `decision` node kind |
|
||||
| Option / alternative entity | ❌ No | `option` node kind |
|
||||
| "This option belongs to this decision" link | ❌ No | `contains_option` edge type |
|
||||
| "These options compete" link | ❌ No (partial: `compares_with` exists but wrong semantics) | `alternative_of` edge type |
|
||||
| Do-nothing baseline representation | ❌ No | `is_baseline` flag on option nodes |
|
||||
| Option consequence attachment | ⚠️ Partially (via existing edges, if options existed) | N/A (works once options exist) |
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed during experiment: NO
|
||||
Validator changed during experiment: NO
|
||||
Vitest run: NO
|
||||
Ollama calls: 0
|
||||
Dev server disturbed: NO
|
||||
Read-only diagnosis: YES
|
||||
Reference in New Issue
Block a user