confidence-engine/docs/experiment-60a1.md

Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions

Branch: feature/question-formulation-v0.24 Date: 2026-08-12 Status: Complete Type: READ-ONLY ARCHITECTURE DIAGNOSIS — No production code changes, no API calls, no test runs.

Following experiments: 59B.2–59B.4 demonstrated a persistent structural pattern: the engine understood trade-offs semantically but could not preserve two competing options (relocate vs stay-put) as independently recoverable structural entities in the graph.

Objective

Answer this architectural question with evidence from schema, prompt rules, and the apply-proposal orchestrator:

Is the model failing to use decision structure that already exists, or does the current graph vocabulary lack an adequate first-class representation for alternatives and decisions?

This diagnosis is read-only. It analyzes whether the issue is prompt-level (the vocabulary exists but rules don't instruct the model) or schema-level (the vocabulary itself lacks the concepts needed).

---

1. Inventory of Current Graph Vocabulary

1a. Node Kinds (SituationKind)

Kind	Semantic Domain	First-Class Option/Decision Support?
observation	Factual claim about a state of the world	No — asserts existence, not choice
reported_claim	Third-party assertion	No
metric	Numerical measure	No
state	World condition / status	No — describes "what is", not "what could be"
transition	Change from one state to another	Partial — can describe a change event, but has no option-anchoring semantics
relationship	Connection between concepts	No
assumption	Taken-for-granted premise	No
unknown	Unresolved question / uncertainty	Partial — the only node kind that could host a decision-related unknown, but has no sub-structure distinguishing "option A vs option B" from "what is X?"
conclusion	Derived answer to an unknown	No — captures outcome, not process of choosing

Total: 9 distinct node kinds. Zero node kinds have semantics for choices, alternatives, or decision structures.

1b. Node Statuses (SituationStatus)

Status	Meaning	Option Relevance
known	Established fact	Irrelevant to options
unknown	Unresolved	Could host "which option?" but has no structure
provisional	Partially supported	Could be a status for an unconfirmed option
supported	Evidence-backed claim	No structural option meaning
weakened	Undermined claim	No option-specific semantics
contradicted	Conflicts with evidence	Could represent a rejected option (conceptually)
resolved	Question answered	No option-specific semantics

1c. Edge Relationships (SituationRelationship)

Relationship	Meaning	Option-Alternative Support?
supports	Evidence strengthens a node	No
weakens	Evidence undermines a node	Could represent negative consequence of an option (if options existed)
contradicts	Two nodes are mutually exclusive	Potentially relevant — mutual exclusivity is related to alternatives, but this expresses contradiction between claims, not choice between options
depends_on	One thing requires another	Could express prerequisite relationship within a decision branch
causes	Direct causal relationship	Could express option → consequence (if options existed as nodes)
may_cause	Probabilistic causal	Same as above, with uncertainty
measures	Metric tracks a concept	No
compares_with	Two things are compared	Existing but underspecified — has no documented semantics for mutual-exclusive alternatives; used generically
updates	One node updates another's value/status	No
other	Unclassified edge type	No semantic meaning

1d. Key Finding

The vocabulary contains the building blocks (nodes, edges, statuses) but lacks a decision-specific primitive. There is no:

• Decision node kind: No way to represent "the system has a decision to make" as a first-class entity
• Option/alternative node kind: No way to represent "relocate" and "stay put" as independently queryable alternatives
• "Alternative-of" edge relationship: No way to say "this option belongs to this decision"
• Explicit mutual-exclusivity semantics: compares_with exists but is semantically underspecified for options

---

2. Test: Representational Adequacy for the Relocate vs Stay-Put Decision (Using Only Current Schema)

2a. Scenario Specification

Using only the schema defined in lib/graph/schema.js:

• User says: "Option 1 is relocate (save £2M/year, lose 2 engineers, delay 2 months). Option 2 is stay put (keep engineers, avoid disruption, continue paying £2M/year)."
• The user's intent: compare these two alternatives to decide which leaves them better off.

2b. Can the schema express a "decision" node?

No. No kind in SituationKind semantically means "a decision point requiring choice between alternatives." The closest candidates are:

Candidate	Why it's inadequate
observation	An observation asserts what is, not what might be chosen
state	A state describes a condition, not a choice about conditions
unknown	Represents uncertainty about a question, not the alternatives themselves
relationship	Can link things but cannot contain structured content like "I must choose between A and B"
transition	Describes a change event, not a decision about which path to take

2c. Can the schema express "these two options are alternatives for the same decision"?

Partially, but with no structural guarantee. The compares_with edge type exists and could theoretically connect two nodes as "comparable." However:

1. No defined semantics for what it means when both endpoints are options (as opposed to two observations being compared).
2. No parent-of-decision relationship: No way to say "these options belong to this decision node."
3. No mutual-exclusivity constraint: compares_with does not express that choosing one precludes the other.
4. Not a structural alternative representation: Without a rule explicitly instructing the model to use it for alternatives, and without schema-level semantics, the model treats it as a generic "other" bucket.

2d. Can consequences attach to options structurally?

Theoretically yes, but only if options exist as nodes first. If Option A and Option B were both represented as nodes (what kind?), then consequences could attach via causes or may_cause. But without option nodes, there is nothing for the causal edges to attach to. This is a chicken-and-egg problem: you need option nodes before you can represent their consequences structurally.

2e. Can the do-nothing baseline be represented?

No dedicated representation exists. The stay-put alternative in 59B.4 was entirely absent from graph structure because:

• There's no kind for "the current state without any action"
• state nodes describe conditions, not baseline alternatives
• Without a decision/option primitive, there's no structural anchor for the baseline

Verdict: The schema is structurally inadequate for representing competing alternatives as first-class entities.

---

3. Concept Mapping: What Maps to What in Experiment 59B.4?

3a. User Input Components vs Schema Elements

User Concept	Attempted Schema Mapping	Result
Option 1 — relocate	No dedicated kind → forced into unknown description prose	Collapsed into single unknown node's text
Option 2 — stay put	No dedicated kind → lost entirely from graph	Zero structural representation
"I am deciding which..." (the decision itself)	No kind for decision/choice point	Ignored structurally
£2M/year saving (relocate benefit)	Could be metric or state, but no anchor node for the option	Embedded in unknown's description
Two engineers leave (relocate cost)	Same as above	Embedded in unknown's description
Two-month delay (relocate cost)	Same as above	Embedded in unknown's description
Keep both engineers (stay-put benefit)	No anchor node for the option	Lost from graph entirely
Avoid delivery disruption (stay-put benefit)	Same as above	Lost from graph entirely
Continue paying £2M/year (stay-put cost)	Same as above	Lost from graph entirely
Two options are alternatives for the same decision	compares_with edge type exists but has no alternative semantics	No edges created between alternatives

3b. The Core Mapping Failure

The user's input structure is:

DECISION (which option?)
├── Option A: relocate
│   ├── Benefit: save £2M/year
│   ├── Cost: lose 2 engineers
│   └── Cost: delay 2 months
└── Option B: stay put
    ├── Benefit: keep both engineers
    ├── Benefit: avoid delivery disruption
    └── Cost: continue paying £2M/year

The graph schema can represent none of the above as structure because it lacks: DECISION, OPTION, and ALTERNATIVE-OF primitives. What the user intended as a structured decision tree was forced into the closest available primitive — unknown — producing a single undifferentiated node whose description contained both options as prose.

---

4. Prompt-vs-Schema Diagnosis: Where Is the Failure?

4a. Testing the "Existing Structure" Hypothesis

If the problem were prompt-level (model fails to use existing vocabulary), we would expect:

• The schema contains a kind/relationship that could express alternatives
• The prompt rules instruct the model to use it
• The model ignores the instruction

Let's check each candidate:

Candidate 1: Use state for option descriptions

• Schema allows it ✓
• Prompt rule instructs it? No. No rule references using state nodes for "what happens if we choose X" |
• Result: Model doesn't do this (no instruction)

Candidate 2: Use compares_with edges between options

• Schema allows it ✓
• Prompt rule defines semantics for alternatives? No. No rule gives compares_with alternative-specific meaning. |
• Result: Model treats it generically (same as always)

Candidate 3: Use contradicts edges between mutually exclusive options

• Schema allows it ✓
• But contradicts expresses factual contradiction, not choice — using it for options would be semantically wrong
• No rule instructs its use for alternatives |
• Result: Not applicable

Candidate 4: Use transition for option outcomes

• Schema allows it ✓ (a transition is "change from one state to another")
• But a transition represents an actual change event, not a hypothetical option's consequences
• No rule instructs its use for options |
• Result: Not used; no instruction

4b. Testing the "Missing Vocabulary" Hypothesis

If the problem is schema-level (vocabulary lacks needed concepts), we would expect:

• The schema has no kind/relationship that correctly represents alternatives or decisions
• Adding prompt rules without adding schema primitives wouldn't help
• The model produces prose because it's the only remaining option

This matches our evidence exactly. Every analysis above shows that:

1. No existing kind semantically means "an available choice" or "a decision point"
2. No existing edge type has alternative-specific semantics
3. All four prompt-level candidates fail for the same reason: no instruction exists because there is no schema concept to instruct about

4c. The Prompt Rules Analysis (Rule #7 and Others)

Looking at prompt-builder.js rule #7:

"Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case."

This rule says "decision" in the sense of an unknown about a decision, not a decision object containing options. It's a quantity constraint (when to create unknowns), not a structure instruction (how to represent options within an unknown). The word "decision" here means "the model should recognize this answer introduces a new decision-related uncertainty," not "model should represent the decision structure itself."

No rule in the entire prompt (rules 1–32, plus additional guidance) instructs the model to:

• Create separate nodes for competing alternatives
• Use any specific node kind for options
• Connect alternatives with a specific edge type
• Represent a do-nothing baseline structurally
• Distinguish "what happens if we choose X" from "what happens if we do nothing"

4d. The Prompt-Builder's Role in the Collapse

The prompt does instruct the model to produce structural mutation (rule #6, additional guidance), and it does do this — but only with the primitives available. Since no primitive exists for alternatives, the model:

1. Identifies a relevant unknown ("uncertainty about operational impact")
2. Creates it as a single unknown node
3. Embeds both options in its description prose
4. Does not (and cannot) create option structure because none exists

Verdict: Both — the schema lacks the primitives AND the prompt lacks the rules to use them. But the root cause is schema-level; adding prompt rules without schema changes would produce inconsistent results (the model might guess which primitive to repurpose, inconsistently).

---

5. Reuse Strategy Evaluation: Could Existing Primitives Be Repurposed?

5a. Strategy: Treat state nodes as option descriptions

Mechanism: Model creates state nodes for "relocate state" and "stay-put state," linked by compares_with.

Pros:

• Schema allows it (no validation error)
• Minimal schema change needed

Cons:

• state semantically means "a condition that holds true." Options are conditional futures, not actual states. This is a category error.
• Prompt rules have no guidance for this repurposing.
• Future reasoning about these nodes would treat them as known facts, not hypotheticals.
• The do-nothing baseline (state) would be indistinguishable from an active option's outcome state.

Verdict: Semantically incorrect. Would cause reasoning errors downstream.

5b. Strategy: Treat observation nodes for option consequences

Mechanism: Each consequence (save £2M, lose engineers) becomes its own observation node attached to the option via supports.

Pros:

• Schema allows it
• supports edges are well-defined

Cons:

• The option itself still has no structural representation.
• Observations assert what is, not what would be if chosen.
• Without a parent option node, consequences float without context.
• No way to say "these observations all belong to Option A."

Verdict: Incomplete. Captures consequences but not the option structure that binds them.

5c. Strategy: Add compares_with semantics for alternatives

Mechanism: Define compares_with edge type as "these two nodes represent competing alternatives for the same decision," and add a prompt rule instructing the model to use it.

Pros:

• Schema already has the edge type (no schema change needed)
• If semantic definition is clear, the model can follow an explicit instruction

Cons:

• compares_with semantically should mean "these two things share comparable properties" not "these are mutually exclusive options for one decision." These are fundamentally different concepts.
• Risk of edge-type confusion when the same relationship type is used for both comparison and alternatives.
• Still doesn't solve the missing decision node or option node problem — you'd have standalone option nodes without a parent decision context.

Verdict: Partially viable as an interim solution, but semantically contaminated. Better to add dedicated types.

5d. Strategy: Use unknown sub-structure via metadata (not supported)

Mechanism: Add a decision_type or option_category field to the existing SituationNode schema.

Cons:

• Requires schema change (adds a field)
• Still doesn't solve "how does the model know when to create option nodes vs standard unknowns?"
• Adds complexity to an already dense node schema.

Verdict: Fragile. Requires both schema and prompt changes with uncertain ROI.

5e. Strategy: Use transition for option outcomes

Mechanism: A transition node represents "what happens if we choose X" (a change from baseline).

Pros:

• Semantically closer than state — a transition is a change, and choosing an option causes a change.
• Schema already has the kind.

Cons:

• transition semantically means "a change event that occurred or is occurring," not "a hypothetical future state contingent on a choice."
• The do-nothing baseline has no transition (it's stasis), so it would still lack structural representation.
• Same problem as above: no parent decision node to group transitions under.

Verdict: Conceptually closer than state, but still a category error for hypothetical option outcomes.

5f. Summary of Reuse Strategies

Strategy	Schema Change Needed	Semantic Fit	Prompt Rule Needed	Viability
state as options	No	Poor (assertion vs hypothesis)	Yes	❌ Not viable
observation for consequences	Partial (need option nodes)	Poor (is vs would-be)	Yes	❌ Incomplete
Repurpose compares_with	No	Contaminated (comparison ≠ alternatives)	Yes	⚠️ Interim only
Add decision_type field to nodes	Yes	N/A (structural fix on existing type)	Maybe	⚠️ Fragile
transition for outcomes	No	Moderate (change event)	Yes	⚠️ Partial

None of the reuse strategies are satisfactory without schema-level changes. All either represent category errors or produce incomplete structures that lose information.

---

6. Minimum Architectural Distinction: What Exactly Is Missing?

6a. The Structural Gap as a Hierarchy of Primitives

To properly represent the 59B.4 decision scenario, the graph needs (from most general to most specific):

1. DECISION — "there is a choice to make here" (parent context)
   ↓ (contains)
2. OPTION — "one available path within this decision" (branch entity)
   ↓ (has consequence)
3. CONSEQUENCE — "an outcome of choosing this option" (leaf detail)

Plus:
4. ALTERNATIVE-OF — "these two options compete for the same decision" (option ↔ option relationship)
5. BASELINE — "the state if no option is chosen" (implicit default option)

Currently available in schema:

❌ DECISION — none exists
❌ OPTION — none exists
✓ CONSEQUENCE — can use `state` or `observation` (semantically imperfect but usable)
❌ ALTERNATIVE-OF — no dedicated edge type
❌ BASELINE — no dedicated primitive

6b. The Minimum Viable Addition

To solve the 59B.4 pattern, the graph needs at minimum:

1. A new node kind option (or alternative) that represents "a choice available within a decision context."
2. A new edge relationship alternative_of (or belongs_to) that says "this option is one of the choices for this decision."
3. A prompt rule instructing the model to create option nodes when the answer explicitly presents competing alternatives.

That's it — three additions. Everything else (consequences, comparisons, baseline) can be built on top of these primitives with existing edge types.

6c. Smallest Improved Graph for 59B.4 (Conceptual)

With the three new primitives above, the 59B.4 answer would produce:

[DECISION: which option leaves us better off overall?]
├── [OPTION A: relocate] — alternative_of → DECISION
│   ├── [CONSEQUENCE: save £2M/year] — may_cause → OPTION A
│   ├── [CONSEQUENCE: lose 2 senior engineers] — may_cause → OPTION A
│   └── [CONSEQUENCE: delay up to 2 months] — may_cause → OPTION A
├── [OPTION B: stay put] — alternative_of → DECISION
│   ├── [CONSEQUENCE: retain both engineers] — causes → OPTION B
│   ├── [CONSEQUENCE: avoid delivery disruption] — causes → OPTION B
│   └── [CONSEQUENCE: continue paying £2M/year] — may_cause → OPTION B
└── [ALTERNATIVE-OF between OPTION A and OPTION B]

Without those primitives, the best the current schema can do is the 59B.4 result: a single unknown node whose description text contains everything.

---

7. Smallest Improved Graph Schema (Concrete Proposal)

7a. New Enum Values

In SituationKind:

option: "option"          // A choice available within a decision context
decision: "decision"      // A decision point requiring selection among alternatives

In SituationRelationship:

alternative_of: "alternative_of"  // This option is one of the choices for a decision
contains_option: "contains_option" // This decision contains this option

7b. Minimal Node Additions to situationNodeSchema (Optional)

Option nodes could carry an additional field:

is_baseline: z.boolean().optional()  // Is this the do-nothing / current-state default?

This is optional — the semantics can be conveyed through description text if preferred.

7c. Prompt Rule Additions Needed (One Sentence Each)

1. "When the answer presents two or more competing alternatives for a decision, create one node of kind 'option' for each alternative."
2. "Connect each option node to its parent decision node using the relationship 'contains_option'."
3. "Connect competing option nodes to each other using the relationship 'alternative_of'."
4. "If the answer explicitly or implicitly references a do-nothing baseline, represent it as an option node with is_baseline = true."

7d. What Changes in the Schema Files

File	Change	Lines Affected
schema.js — SituationKind enum	Add option and decision	~5 lines
schema.js — SituationRelationship enum	Add alternative_of and contains_option	~3 lines
schema.js — situationNodeSchema	Add optional is_baseline to option nodes	~2 lines (optional)
prompt-builder.js — rules	Add 4 new rules or extend existing rules	~15 lines
utils.js — applyGraphUpdate	No changes needed (new kinds are just more enum values)	0
apply-proposal.js — validation	No mandatory changes; pattern compatibility logic may optionally extend to support decision reasoning patterns	0

Total: ~25 lines of schema + prompt changes.

7e. What Does NOT Change

• Existing node kinds, statuses, and edge types remain unchanged.
• The GraphUpdate contract (addedNodes, updatedNodes, etc.) remains unchanged.
• No existing nodes need migration or restructuring.
• No propagation logic needs modification (the decision/option structure sits at the same level as the existing unknown hierarchy).

---

8. Implementation Readiness Assessment

8a. What Has Already Been Established by Prior Experiments

Finding	Experiment	Implication for 60A.1
Explicit dual-option framing does not produce structural alternatives	59B.4	Confirms the vocabulary gap is active, not theoretical
Do-nothing baseline remains invisible as structure	59B.3	Baseline needs explicit representation, not implicit inference
Engine extracts facts from structured prose correctly	59B.4 (what it did right)	Model can extract option details; the gap is structural anchoring
No resolution of savings_realism despite confirmatory language	59B.4	Resolution logic needs to recognize option-based confirmation patterns
Single unknown node with multi-option description prose	59B.2, 59B.3, 59B.4	Confirmed persistent pattern across multiple inputs

8b. What Is Ready to Implement (Low Risk)

1. Schema additions (option, decision kinds; alternative_of, contains_option edges): Trivial — new enum values. No breaking changes. No validation logic changes needed (new enum values are valid per Zod).
2. Prompt rule additions: Straightforward — 4–5 sentences of explicit instruction for the model. Low risk, high clarity.
3. Deterministic decomposition templates for option-based decisions: Can be added to buildDecompositionTemplates() in apply-proposal.js when a decision node is active.

8c. What Would Benefit from a Follow-Up Experiment (Medium Risk)

1. Option-based reasoning pattern: The current reasoning pattern system (decision, explanation, contradiction, etc.) could benefit from an explicit option_comparison pattern that governs how the engine reasons over option nodes.
2. Consequence propagation through option structure: How should resolving one option's unknown propagate? If we resolve "the two engineers won't leave" for Option A, does that affect the comparison with Option B? This needs design.
3. Baseline visibility in reasoning patterns: The prioritisation pattern could be extended to explicitly consider do-nothing baselines when an active reasoning context involves decisions.

8d. What Is Not Ready / Needs More Investigation (High Risk)

1. Interim strategy via compares_with repurposing: While technically possible, giving it alternative semantics risks confusion with the comparison semantics that already exist for observation comparisons (e.g., "compare two months' sales data"). A dedicated edge type is strongly preferred.
2. Automatic baseline detection: Whether the model can infer a do-nothing baseline without explicit user framing needs testing. Some answers imply it; others don't. The prompt would need precise trigger conditions.
3. Multi-option decisions (> 2 options): How should N competing alternatives be represented? Linear chains of alternative_of edges, or a star topology centered on the decision node? This needs design.

8e. Recommended Next Experiment

Experiment 60A.2 (proposed): Implement the three-primitive addition and verify 59B.4 structure.

• Add option, decision kinds; alternative_of, contains_option edges to schema.
• Add 4 prompt rules for option creation.
• Run 59B.4 scenario against updated engine.
• Measure: do both options appear as structural entities? Can the do-nothing baseline be represented? Is the comparison question structurally grounded?

---

Diagnosis Conclusion

The Answer to the Research Question

The current graph vocabulary lacks an adequate first-class representation for alternatives and decisions. It is not primarily a prompt problem — it is a schema problem. The model cannot represent what the schema does not define. Without option or decision node kinds and without alternative_of edge semantics, any dual-option input will always collapse into undifferentiated unknown prose.

Evidence Chain

1. Schema analysis: Zero node kinds in SituationKind semantically represent choices or alternatives. The 9 available kinds cover observations, states, metrics, relationships, assumptions, unknowns, and conclusions — but no "option" or "decision."
2. Edge analysis: compares_with exists but has no alternative-specific semantics. No dedicated "this option is one of the choices for this decision" relationship type exists.
3. Prompt analysis: None of the 32 rules instruct the model to create structural options. The word "decision" in rule #7 refers to a question about a decision, not a structural representation of the decision.
4. Empirical evidence: Experiments 59B.2–59B.4 consistently showed the same pattern — explicit dual-option input collapsed into a single undifferentiated unknown node, regardless of how strongly the user framed the comparison.
5. Reuse analysis: All four candidate reuse strategies (state, observation, repurposed compares_with, transition) are either semantically incorrect or incomplete without schema-level support.

Why Adding Prompt Rules Without Schema Changes Would Not Help

The model follows instructions precisely. If no instruction references a concept that doesn't exist in the vocabulary, the model cannot invent it. Telling the model to "create option nodes" without a valid kind value would cause validation errors. Repurposing existing kinds requires both schema changes (new enum values) and prompt rules anyway — so the schema change is unavoidable regardless of approach.

Recommendation

Add two node kinds (option, decision) and two edge relationships (alternative_of, contains_option) to the schema. Add four prompt rules for option creation. Total impact: ~25 lines of code. This addresses the root cause rather than treating symptoms.

---

Summary Tables

Key Findings Matrix

Finding	Evidence	Confidence
Schema lacks option/decision primitives	schema.js SituationKind has 9 kinds, none for options	HIGH — direct code analysis
compares_with has no alternative semantics	No documented semantics; used generically	HIGH — code + experiment history
Prompt has no option creation rules	prompt-builder.js rules 1–32, no mention of options	HIGH — direct code analysis
Empirical pattern persists across experiments	59B.2, 59B.3, 59B.4 all same failure mode	HIGH — observed results
Schema-only fix is ~25 lines	Two kinds + two relationships + four rules	HIGH — direct enumeration

Vocabulary Gap Summary

Needed Primitive	Exists?	If not: What to Add
Decision point representation	❌ No	decision node kind
Option / alternative entity	❌ No	option node kind
"This option belongs to this decision" link	❌ No	contains_option edge type
"These options compete" link	❌ No (partial: compares_with exists but wrong semantics)	alternative_of edge type
Do-nothing baseline representation	❌ No	is_baseline flag on option nodes
Option consequence attachment	⚠️ Partially (via existing edges, if options existed)	N/A (works once options exist)

---

Production code changed: NO Prompt changed during experiment: NO Validator changed during experiment: NO Vitest run: NO Ollama calls: 0 Dev server disturbed: NO Read-only diagnosis: YES