docs: archive historical Confidence Engine evidence
This commit is contained in:
@@ -0,0 +1,527 @@
|
||||
# Experiment 60A.1 — Read-Only Vocabulary Adequacy Diagnosis for Alternatives and Decisions
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Type:** READ-ONLY ARCHITECTURE DIAGNOSIS — No production code changes, no API calls, no test runs.
|
||||
|
||||
**Following experiments:** 59B.2–59B.4 demonstrated a persistent structural pattern: the engine understood trade-offs semantically but could not preserve two competing options (relocate vs stay-put) as independently recoverable structural entities in the graph.
|
||||
|
||||
## Objective
|
||||
|
||||
Answer this architectural question with evidence from schema, prompt rules, and the apply-proposal orchestrator:
|
||||
|
||||
> Is the model failing to use decision structure that already exists, or does the current graph vocabulary lack an adequate first-class representation for alternatives and decisions?
|
||||
|
||||
This diagnosis is read-only. It analyzes whether the issue is **prompt-level** (the vocabulary exists but rules don't instruct the model) or **schema-level** (the vocabulary itself lacks the concepts needed).
|
||||
|
||||
---
|
||||
|
||||
## 1. Inventory of Current Graph Vocabulary
|
||||
|
||||
### 1a. Node Kinds (`SituationKind`)
|
||||
|
||||
| Kind | Semantic Domain | First-Class Option/Decision Support? |
|
||||
|------|----------------|-------------------------------------|
|
||||
| `observation` | Factual claim about a state of the world | No — asserts existence, not choice |
|
||||
| `reported_claim` | Third-party assertion | No |
|
||||
| `metric` | Numerical measure | No |
|
||||
| `state` | World condition / status | No — describes "what is", not "what could be" |
|
||||
| `transition` | Change from one state to another | Partial — can describe a change event, but has no option-anchoring semantics |
|
||||
| `relationship` | Connection between concepts | No |
|
||||
| `assumption` | Taken-for-granted premise | No |
|
||||
| `unknown` | Unresolved question / uncertainty | **Partial** — the only node kind that *could* host a decision-related unknown, but has no sub-structure distinguishing "option A vs option B" from "what is X?" |
|
||||
| `conclusion` | Derived answer to an unknown | No — captures outcome, not process of choosing |
|
||||
|
||||
**Total: 9 distinct node kinds. Zero node kinds have semantics for choices, alternatives, or decision structures.**
|
||||
|
||||
### 1b. Node Statuses (`SituationStatus`)
|
||||
|
||||
| Status | Meaning | Option Relevance |
|
||||
|--------|---------|-----------------|
|
||||
| `known` | Established fact | Irrelevant to options |
|
||||
| `unknown` | Unresolved | Could host "which option?" but has no structure |
|
||||
| `provisional` | Partially supported | Could be a status for an unconfirmed option |
|
||||
| `supported` | Evidence-backed claim | No structural option meaning |
|
||||
| `weakened` | Undermined claim | No option-specific semantics |
|
||||
| `contradicted` | Conflicts with evidence | Could represent a rejected option (conceptually) |
|
||||
| `resolved` | Question answered | No option-specific semantics |
|
||||
|
||||
### 1c. Edge Relationships (`SituationRelationship`)
|
||||
|
||||
| Relationship | Meaning | Option-Alternative Support? |
|
||||
|--------------|---------|---------------------------|
|
||||
| `supports` | Evidence strengthens a node | No |
|
||||
| `weakens` | Evidence undermines a node | Could represent negative consequence of an option (if options existed) |
|
||||
| `contradicts` | Two nodes are mutually exclusive | **Potentially relevant** — mutual exclusivity is related to alternatives, but this expresses contradiction between *claims*, not choice between *options* |
|
||||
| `depends_on` | One thing requires another | Could express prerequisite relationship within a decision branch |
|
||||
| `causes` | Direct causal relationship | Could express option → consequence (if options existed as nodes) |
|
||||
| `may_cause` | Probabilistic causal | Same as above, with uncertainty |
|
||||
| `measures` | Metric tracks a concept | No |
|
||||
| `compares_with` | Two things are compared | **Existing but underspecified** — has no documented semantics for mutual-exclusive alternatives; used generically |
|
||||
| `updates` | One node updates another's value/status | No |
|
||||
| `other` | Unclassified edge type | No semantic meaning |
|
||||
|
||||
### 1d. Key Finding
|
||||
|
||||
The vocabulary contains the **building blocks** (nodes, edges, statuses) but lacks a **decision-specific primitive**. There is no:
|
||||
|
||||
- **Decision node kind**: No way to represent "the system has a decision to make" as a first-class entity
|
||||
- **Option/alternative node kind**: No way to represent "relocate" and "stay put" as independently queryable alternatives
|
||||
- **"Alternative-of" edge relationship**: No way to say "this option belongs to this decision"
|
||||
- **Explicit mutual-exclusivity semantics**: `compares_with` exists but is semantically underspecified for options
|
||||
|
||||
---
|
||||
|
||||
## 2. Test: Representational Adequacy for the Relocate vs Stay-Put Decision (Using Only Current Schema)
|
||||
|
||||
### 2a. Scenario Specification
|
||||
|
||||
Using only the schema defined in `lib/graph/schema.js`:
|
||||
|
||||
- User says: "Option 1 is relocate (save £2M/year, lose 2 engineers, delay 2 months). Option 2 is stay put (keep engineers, avoid disruption, continue paying £2M/year)."
|
||||
- The user's intent: compare these two alternatives to decide which leaves them better off.
|
||||
|
||||
### 2b. Can the schema express a "decision" node?
|
||||
|
||||
**No.** No kind in `SituationKind` semantically means "a decision point requiring choice between alternatives." The closest candidates are:
|
||||
|
||||
| Candidate | Why it's inadequate |
|
||||
|-----------|-------------------|
|
||||
| `observation` | An observation asserts what *is*, not what *might be chosen* |
|
||||
| `state` | A state describes a condition, not a choice about conditions |
|
||||
| `unknown` | Represents uncertainty about a question, not the alternatives themselves |
|
||||
| `relationship` | Can link things but cannot contain structured content like "I must choose between A and B" |
|
||||
| `transition` | Describes a change event, not a decision about which path to take |
|
||||
|
||||
### 2c. Can the schema express "these two options are alternatives for the same decision"?
|
||||
|
||||
**Partially, but with no structural guarantee.** The `compares_with` edge type exists and could theoretically connect two nodes as "comparable." However:
|
||||
|
||||
1. **No defined semantics** for what it means when both endpoints are *options* (as opposed to two observations being compared).
|
||||
2. **No parent-of-decision relationship**: No way to say "these options belong to this decision node."
|
||||
3. **No mutual-exclusivity constraint**: `compares_with` does not express that choosing one precludes the other.
|
||||
4. **Not a structural alternative representation**: Without a rule explicitly instructing the model to use it for alternatives, and without schema-level semantics, the model treats it as a generic "other" bucket.
|
||||
|
||||
### 2d. Can consequences attach to options structurally?
|
||||
|
||||
**Theoretically yes, but only if options exist as nodes first.** If Option A and Option B were both represented as nodes (what kind?), then consequences could attach via `causes` or `may_cause`. But without option nodes, there is nothing for the causal edges to attach to. This is a **chicken-and-egg problem**: you need option nodes before you can represent their consequences structurally.
|
||||
|
||||
### 2e. Can the do-nothing baseline be represented?
|
||||
|
||||
**No dedicated representation exists.** The stay-put alternative in 59B.4 was entirely absent from graph structure because:
|
||||
- There's no kind for "the current state without any action"
|
||||
- `state` nodes describe conditions, not baseline alternatives
|
||||
- Without a decision/option primitive, there's no structural anchor for the baseline
|
||||
|
||||
**Verdict: The schema is structurally inadequate for representing competing alternatives as first-class entities.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Concept Mapping: What Maps to What in Experiment 59B.4?
|
||||
|
||||
### 3a. User Input Components vs Schema Elements
|
||||
|
||||
| User Concept | Attempted Schema Mapping | Result |
|
||||
|-------------|------------------------|--------|
|
||||
| **Option 1 — relocate** | No dedicated kind → forced into `unknown` description prose | Collapsed into single unknown node's text |
|
||||
| **Option 2 — stay put** | No dedicated kind → lost entirely from graph | Zero structural representation |
|
||||
| **"I am deciding which..."** (the decision itself) | No kind for decision/choice point | Ignored structurally |
|
||||
| **£2M/year saving** (relocate benefit) | Could be `metric` or `state`, but no anchor node for the option | Embedded in unknown's description |
|
||||
| **Two engineers leave** (relocate cost) | Same as above | Embedded in unknown's description |
|
||||
| **Two-month delay** (relocate cost) | Same as above | Embedded in unknown's description |
|
||||
| **Keep both engineers** (stay-put benefit) | No anchor node for the option | Lost from graph entirely |
|
||||
| **Avoid delivery disruption** (stay-put benefit) | Same as above | Lost from graph entirely |
|
||||
| **Continue paying £2M/year** (stay-put cost) | Same as above | Lost from graph entirely |
|
||||
| **Two options are alternatives for the same decision** | `compares_with` edge type exists but has no alternative semantics | No edges created between alternatives |
|
||||
|
||||
### 3b. The Core Mapping Failure
|
||||
|
||||
The user's input structure is:
|
||||
|
||||
```
|
||||
DECISION (which option?)
|
||||
├── Option A: relocate
|
||||
│ ├── Benefit: save £2M/year
|
||||
│ ├── Cost: lose 2 engineers
|
||||
│ └── Cost: delay 2 months
|
||||
└── Option B: stay put
|
||||
├── Benefit: keep both engineers
|
||||
├── Benefit: avoid delivery disruption
|
||||
└── Cost: continue paying £2M/year
|
||||
```
|
||||
|
||||
The graph schema can represent **none** of the above as structure because it lacks: `DECISION`, `OPTION`, and `ALTERNATIVE-OF` primitives. What the user intended as a **structured decision tree** was forced into the closest available primitive — `unknown` — producing a single undifferentiated node whose description contained both options as prose.
|
||||
|
||||
---
|
||||
|
||||
## 4. Prompt-vs-Schema Diagnosis: Where Is the Failure?
|
||||
|
||||
### 4a. Testing the "Existing Structure" Hypothesis
|
||||
|
||||
If the problem were **prompt-level** (model fails to use existing vocabulary), we would expect:
|
||||
- The schema contains a kind/relationship that *could* express alternatives
|
||||
- The prompt rules instruct the model to use it
|
||||
- The model ignores the instruction
|
||||
|
||||
Let's check each candidate:
|
||||
|
||||
**Candidate 1: Use `state` for option descriptions**
|
||||
- Schema allows it ✓
|
||||
- Prompt rule instructs it? **No.** No rule references using `state` nodes for "what happens if we choose X" |
|
||||
- Result: Model doesn't do this (no instruction)
|
||||
|
||||
**Candidate 2: Use `compares_with` edges between options**
|
||||
- Schema allows it ✓
|
||||
- Prompt rule defines semantics for alternatives? **No.** No rule gives `compares_with` alternative-specific meaning. |
|
||||
- Result: Model treats it generically (same as always)
|
||||
|
||||
**Candidate 3: Use `contradicts` edges between mutually exclusive options**
|
||||
- Schema allows it ✓
|
||||
- But `contradicts` expresses factual contradiction, not choice — using it for options would be semantically wrong
|
||||
- **No rule instructs its use for alternatives** |
|
||||
- Result: Not applicable
|
||||
|
||||
**Candidate 4: Use `transition` for option outcomes**
|
||||
- Schema allows it ✓ (a transition is "change from one state to another")
|
||||
- But a transition represents an actual change event, not a hypothetical option's consequences
|
||||
- **No rule instructs its use for options** |
|
||||
- Result: Not used; no instruction
|
||||
|
||||
### 4b. Testing the "Missing Vocabulary" Hypothesis
|
||||
|
||||
If the problem is **schema-level** (vocabulary lacks needed concepts), we would expect:
|
||||
- The schema has no kind/relationship that correctly represents alternatives or decisions
|
||||
- Adding prompt rules without adding schema primitives wouldn't help
|
||||
- The model produces prose because it's the only remaining option
|
||||
|
||||
This matches our evidence exactly. Every analysis above shows that:
|
||||
1. No existing kind semantically means "an available choice" or "a decision point"
|
||||
2. No existing edge type has alternative-specific semantics
|
||||
3. All four prompt-level candidates fail for the same reason: **no instruction exists** because there is no schema concept to instruct about
|
||||
|
||||
### 4c. The Prompt Rules Analysis (Rule #7 and Others)
|
||||
|
||||
Looking at `prompt-builder.js` rule #7:
|
||||
|
||||
> "Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case."
|
||||
|
||||
This rule says "decision" in the sense of *an unknown about a decision*, not *a decision object containing options*. It's a **quantity constraint** (when to create unknowns), not a **structure instruction** (how to represent options within an unknown). The word "decision" here means "the model should recognize this answer introduces a new decision-related uncertainty," not "model should represent the decision structure itself."
|
||||
|
||||
**No rule in the entire prompt (rules 1–32, plus additional guidance) instructs the model to:**
|
||||
- Create separate nodes for competing alternatives
|
||||
- Use any specific node kind for options
|
||||
- Connect alternatives with a specific edge type
|
||||
- Represent a do-nothing baseline structurally
|
||||
- Distinguish "what happens if we choose X" from "what happens if we do nothing"
|
||||
|
||||
### 4d. The Prompt-Builder's Role in the Collapse
|
||||
|
||||
The prompt does instruct the model to produce structural mutation (rule #6, additional guidance), and it *does* do this — but only with the primitives available. Since no primitive exists for alternatives, the model:
|
||||
1. Identifies a relevant unknown ("uncertainty about operational impact")
|
||||
2. Creates it as a single `unknown` node
|
||||
3. Embeds both options in its description prose
|
||||
4. Does not (and cannot) create option structure because none exists
|
||||
|
||||
**Verdict: Both — the schema lacks the primitives AND the prompt lacks the rules to use them.** But the root cause is schema-level; adding prompt rules without schema changes would produce inconsistent results (the model might guess which primitive to repurpose, inconsistently).
|
||||
|
||||
---
|
||||
|
||||
## 5. Reuse Strategy Evaluation: Could Existing Primitives Be Repurposed?
|
||||
|
||||
### 5a. Strategy: Treat `state` nodes as option descriptions
|
||||
|
||||
**Mechanism:** Model creates `state` nodes for "relocate state" and "stay-put state," linked by `compares_with`.
|
||||
|
||||
**Pros:**
|
||||
- Schema allows it (no validation error)
|
||||
- Minimal schema change needed
|
||||
|
||||
**Cons:**
|
||||
- `state` semantically means "a condition that holds true." Options are *conditional futures*, not actual states. This is a category error.
|
||||
- Prompt rules have no guidance for this repurposing.
|
||||
- Future reasoning about these nodes would treat them as known facts, not hypotheticals.
|
||||
- The do-nothing baseline (`state`) would be indistinguishable from an active option's outcome state.
|
||||
|
||||
**Verdict: Semantically incorrect. Would cause reasoning errors downstream.**
|
||||
|
||||
### 5b. Strategy: Treat `observation` nodes for option consequences
|
||||
|
||||
**Mechanism:** Each consequence (save £2M, lose engineers) becomes its own `observation` node attached to the option via `supports`.
|
||||
|
||||
**Pros:**
|
||||
- Schema allows it
|
||||
- `supports` edges are well-defined
|
||||
|
||||
**Cons:**
|
||||
- The *option* itself still has no structural representation.
|
||||
- Observations assert what *is*, not what *would be if chosen*.
|
||||
- Without a parent option node, consequences float without context.
|
||||
- No way to say "these observations all belong to Option A."
|
||||
|
||||
**Verdict: Incomplete. Captures consequences but not the option structure that binds them.**
|
||||
|
||||
### 5c. Strategy: Add `compares_with` semantics for alternatives
|
||||
|
||||
**Mechanism:** Define `compares_with` edge type as "these two nodes represent competing alternatives for the same decision," and add a prompt rule instructing the model to use it.
|
||||
|
||||
**Pros:**
|
||||
- Schema already has the edge type (no schema change needed)
|
||||
- If semantic definition is clear, the model can follow an explicit instruction
|
||||
|
||||
**Cons:**
|
||||
- `compares_with` semantically should mean "these two things share comparable properties" not "these are mutually exclusive options for one decision." These are fundamentally different concepts.
|
||||
- Risk of edge-type confusion when the same relationship type is used for both comparison and alternatives.
|
||||
- Still doesn't solve the missing **decision node** or **option node** problem — you'd have standalone option nodes without a parent decision context.
|
||||
|
||||
**Verdict: Partially viable as an interim solution, but semantically contaminated. Better to add dedicated types.**
|
||||
|
||||
### 5d. Strategy: Use `unknown` sub-structure via metadata (not supported)
|
||||
|
||||
**Mechanism:** Add a `decision_type` or `option_category` field to the existing `SituationNode` schema.
|
||||
|
||||
**Cons:**
|
||||
- Requires schema change (adds a field)
|
||||
- Still doesn't solve "how does the model know when to create option nodes vs standard unknowns?"
|
||||
- Adds complexity to an already dense node schema.
|
||||
|
||||
**Verdict: Fragile. Requires both schema and prompt changes with uncertain ROI.**
|
||||
|
||||
### 5e. Strategy: Use `transition` for option outcomes
|
||||
|
||||
**Mechanism:** A `transition` node represents "what happens if we choose X" (a change from baseline).
|
||||
|
||||
**Pros:**
|
||||
- Semantically closer than `state` — a transition *is* a change, and choosing an option causes a change.
|
||||
- Schema already has the kind.
|
||||
|
||||
**Cons:**
|
||||
- `transition` semantically means "a change event that occurred or is occurring," not "a hypothetical future state contingent on a choice."
|
||||
- The do-nothing baseline has no transition (it's stasis), so it would still lack structural representation.
|
||||
- Same problem as above: no parent decision node to group transitions under.
|
||||
|
||||
**Verdict: Conceptually closer than `state`, but still a category error for hypothetical option outcomes.**
|
||||
|
||||
### 5f. Summary of Reuse Strategies
|
||||
|
||||
| Strategy | Schema Change Needed | Semantic Fit | Prompt Rule Needed | Viability |
|
||||
|----------|-------------------|-------------|-------------------|-----------|
|
||||
| `state` as options | No | Poor (assertion vs hypothesis) | Yes | ❌ Not viable |
|
||||
| `observation` for consequences | Partial (need option nodes) | Poor (is vs would-be) | Yes | ❌ Incomplete |
|
||||
| Repurpose `compares_with` | No | Contaminated (comparison ≠ alternatives) | Yes | ⚠️ Interim only |
|
||||
| Add `decision_type` field to nodes | Yes | N/A (structural fix on existing type) | Maybe | ⚠️ Fragile |
|
||||
| `transition` for outcomes | No | Moderate (change event) | Yes | ⚠️ Partial |
|
||||
|
||||
**None of the reuse strategies are satisfactory without schema-level changes.** All either represent category errors or produce incomplete structures that lose information.
|
||||
|
||||
---
|
||||
|
||||
## 6. Minimum Architectural Distinction: What Exactly Is Missing?
|
||||
|
||||
### 6a. The Structural Gap as a Hierarchy of Primitives
|
||||
|
||||
To properly represent the 59B.4 decision scenario, the graph needs (from most general to most specific):
|
||||
|
||||
```
|
||||
1. DECISION — "there is a choice to make here" (parent context)
|
||||
↓ (contains)
|
||||
2. OPTION — "one available path within this decision" (branch entity)
|
||||
↓ (has consequence)
|
||||
3. CONSEQUENCE — "an outcome of choosing this option" (leaf detail)
|
||||
|
||||
Plus:
|
||||
4. ALTERNATIVE-OF — "these two options compete for the same decision" (option ↔ option relationship)
|
||||
5. BASELINE — "the state if no option is chosen" (implicit default option)
|
||||
```
|
||||
|
||||
Currently available in schema:
|
||||
```
|
||||
❌ DECISION — none exists
|
||||
❌ OPTION — none exists
|
||||
✓ CONSEQUENCE — can use `state` or `observation` (semantically imperfect but usable)
|
||||
❌ ALTERNATIVE-OF — no dedicated edge type
|
||||
❌ BASELINE — no dedicated primitive
|
||||
```
|
||||
|
||||
### 6b. The Minimum Viable Addition
|
||||
|
||||
To solve the 59B.4 pattern, the graph needs at minimum:
|
||||
|
||||
1. **A new node kind `option`** (or `alternative`) that represents "a choice available within a decision context."
|
||||
2. **A new edge relationship `alternative_of`** (or belongs_to) that says "this option is one of the choices for this decision."
|
||||
3. **A prompt rule** instructing the model to create option nodes when the answer explicitly presents competing alternatives.
|
||||
|
||||
That's it — three additions. Everything else (consequences, comparisons, baseline) can be built on top of these primitives with existing edge types.
|
||||
|
||||
### 6c. Smallest Improved Graph for 59B.4 (Conceptual)
|
||||
|
||||
With the three new primitives above, the 59B.4 answer would produce:
|
||||
|
||||
```
|
||||
[DECISION: which option leaves us better off overall?]
|
||||
├── [OPTION A: relocate] — alternative_of → DECISION
|
||||
│ ├── [CONSEQUENCE: save £2M/year] — may_cause → OPTION A
|
||||
│ ├── [CONSEQUENCE: lose 2 senior engineers] — may_cause → OPTION A
|
||||
│ └── [CONSEQUENCE: delay up to 2 months] — may_cause → OPTION A
|
||||
├── [OPTION B: stay put] — alternative_of → DECISION
|
||||
│ ├── [CONSEQUENCE: retain both engineers] — causes → OPTION B
|
||||
│ ├── [CONSEQUENCE: avoid delivery disruption] — causes → OPTION B
|
||||
│ └── [CONSEQUENCE: continue paying £2M/year] — may_cause → OPTION B
|
||||
└── [ALTERNATIVE-OF between OPTION A and OPTION B]
|
||||
```
|
||||
|
||||
Without those primitives, the best the current schema can do is the 59B.4 result: a single unknown node whose description text contains everything.
|
||||
|
||||
---
|
||||
|
||||
## 7. Smallest Improved Graph Schema (Concrete Proposal)
|
||||
|
||||
### 7a. New Enum Values
|
||||
|
||||
In `SituationKind`:
|
||||
```javascript
|
||||
option: "option" // A choice available within a decision context
|
||||
decision: "decision" // A decision point requiring selection among alternatives
|
||||
```
|
||||
|
||||
In `SituationRelationship`:
|
||||
```javascript
|
||||
alternative_of: "alternative_of" // This option is one of the choices for a decision
|
||||
contains_option: "contains_option" // This decision contains this option
|
||||
```
|
||||
|
||||
### 7b. Minimal Node Additions to `situationNodeSchema` (Optional)
|
||||
|
||||
Option nodes could carry an additional field:
|
||||
```javascript
|
||||
is_baseline: z.boolean().optional() // Is this the do-nothing / current-state default?
|
||||
```
|
||||
|
||||
This is optional — the semantics can be conveyed through `description` text if preferred.
|
||||
|
||||
### 7c. Prompt Rule Additions Needed (One Sentence Each)
|
||||
|
||||
1. "When the answer presents two or more competing alternatives for a decision, create one node of kind 'option' for each alternative."
|
||||
2. "Connect each option node to its parent decision node using the relationship 'contains_option'."
|
||||
3. "Connect competing option nodes to each other using the relationship 'alternative_of'."
|
||||
4. "If the answer explicitly or implicitly references a do-nothing baseline, represent it as an option node with is_baseline = true."
|
||||
|
||||
### 7d. What Changes in the Schema Files
|
||||
|
||||
| File | Change | Lines Affected |
|
||||
|------|--------|---------------|
|
||||
| `schema.js` — SituationKind enum | Add `option` and `decision` | ~5 lines |
|
||||
| `schema.js` — SituationRelationship enum | Add `alternative_of` and `contains_option` | ~3 lines |
|
||||
| `schema.js` — situationNodeSchema | Add optional `is_baseline` to option nodes | ~2 lines (optional) |
|
||||
| `prompt-builder.js` — rules | Add 4 new rules or extend existing rules | ~15 lines |
|
||||
| `utils.js` — applyGraphUpdate | No changes needed (new kinds are just more enum values) | 0 |
|
||||
| `apply-proposal.js` — validation | No mandatory changes; pattern compatibility logic may optionally extend to support decision reasoning patterns | 0 |
|
||||
|
||||
**Total: ~25 lines of schema + prompt changes.**
|
||||
|
||||
### 7e. What Does NOT Change
|
||||
|
||||
- Existing node kinds, statuses, and edge types remain unchanged.
|
||||
- The GraphUpdate contract (addedNodes, updatedNodes, etc.) remains unchanged.
|
||||
- No existing nodes need migration or restructuring.
|
||||
- No propagation logic needs modification (the decision/option structure sits at the same level as the existing unknown hierarchy).
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Readiness Assessment
|
||||
|
||||
### 8a. What Has Already Been Established by Prior Experiments
|
||||
|
||||
| Finding | Experiment | Implication for 60A.1 |
|
||||
|---------|-----------|----------------------|
|
||||
| Explicit dual-option framing does not produce structural alternatives | 59B.4 | Confirms the vocabulary gap is active, not theoretical |
|
||||
| Do-nothing baseline remains invisible as structure | 59B.3 | Baseline needs explicit representation, not implicit inference |
|
||||
| Engine extracts facts from structured prose correctly | 59B.4 (what it did right) | Model can extract option details; the gap is structural anchoring |
|
||||
| No resolution of `savings_realism` despite confirmatory language | 59B.4 | Resolution logic needs to recognize option-based confirmation patterns |
|
||||
| Single unknown node with multi-option description prose | 59B.2, 59B.3, 59B.4 | Confirmed persistent pattern across multiple inputs |
|
||||
|
||||
### 8b. What Is Ready to Implement (Low Risk)
|
||||
|
||||
1. **Schema additions** (`option`, `decision` kinds; `alternative_of`, `contains_option` edges): Trivial — new enum values. No breaking changes. No validation logic changes needed (new enum values are valid per Zod).
|
||||
2. **Prompt rule additions**: Straightforward — 4–5 sentences of explicit instruction for the model. Low risk, high clarity.
|
||||
3. **Deterministic decomposition templates** for option-based decisions: Can be added to `buildDecompositionTemplates()` in `apply-proposal.js` when a decision node is active.
|
||||
|
||||
### 8c. What Would Benefit from a Follow-Up Experiment (Medium Risk)
|
||||
|
||||
1. **Option-based reasoning pattern**: The current reasoning pattern system (`decision`, `explanation`, `contradiction`, etc.) could benefit from an explicit `option_comparison` pattern that governs how the engine reasons over option nodes.
|
||||
2. **Consequence propagation through option structure**: How should resolving one option's unknown propagate? If we resolve "the two engineers won't leave" for Option A, does that affect the comparison with Option B? This needs design.
|
||||
3. **Baseline visibility in reasoning patterns**: The `prioritisation` pattern could be extended to explicitly consider do-nothing baselines when an active reasoning context involves decisions.
|
||||
|
||||
### 8d. What Is Not Ready / Needs More Investigation (High Risk)
|
||||
|
||||
1. **Interim strategy via `compares_with` repurposing**: While technically possible, giving it alternative semantics risks confusion with the comparison semantics that already exist for observation comparisons (e.g., "compare two months' sales data"). A dedicated edge type is strongly preferred.
|
||||
2. **Automatic baseline detection**: Whether the model can infer a do-nothing baseline without explicit user framing needs testing. Some answers imply it; others don't. The prompt would need precise trigger conditions.
|
||||
3. **Multi-option decisions (> 2 options)**: How should N competing alternatives be represented? Linear chains of `alternative_of` edges, or a star topology centered on the decision node? This needs design.
|
||||
|
||||
### 8e. Recommended Next Experiment
|
||||
|
||||
**Experiment 60A.2 (proposed): Implement the three-primitive addition and verify 59B.4 structure.**
|
||||
|
||||
- Add `option`, `decision` kinds; `alternative_of`, `contains_option` edges to schema.
|
||||
- Add 4 prompt rules for option creation.
|
||||
- Run 59B.4 scenario against updated engine.
|
||||
- Measure: do both options appear as structural entities? Can the do-nothing baseline be represented? Is the comparison question structurally grounded?
|
||||
|
||||
---
|
||||
|
||||
## Diagnosis Conclusion
|
||||
|
||||
### The Answer to the Research Question
|
||||
|
||||
**The current graph vocabulary lacks an adequate first-class representation for alternatives and decisions.** It is not primarily a prompt problem — it is a schema problem. The model cannot represent what the schema does not define. Without `option` or `decision` node kinds and without `alternative_of` edge semantics, any dual-option input will always collapse into undifferentiated unknown prose.
|
||||
|
||||
### Evidence Chain
|
||||
|
||||
1. **Schema analysis**: Zero node kinds in SituationKind semantically represent choices or alternatives. The 9 available kinds cover observations, states, metrics, relationships, assumptions, unknowns, and conclusions — but no "option" or "decision."
|
||||
2. **Edge analysis**: `compares_with` exists but has no alternative-specific semantics. No dedicated "this option is one of the choices for this decision" relationship type exists.
|
||||
3. **Prompt analysis**: None of the 32 rules instruct the model to create structural options. The word "decision" in rule #7 refers to *a question about a decision*, not *a structural representation of the decision*.
|
||||
4. **Empirical evidence**: Experiments 59B.2–59B.4 consistently showed the same pattern — explicit dual-option input collapsed into a single undifferentiated unknown node, regardless of how strongly the user framed the comparison.
|
||||
5. **Reuse analysis**: All four candidate reuse strategies (`state`, `observation`, repurposed `compares_with`, `transition`) are either semantically incorrect or incomplete without schema-level support.
|
||||
|
||||
### Why Adding Prompt Rules Without Schema Changes Would Not Help
|
||||
|
||||
The model follows instructions precisely. If no instruction references a concept that doesn't exist in the vocabulary, the model cannot invent it. Telling the model to "create option nodes" without a valid `kind` value would cause validation errors. Repurposing existing kinds requires both schema changes (new enum values) and prompt rules anyway — so the schema change is unavoidable regardless of approach.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Add two node kinds (`option`, `decision`) and two edge relationships (`alternative_of`, `contains_option`) to the schema. Add four prompt rules for option creation. Total impact: ~25 lines of code. This addresses the root cause rather than treating symptoms.
|
||||
|
||||
---
|
||||
|
||||
## Summary Tables
|
||||
|
||||
### Key Findings Matrix
|
||||
|
||||
| Finding | Evidence | Confidence |
|
||||
|---------|----------|-----------|
|
||||
| Schema lacks option/decision primitives | schema.js SituationKind has 9 kinds, none for options | HIGH — direct code analysis |
|
||||
| `compares_with` has no alternative semantics | No documented semantics; used generically | HIGH — code + experiment history |
|
||||
| Prompt has no option creation rules | prompt-builder.js rules 1–32, no mention of options | HIGH — direct code analysis |
|
||||
| Empirical pattern persists across experiments | 59B.2, 59B.3, 59B.4 all same failure mode | HIGH — observed results |
|
||||
| Schema-only fix is ~25 lines | Two kinds + two relationships + four rules | HIGH — direct enumeration |
|
||||
|
||||
### Vocabulary Gap Summary
|
||||
|
||||
| Needed Primitive | Exists? | If not: What to Add |
|
||||
|-----------------|---------|-------------------|
|
||||
| Decision point representation | ❌ No | `decision` node kind |
|
||||
| Option / alternative entity | ❌ No | `option` node kind |
|
||||
| "This option belongs to this decision" link | ❌ No | `contains_option` edge type |
|
||||
| "These options compete" link | ❌ No (partial: `compares_with` exists but wrong semantics) | `alternative_of` edge type |
|
||||
| Do-nothing baseline representation | ❌ No | `is_baseline` flag on option nodes |
|
||||
| Option consequence attachment | ⚠️ Partially (via existing edges, if options existed) | N/A (works once options exist) |
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed during experiment: NO
|
||||
Validator changed during experiment: NO
|
||||
Vitest run: NO
|
||||
Ollama calls: 0
|
||||
Dev server disturbed: NO
|
||||
Read-only diagnosis: YES
|
||||
@@ -0,0 +1,659 @@
|
||||
# Experiment 60A.2 — Choosing the Minimum Decision Representation
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Type:** READ-ONLY ARCHITECTURE DESIGN — No production code changes, no API calls, no test runs.
|
||||
**Following:** 60A.1 which diagnosed that the vocabulary lacks both a decision node kind and an option node kind.
|
||||
|
||||
## Objective
|
||||
|
||||
Choose the smallest semantically honest graph structure that can represent:
|
||||
|
||||
```text
|
||||
Decision:
|
||||
Relocate or stay put?
|
||||
|
||||
Option A (Relocate):
|
||||
- save £2M/year
|
||||
- two senior engineers leave
|
||||
- up to two months delay
|
||||
|
||||
Option B (Stay put):
|
||||
- retain both engineers
|
||||
- avoid delivery disruption
|
||||
- continue paying extra £2M/year
|
||||
```
|
||||
|
||||
and later allow graph-only reasoning to compare the alternatives without reparsing the user's prose.
|
||||
|
||||
Three candidates evaluated. Not implemented. No code changed.
|
||||
|
||||
## Context Sources Loaded
|
||||
|
||||
1. `docs/current-handoff.md` (sections 59B series, current-state)
|
||||
2. `docs/experiment-60a1.md` (full vocabulary gap diagnosis)
|
||||
3. `lib/graph/schema.js` (exact schema: 9 node kinds, 7 statuses, 10 edge types)
|
||||
4. `lib/graph/prompt-builder.js` (exact rules 1–32 + additional guidance)
|
||||
5. `docs/experiment-59b4.md` (full results showing collapse of explicit dual-option into single unknown)
|
||||
|
||||
## Candidates Evaluated
|
||||
|
||||
### CANDIDATE A — DECISION + OPTION
|
||||
|
||||
**Conceptual shape:**
|
||||
|
||||
```text
|
||||
[decision: "Which option leaves us better off overall?"]
|
||||
├── [option: Relocate]
|
||||
│ ├── (consequences on relocate via existing edges)
|
||||
│ └── is_baseline: false
|
||||
└── [option: Stay put]
|
||||
├── (consequences on stay-put via existing edges)
|
||||
└── is_baseline: true
|
||||
```
|
||||
|
||||
**Required new primitives:**
|
||||
|
||||
| Primitive | Type | Value | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `decision` | node kind | SituationKind enum value | Represents the decision point requiring choice |
|
||||
| `option` | node kind | SituationKind enum value | Represents a choice available within this decision |
|
||||
| `contained_in` | edge relationship | SituationRelationship enum value | Links option → its parent decision (or unknown) |
|
||||
| `is_baseline` | optional field on option nodes | boolean | Marks the do-nothing / current-state default |
|
||||
|
||||
**Total: 2 node kinds + 1 edge type + 1 optional field type = 4 new primitives**
|
||||
|
||||
### Assessment
|
||||
|
||||
#### 1. Semantic honesty: HIGH
|
||||
|
||||
Each primitive means what it claims to mean:
|
||||
- `decision` = a decision point requiring choice between alternatives — clear, unambiguous
|
||||
- `option` = a specific choice available within this decision — clear, distinct from state (which asserts what *is*)
|
||||
- `contained_in` = "this option is contained within this decision" — natural parent-child semantics
|
||||
- `is_baseline` on options = marks the default/current-state alternative — unambiguous
|
||||
|
||||
No stretching of existing concepts. Each new concept fills a genuine vocabulary gap identified in 60A.1.
|
||||
|
||||
#### 2. Recoverability: FULL
|
||||
|
||||
| Query | How recovered |
|
||||
|-------|---------------|
|
||||
| "there is a decision" | Any node with `kind = decision` |
|
||||
| "what the alternatives are" | All nodes where `contained_in → that decision` and `kind = option` |
|
||||
| "which consequences belong to which option" | Existing edges from option nodes (causes/may_cause/etc.) — each consequence's `fromNodeId` is unambiguous |
|
||||
|
||||
All three independently recoverable via graph traversal with no textual parsing.
|
||||
|
||||
#### 3. Decision lifecycle: NATIVE
|
||||
|
||||
| Lifecycle event | How expressed |
|
||||
|-----------------|---------------|
|
||||
| decision still open | `decision` node status = unknown (or status of option nodes = unknown) |
|
||||
| decision resolved / option chosen | One or more option nodes transition to a chosen/resolved status |
|
||||
| new option added later | Add another `option` node with `contained_in → the same decision` |
|
||||
| option removed/rejected | Option node status = contradicted, or edge removal — no abuse needed |
|
||||
|
||||
All four states supported without any semantic workarounds. The distinction between "open" and "resolved" is naturally expressed through standard status transitions on nodes that already exist at the right structural level.
|
||||
|
||||
#### 4. Question compatibility: CLEAN
|
||||
|
||||
The `decision` node can carry a label/question that maps directly to `selectedQuestion`:
|
||||
- `questionNodeId` → the decision node's ID (or the active unknown within it)
|
||||
- The question text lives on the decision node itself ("Which option leaves us better off overall?")
|
||||
- No duplication of decision state — the single decision node IS the state
|
||||
|
||||
No conflict with existing `unknown` nodes because decisions are a distinct structural concept.
|
||||
|
||||
#### 5. Consequence attachment: YES
|
||||
|
||||
Consequences attach directly to option nodes via existing edge types (`causes`, `may_cause`, `weakens`, etc.). Each consequence's `fromNodeId` explicitly identifies which option it belongs to. No ambiguity, no grouping required.
|
||||
|
||||
#### 6. Baseline representation: CLEAN
|
||||
|
||||
One option node carries `is_baseline: true`. The decision context naturally includes "do nothing" as a special option type. No inference needed — explicit structural marking.
|
||||
|
||||
If we only need the label/consequences to carry baseline meaning (without an explicit flag), that is also feasible because the option labeled "stay put" or "current state" conveys this semantically. The boolean field is useful but not strictly required for the basic case.
|
||||
|
||||
#### 7. Minimality: 4 new primitives
|
||||
|
||||
```
|
||||
new node kinds: decision, option (2)
|
||||
new relationships: contained_in (1)
|
||||
new fields: is_baseline on option nodes (1 optional field type)
|
||||
```
|
||||
|
||||
Prompt rules are not counted as schema primitives per the criteria.
|
||||
|
||||
#### 8. Semantic overload: NONE
|
||||
|
||||
No existing concept is stretched:
|
||||
- `decision` fills a genuinely missing vocabulary slot
|
||||
- `option` fills a genuinely missing vocabulary slot
|
||||
- `contained_in` uses natural parent-child semantics
|
||||
- `is_baseline` is a metadata flag, not a repurposed concept
|
||||
|
||||
### 59B.4 Paper Graph (Candidate A)
|
||||
|
||||
```text
|
||||
[decision: "Which option leaves us better off overall?"]
|
||||
id: n_relocate_or_stay_decision
|
||||
kind: decision
|
||||
status: unknown
|
||||
label: "Relocate versus stay-put comparison"
|
||||
|
||||
[option: Relocate]
|
||||
id: n_option_relocate
|
||||
kind: option
|
||||
status: unknown
|
||||
contained_in: n_relocate_or_stay_decision
|
||||
is_baseline: false
|
||||
|
||||
[option: Stay put]
|
||||
id: n_option_stay_put
|
||||
kind: option
|
||||
status: unknown
|
||||
contained_in: n_relocate_or_stay_decision
|
||||
is_baseline: true
|
||||
|
||||
Consequences (each on its own structural node, attached to correct option):
|
||||
|
||||
[metric: "Annual savings from relocation"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Two senior engineers leave"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Up to two months delivery delay"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Both engineers retained"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[observation: "Avoid delivery disruption"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[metric: "Continuing extra £2M/year"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_stay_put
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### CANDIDATE B — UNKNOWN + OPTION
|
||||
|
||||
**Conceptual shape:**
|
||||
|
||||
```text
|
||||
[unknown: "Which option leaves us better off overall?"]
|
||||
id: n_active_unknown (existing infrastructure)
|
||||
├── [option: Relocate]
|
||||
│ ├── (consequences via existing edges)
|
||||
│ └── is_baseline: false (optional)
|
||||
└── [option: Stay put]
|
||||
├── (consequences via existing edges)
|
||||
└── is_baseline: true
|
||||
```
|
||||
|
||||
**Required new primitives:**
|
||||
|
||||
| Primitive | Type | Value | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `option` | node kind | SituationKind enum value | Represents a choice available within this decision context |
|
||||
| `contained_in` | edge relationship | SituationRelationship enum value | Links option → its parent decision context (which is the existing `unknown`) |
|
||||
| `is_baseline` | optional field on option nodes | boolean | Marks the do-nothing / current-state default |
|
||||
|
||||
**Total: 1 node kind + 1 edge type + 1 optional field type = 3 new primitives**
|
||||
|
||||
One fewer primitive than Candidate A because it reuses the existing `unknown` node as the decision context instead of creating a new `decision` node kind.
|
||||
|
||||
### Assessment
|
||||
|
||||
#### 1. Semantic honesty: MEDIUM
|
||||
|
||||
- `option` = clear, means what it says
|
||||
- `contained_in` = natural parent-child semantics (same as A)
|
||||
- `is_baseline` on options = clear
|
||||
|
||||
The honest assessment is that `unknown` carries decision context in this candidate — and `unknown` semantically means "unresolved question/uncertainty." The overlap between "decision point" and "unresolved question" is partial: every decision with alternatives implies an unresolved question, but not every unresolved question is a decision. This means the `unknown` node does double duty (both uncertainty and decision), which is imperfect but not contradictory because both concepts share the unresolved state.
|
||||
|
||||
This is MEDIUM, not LOW, because:
|
||||
- The overlap is natural (decisions inherently involve uncertainty)
|
||||
- No semantic contradiction is introduced — `unknown` status correctly reflects that the comparison hasn't been resolved yet
|
||||
- A future status transition on `unknown` → `resolved` naturally resolves both aspects simultaneously
|
||||
|
||||
#### 2. Recoverability: FULL
|
||||
|
||||
| Query | How recovered |
|
||||
|-------|---------------|
|
||||
| "there is a decision" | Any `unknown` node with children of kind `option` (or more conservatively: any `unknown` node that has option-type descendants) |
|
||||
| "what the alternatives are" | All nodes where `contained_in → that unknown` and `kind = option` |
|
||||
| "which consequences belong to which option" | Same as A — edges from each option node are unambiguous |
|
||||
|
||||
The recoverability is FULL because:
|
||||
- If a `unknown` has children of kind `option`, it structurally represents a decision (the question *is* the decision context)
|
||||
- This inference is deterministic and graph-only, requiring no text parsing
|
||||
- Consequence attachment to specific options works identically to A
|
||||
|
||||
**Caveat:** Recovering "there is a decision" requires checking for the presence of `option` children. Without them, the `unknown` node means exactly what it always meant (a generic uncertainty). This is deterministic but not as direct as A's single-node lookup (`kind = decision`). The criterion still rates FULL because recovery works correctly — just with an extra traversal step rather than a kind-check.
|
||||
|
||||
#### 3. Decision lifecycle: NATIVE
|
||||
|
||||
| Lifecycle event | How expressed |
|
||||
|-----------------|---------------|
|
||||
| decision still open | `unknown` node status remains unknown (existing mechanism) |
|
||||
| decision resolved / option chosen | `unknown` transitions to resolved; selected option could get a distinguished marker (status = supported, or additional flag) |
|
||||
| new option added later | Add another `option` node with `contained_in → same unknown` |
|
||||
| option removed/rejected | Option status = contradicted, edge removed — standard mechanisms |
|
||||
|
||||
The lifecycle is NATIVE because:
|
||||
- Open/resolved maps directly to existing `unknown` status transitions
|
||||
- The distinction between "decision context" and "concrete options" is handled by node kinds (unknown vs option), not statuses
|
||||
- No abuse of existing concepts is required
|
||||
- Adding/removing options uses standard graph operations
|
||||
|
||||
#### 4. Question compatibility: CLEAN
|
||||
|
||||
The `unknown` node already integrates with the engine's `selectedQuestion` mechanism:
|
||||
- `selectedQuestion.nodeId` → this unknown's ID (already how it works today)
|
||||
- The question text lives in the unknown's label/description
|
||||
- No duplication of decision state — the single `unknown` node IS both the context and the question carrier
|
||||
|
||||
This is CLEAN because it reuses the exact same mechanism without extension. No new mapping logic needed.
|
||||
|
||||
#### 5. Consequence attachment: YES
|
||||
|
||||
Consequences attach directly to option nodes via existing edge types (`causes`, `may_cause`, `weakens`, etc.). Each consequence's endpoint explicitly identifies its parent option. Identical capability to Candidate A.
|
||||
|
||||
#### 6. Baseline representation: WORKABLE
|
||||
|
||||
Baseline is represented as one option node with a distinguishing feature (label or optional `is_baseline` flag). This works cleanly but requires the explicit marker because "stay put" and "relocate" are just labels without inherent baseline semantics. The label "stay put" *suggests* baseline but doesn't *encode* it — so either a flag or convention is needed to distinguish baseline options from active alternatives.
|
||||
|
||||
This is WORKABLE (not CLEAN) because:
|
||||
- Without `is_baseline`, the system would need convention-based detection ("the option whose label suggests current state") which is less robust
|
||||
- With `is_baseline`, it becomes clean — so it's close to clean but the field adds complexity
|
||||
|
||||
#### 7. Minimality: 3 new primitives
|
||||
|
||||
```
|
||||
new node kinds: option (1)
|
||||
new relationships: contained_in (1)
|
||||
new fields: is_baseline on option nodes (1 optional field type)
|
||||
```
|
||||
|
||||
One fewer primitive than A because it reuses `unknown` instead of creating a separate `decision` kind.
|
||||
|
||||
#### 8. Semantic overload: LOW
|
||||
|
||||
The one stretching concern is using `unknown` to carry both "unresolved question" and "decision context" meanings. As noted above, the overlap is natural (decisions inherently involve uncertainty), so this is LOW not MEDIUM. No other existing concepts are stretched.
|
||||
|
||||
### 59B.4 Paper Graph (Candidate B)
|
||||
|
||||
```text
|
||||
[unknown: "Which option leaves us better off overall?"]
|
||||
id: n_active_unknown
|
||||
kind: unknown
|
||||
status: unknown
|
||||
label: "Relocate versus stay-put net value comparison"
|
||||
|
||||
[option: Relocate]
|
||||
id: n_option_relocate
|
||||
kind: option
|
||||
status: unknown
|
||||
contained_in: n_active_unknown
|
||||
is_baseline: false
|
||||
|
||||
[option: Stay put]
|
||||
id: n_option_stay_put
|
||||
kind: option
|
||||
status: unknown
|
||||
contained_in: n_active_unknown
|
||||
is_baseline: true
|
||||
|
||||
Consequences (each on its own structural node, attached to correct option):
|
||||
|
||||
[metric: "Annual savings from relocation"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Two senior engineers leave"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Up to two months delivery delay"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Both engineers retained"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[observation: "Avoid delivery disruption"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[metric: "Continuing extra £2M/year"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_stay_put
|
||||
```
|
||||
|
||||
Note: The structural graph is identical to Candidate A except `decision` → `unknown`. Consequence edges are identical. This demonstrates that the choice between A and B is purely about whether we need a separate decision node kind, not about consequence representation.
|
||||
|
||||
---
|
||||
|
||||
### CANDIDATE C — OPTION PAIR ONLY
|
||||
|
||||
**Conceptual shape:**
|
||||
|
||||
```text
|
||||
[option: Relocate]
|
||||
↔ [option: Stay put]
|
||||
|
||||
(with `alternative_to` edge between them)
|
||||
(no decision-context node at all)
|
||||
```
|
||||
|
||||
**Required new primitives:**
|
||||
|
||||
| Primitive | Type | Value | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `option` | node kind | SituationKind enum value | Represents a choice/alternative (no parent context) |
|
||||
| `alternative_to` | edge relationship | SituationRelationship enum value | Links one option to its competing alternative |
|
||||
|
||||
**Total: 1 node kind + 1 edge type = 2 new primitives**
|
||||
|
||||
The absolute minimum in terms of new schema additions. No decision-context node. No baseline field. Just two options pointing at each other.
|
||||
|
||||
### Assessment
|
||||
|
||||
#### 1. Semantic honesty: LOW
|
||||
|
||||
- `option` on its own = "a choice" — clear
|
||||
- `alternative_to` between options = "these are alternatives" — but without any parent context, this edge type is ambiguous in the general graph: any two nodes could have an `alternative_to` edge, and there's no way to distinguish a structured decision pair from random mutual exclusion
|
||||
|
||||
The critical issue: an option node with only an `alternative_to` link to another option tells us nothing about WHAT the alternatives are for. Two floating option nodes could represent "which ice cream flavor?" or "which office location?" or "which delivery method?" — and there is no graph structure distinguishing these cases. This is a LOW (not very low) honesty rating because the primitives themselves mean something, but their structural relationship to each other is incomplete without parent context.
|
||||
|
||||
#### 2. Recoverability: POOR
|
||||
|
||||
| Query | How recovered | Result |
|
||||
|-------|---------------|--------|
|
||||
| "there is a decision" | ??? | **POOR** — no node carries decision context. The pair exists but what they're alternatives for is not in the graph |
|
||||
| "what the alternatives are" | Both option nodes (trivially) | FULL (but useless without knowing what they're alternatives for) |
|
||||
| "which consequences belong to which option" | Edge endpoints on each option node | FULL (same as A/B) |
|
||||
|
||||
The critical failure: "there is a decision" cannot be answered from the graph. Two options with `alternative_to` between them could represent anything — a pairwise comparison, historical alternatives, mutually exclusive facts. The structural context (the question being decided) is entirely absent.
|
||||
|
||||
#### 3. Decision lifecycle: AWKWARD
|
||||
|
||||
| Lifecycle event | How expressed | Assessment |
|
||||
|-----------------|---------------|------------|
|
||||
| decision still open | ??? | **AWKWARD** — no node to track the open/closed state of the decision itself |
|
||||
| decision resolved / option chosen | One option gets a distinguished status/marker | Workable but ad hoc — which marker? How does it relate to existing statuses? |
|
||||
| new option added later | Add another option with `alternative_to` edges to both existing options | Workable for 3+ options (fan-out) but no anchor for "these all belong to the same decision" |
|
||||
| option removed/rejected | Remove node or edge | Standard graph operation, not a problem |
|
||||
|
||||
The critical gap: without a parent context node, there is nothing that can be "open" or "resolved." The open/resolved distinction only applies at the decision level (the pair is still being compared), not at the individual option level. Options within an active comparison don't have their own lifecycle states independent of the comparison itself — they are either "active candidates" or "chosen," but distinguishing "active candidate" from "just a node with alternative edges to something else" requires external state.
|
||||
|
||||
#### 4. Question compatibility: WORKABLE
|
||||
|
||||
The question could theoretically live on one of the option nodes (e.g., the label/question on Option A describes why we're comparing). But this is ad hoc — there's no contract saying "the first/primary option in a pair carries the question." This would be convention, not schema-enforced.
|
||||
|
||||
WORKABLE because it can work with conventions but isn't clean because:
|
||||
- No single node carries both the question and the alternatives
|
||||
- Adding new options later creates ambiguity about which node should carry the question
|
||||
- The `selectedQuestion` mechanism expects a nodeId — that nodeId would be an option, not a decision context
|
||||
|
||||
#### 5. Consequence attachment: YES
|
||||
|
||||
Consequences attach to each option node identically to A/B. Each consequence's edge endpoint identifies its parent option. No issue here — this criterion passes across all three candidates equally.
|
||||
|
||||
#### 6. Baseline representation: WORKABLE
|
||||
|
||||
Without a parent decision context, there is no place to conventionally say "this is the do-nothing alternative." The baseline would have to be carried by:
|
||||
- The option label alone (semantic inference by consumers)
|
||||
- An `is_baseline` field on the option node itself (adds a field that C sought to avoid)
|
||||
|
||||
Either approach works but neither is clean. The first relies on text parsing; the second defeats the minimality argument of this candidate. This is WORKABLE because workarounds exist, but it exposes why C's minimalism is expensive semantically.
|
||||
|
||||
#### 7. Minimality: 2 new primitives
|
||||
|
||||
```
|
||||
new node kinds: option (1)
|
||||
new relationships: alternative_to (1)
|
||||
new fields: none
|
||||
```
|
||||
|
||||
The absolute smallest in raw primitive count, but the semantic cost (see criteria 1–4) makes this cheapness misleading.
|
||||
|
||||
#### 8. Semantic overload: LOW-MEDIUM
|
||||
|
||||
`alternative_to` is a new edge type that C would have to document as meaning "these two options compete for an unnamed decision." Without the parent context, this edge carries partial semantics only. The risk isn't overloading an existing concept (no existing concept is stretched) — the risk is that `alternative_to` becomes underspecified in practice because consumers can't answer "alternatives for what?" from the graph alone.
|
||||
|
||||
LOW-MEDIUM because no existing concept is overstretched, but the new edge type itself has incomplete semantics without parent context.
|
||||
|
||||
### 59B.4 Paper Graph (Candidate C)
|
||||
|
||||
```text
|
||||
[option: Relocate]
|
||||
id: n_option_relocate
|
||||
kind: option
|
||||
status: unknown
|
||||
label: "Relocate — save £2M/year, lose 2 engineers, delay 2 months"
|
||||
|
||||
[option: Stay put]
|
||||
id: n_option_stay_put
|
||||
kind: option
|
||||
status: unknown
|
||||
label: "Stay put — retain engineers, avoid disruption, continue £2M/year"
|
||||
|
||||
alternative_to: n_option_relocate ↔ n_option_stay_put
|
||||
|
||||
Consequences (on each option):
|
||||
|
||||
[metric: "Annual savings from relocation"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Two senior engineers leave"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Up to two months delivery delay"]
|
||||
may_cause → n_option_relocate
|
||||
|
||||
[observation: "Both engineers retained"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[observation: "Avoid delivery disruption"]
|
||||
causes → n_option_stay_put
|
||||
|
||||
[metric: "Continuing extra £2M/year"]
|
||||
value: 2000000, unit: "GBP/year"
|
||||
may_cause → n_option_stay_put
|
||||
```
|
||||
|
||||
Note: The consequence edges work identically to A/B. The structural gap is that neither option has any parent context — there is no graph structure answering "what decision are we making?"
|
||||
|
||||
---
|
||||
|
||||
## Decision Rule Application
|
||||
|
||||
Required satisfying conditions:
|
||||
|
||||
```
|
||||
1. both alternatives independently recoverable → A: YES, B: YES, C: PARTIAL (no context)
|
||||
2. consequences attach to one specific option → A: YES, B: YES, C: YES
|
||||
3. decision can remain open and later resolve → A: NATIVE, B: NATIVE, C: AWKWARD
|
||||
4. no severe semantic overload → A: NONE, B: LOW, C: LOW-MEDIUM
|
||||
5. do-nothing can be represented cleanly → A: CLEAN, B: WORKABLE, C: WORKABLE
|
||||
```
|
||||
|
||||
Candidate C fails criteria 1 (decision context not recoverable), 3 (no open/resolved lifecycle support), and produces misleading minimality due to semantic gaps.
|
||||
|
||||
Between A and B — both satisfy all five conditions. The question is which is smaller while still meeting all requirements.
|
||||
|
||||
**B wins on minimality (3 primitives vs 4) while satisfying all decision-rule conditions.**
|
||||
|
||||
The marginal semantic cost of using `unknown` as decision context (MEDIUM honesty, LOW overload) is justified because:
|
||||
- The overlap between "decision" and "uncertainty about a decision" is natural and non-contradictory
|
||||
- The engine already tracks the open/resolved state of `unknown` nodes — this maps exactly to the decision lifecycle
|
||||
- Question compatibility uses the existing `selectedQuestion` mechanism without extension
|
||||
|
||||
## Additional Question 1 — Is `alternative_to` Actually Needed?
|
||||
|
||||
**Answer: NO**
|
||||
|
||||
If both options are linked to the same decision context (whether that context is a `decision` node in Candidate A or an `unknown` node in Candidate B), the shared membership already implies they are alternatives of each other. An explicit `alternative_to` edge between options carries no unique semantics recoverable from the graph structure — any traversal from option A can reach option B through their shared parent, and the relationship is implicit in the tree topology.
|
||||
|
||||
An explicit edge would be useful for direct traversal (go straight from A to its alternatives without going up-and-down the tree), but it is semantically redundant with shared-parent membership. If added in a future iteration as an optional convenience edge, it should not be required for correctness.
|
||||
|
||||
**Verdict: NO — shared decision membership already implies alternatives.**
|
||||
|
||||
## Additional Question 2 — Is a Baseline Flag Actually Needed?
|
||||
|
||||
**Answer: NOT NEEDED YET**
|
||||
|
||||
"Stay put" is just another option whose meaning is carried by its label and consequences. The graph does not need an explicit `is_baseline` marker in the initial design because:
|
||||
|
||||
1. Labels ("Stay put", "Current state", "Status quo") carry sufficient semantic signal for both human consumption and simple heuristics
|
||||
2. Consequences of the baseline option (typically lower urgency, different causal patterns) are structurally distinct from active options
|
||||
3. A future heuristic could identify baselines by consequence-pattern analysis rather than requiring explicit markers
|
||||
|
||||
**Verdict: NOT NEEDED YET.** If baseline detection becomes important later, adding `is_baseline` is a one-field addition to the option schema that does not require any structural redesign.
|
||||
|
||||
---
|
||||
|
||||
## 59B.4 Paper Graph — Candidate Comparison Summary
|
||||
|
||||
All three candidates produce identical consequence edges (each consequence attached to its correct option). The difference is purely in how the decision context and option membership are structured:
|
||||
|
||||
| Aspect | A (Decision+Option) | B (Unknown+Option) | C (Option Pair) |
|
||||
|--------|---------------------|--------------------|-----------------|
|
||||
| Decision context | Dedicated `decision` node | Existing `unknown` node | None — implicit in pair |
|
||||
| Option membership | `contained_in → decision` | `contained_in → unknown` | `alternative_to` peer link |
|
||||
| Open/resolved state | On decision node | Via `unknown` status | Not tracked structurally |
|
||||
| New primitives | 2 kinds + 1 edge + 1 field | 1 kind + 1 edge + 1 field | 1 kind + 1 edge |
|
||||
| Semantic cost | None | LOW (unknown carries dual role) | MEDIUM (pairs have no context) |
|
||||
|
||||
---
|
||||
|
||||
## Final Architectural Choice
|
||||
|
||||
### B — UNKNOWN + OPTION
|
||||
|
||||
**Chosen because it is the smallest model that satisfies all five decision-rule conditions.**
|
||||
|
||||
Minimum node kinds: `option` (1 new kind; reuses existing `unknown`)
|
||||
Minimum relationships: `contained_in` (1 new relationship type)
|
||||
Minimum fields: none required in initial design (baseline detection by label/consequence pattern is feasible later)
|
||||
|
||||
### Why B over A?
|
||||
|
||||
A adds a separate `decision` node kind, which is semantically cleaner for the "what's the question?" layer but costs one additional primitive. The incremental cleanliness of B is justified because:
|
||||
- `unknown` naturally expresses "unresolved decision context" (the semantic overlap is natural, not forced)
|
||||
- Question compatibility uses existing `selectedQuestion` infrastructure without extension
|
||||
- Lifecycle mapping is identical to what the engine already tracks (open/resolved unknowns)
|
||||
|
||||
### Why B over C?
|
||||
|
||||
C fails on recoverability of decision context and open/resolved lifecycle. The cost savings (2 primitives vs 3) come at the expense of losing the question that makes two options meaningful as a pair. Two floating options are not a decision — they are just two things with a mutual-exclusion edge.
|
||||
|
||||
---
|
||||
|
||||
## Smallest Winning 59B.4 Graph
|
||||
|
||||
**Decision context:**
|
||||
```text
|
||||
[unknown: "Which option leaves us better off overall?"]
|
||||
kind: unknown (existing)
|
||||
status: unknown (existing)
|
||||
id: n_active_unknown
|
||||
label: "Relocate versus stay-put net value comparison"
|
||||
```
|
||||
|
||||
**Options:**
|
||||
```text
|
||||
[option: Relocate]
|
||||
kind: option (NEW)
|
||||
status: unknown
|
||||
contained_in → n_active_unknown (via new edge type)
|
||||
|
||||
[option: Stay put]
|
||||
kind: option (NEW)
|
||||
status: unknown
|
||||
contained_in → n_active_unknown (via new edge type)
|
||||
```
|
||||
|
||||
**Consequences (each on its own structural node):**
|
||||
|
||||
For relocate:
|
||||
- `metric` — "Annual savings from relocation" — value=2000000 GBP/year — may_cause → option_relocate
|
||||
- `observation` — "Two senior engineers leave" — may_cause → option_relocate
|
||||
- `observation` — "Up to two months delivery delay" — may_cause → option_relocate
|
||||
|
||||
For stay put:
|
||||
- `observation` — "Both engineers retained" — causes → option_stay_put
|
||||
- `observation` — "Avoid delivery disruption" — causes → option_stay_put
|
||||
- `metric` — "Continuing extra £2M/year" — value=2000000 GBP/year — may_cause → option_stay_put
|
||||
|
||||
**Relationships:**
|
||||
- 6 consequence edges (3 per option, using existing `causes`/`may_cause` types)
|
||||
- 2 membership edges: option_relocate.contained_in → unknown, option_stay_put.contained_in → unknown (new edge type)
|
||||
|
||||
**Graph-only recover decision:** YES — `unknown` node with `option` children IS the decision structure.
|
||||
|
||||
**Graph-only recover relocate:** YES — any node where `contained_in → n_active_unknown` and label contains "relocate."
|
||||
|
||||
**Graph-only recover stay-put:** YES — any node where `contained_in → n_active_unknown` and label contains "stay" or "current state."
|
||||
|
||||
**Graph-only attach consequences to correct option:** YES — each consequence edge's `fromNodeId` explicitly identifies the parent option.
|
||||
|
||||
**Decision can later resolve without semantic abuse:** YES — `unknown` transitions from `status=unknown` to `status=resolved`, and one option could get a distinguished marker (status=supported, or any existing convention). No abuse of unrelated statuses or node kinds required.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Readiness
|
||||
|
||||
### A — READY FOR BOUNDED IMPLEMENTATION
|
||||
|
||||
The minimum new primitives and semantics are precise enough to implement:
|
||||
|
||||
**Schema changes (exact):**
|
||||
```javascript
|
||||
// In SituationKind enum:
|
||||
option: "option" // a choice available within a decision context
|
||||
|
||||
// In SituationRelationship enum:
|
||||
contained_in: "contained_in" // this option is contained within a decision/unknown context
|
||||
|
||||
// In situationNodeSchema — optional on option nodes only:
|
||||
is_baseline: z.boolean().optional() // future extension, not required for v1
|
||||
```
|
||||
|
||||
**Prompt additions (4 sentences):**
|
||||
1. "When the answer presents competing alternatives for a decision, create one node of kind 'option' for each alternative."
|
||||
2. "Connect each option to its decision context node using relationship 'contained_in'."
|
||||
3. "If the answer references a do-nothing baseline, label the corresponding option clearly (e.g., 'Stay put', 'Current state'). Detection can be by label convention; no is_baseline field required in v1."
|
||||
4. "Attach consequences of each option to that option node using existing causal edges (causes/may_cause/etc.)."
|
||||
|
||||
**No schema-level change to:** `SituationStatus`, existing edge types, graph topology rules, validation logic beyond accepting the two new enum values.
|
||||
|
||||
**If one more design question were needed**, it would be: "Should `option` nodes themselves track a lifecycle status (e.g., `status=chosen`) or should resolution flow entirely through the parent `unknown` node?" For v1 implementation, this is deferred — existing statuses on options are sufficient for initial use.
|
||||
|
||||
---
|
||||
|
||||
## Exact Smallest Implementation Boundary
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed: NO
|
||||
Validator changed: NO
|
||||
Schema changed: NO
|
||||
Tests changed: NO
|
||||
Ollama calls: 0
|
||||
Live API calls: 0
|
||||
Vitest run: NO
|
||||
Dev server disturbed: NO
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updated
|
||||
|
||||
- `docs/experiment-60a2.md` (this file) — full evaluation of all three candidates, architectural choice, and rationale
|
||||
- `docs/current-handoff.md` — appended 60A.2 entry to the latest section
|
||||
@@ -0,0 +1,268 @@
|
||||
# Experiment 60A.4 — Native Two-Option Structure Live Validation
|
||||
|
||||
**Branch:** `feature/decision-options-v0.25`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Type:** LIVE RUN — Bounded single-call experiment to verify the model actually uses the new vocabulary in practice.
|
||||
**Following:** 60A.3 which committed `option` node kind and `contained_in` edge to production.
|
||||
|
||||
## Objective
|
||||
|
||||
Test only whether the live model represents both "relocate" and "stay put" as separate option nodes linked to one shared unresolved decision context — exactly the two-option case that previously collapsed.
|
||||
|
||||
## Context Sources Loaded
|
||||
|
||||
1. `docs/current-handoff.md` (sections 59B series, current-state)
|
||||
2. `docs/experiment-60a3.md` (commit: feat: add 'option' node kind and 'contained_in' edge — 60A.3)
|
||||
3. `docs/experiment-59b4.md` (exact previous regression case)
|
||||
4. Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
5. Canonical harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Pre-existing uncertainties:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed Answer (verbatim, exact 59B.4 answer)
|
||||
|
||||
> There are really two options now.
|
||||
>
|
||||
> Option 1 is relocate: we save £2 million per year, but two senior engineers leave and delivery could be delayed by up to two months.
|
||||
>
|
||||
> Option 2 is stay where we are: we keep both engineers and avoid the delivery disruption, but we continue paying the extra £2 million every year.
|
||||
>
|
||||
> I am deciding which of those two options leaves us better off overall.
|
||||
|
||||
## Configured Model
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A successful result should produce:
|
||||
|
||||
```text
|
||||
one unresolved decision-context unknown
|
||||
two option nodes
|
||||
- Relocate
|
||||
- Stay put
|
||||
two contained_in relationships from those options to the same unknown
|
||||
option-specific consequences preserved under the correct branch
|
||||
```
|
||||
|
||||
Do not require every consequence to become its own node in this first live regression. The primary question is whether the two alternatives now survive as first-class graph structure.
|
||||
|
||||
## Execution
|
||||
|
||||
One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`.
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| startCalls | 0 |
|
||||
| updateCalls | 1 |
|
||||
| totalCalls | 1 |
|
||||
| Retries | 0 |
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE
|
||||
|
||||
**HTTP status:** 200
|
||||
**Stage:** `update_applied`
|
||||
**Validation errors:** none
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
```
|
||||
|
||||
#### Added nodes
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "n_relocation_decision",
|
||||
"label": "Which option leaves us better off overall?",
|
||||
"description": "Decision uncertainty between relocating to Manchester or staying in London, so that we can determine the best path forward based on financial, personnel, and delivery impacts.",
|
||||
"kind": "unknown",
|
||||
"status": "unknown",
|
||||
"confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "n_option_relocate",
|
||||
"label": "Relocate to Manchester",
|
||||
"description": "Move engineering team to save £2M/year, but two senior engineers will leave and delivery may be delayed by up to two months.",
|
||||
"kind": "option",
|
||||
"status": "known",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "n_option_stay",
|
||||
"label": "Stay in London (Status Quo)",
|
||||
"description": "Keep the team in place to retain two senior engineers and avoid delivery disruption, but continue paying an extra £2M/year.",
|
||||
"kind": "option",
|
||||
"status": "known",
|
||||
"confidence": "high"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### Added edges
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "e-relocate-to-decision",
|
||||
"fromNodeId": "n_option_relocate",
|
||||
"toNodeId": "n_relocation_decision",
|
||||
"relationship": "contained_in",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "e-stay-to-decision",
|
||||
"fromNodeId": "n_option_stay",
|
||||
"toNodeId": "n_relocation_decision",
|
||||
"relationship": "contained_in",
|
||||
"confidence": "high"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### Selected question
|
||||
|
||||
**Question:** "What evidence would clarify which option leaves us better off overall?"
|
||||
**nodeId:** `n_relocation_decision`
|
||||
|
||||
### Resulting persistent graph (5 nodes, 3 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| n_savings_realism | unknown | unknown | Are the projected office savings from relocation realistic? |
|
||||
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
||||
| n_option_relocate | **option** | known | Relocate to Manchester |
|
||||
| n_option_stay | **option** | known | Stay in London (Status Quo) |
|
||||
|
||||
| Edge | From | To | Relationship |
|
||||
|------|------|----|-------------|
|
||||
| e-savings-realism→state | n_savings_realism | n_relocation_state | depends_on |
|
||||
| e-relocate-to-decision | n_option_relocate | n_relocation_decision | **contained_in** |
|
||||
| e-stay-to-decision | n_option_stay | n_relocation_decision | **contained_in** |
|
||||
|
||||
## Primary Assessment
|
||||
|
||||
### 1. Decision context
|
||||
|
||||
**EXPLICIT SHARED DECISION UNKNOWN**
|
||||
|
||||
Node `n_relocation_decision` (kind=unknown, status=unknown) with label "Which option leaves us better off overall?" represents a single shared unresolved decision context that both options feed into via `contained_in`.
|
||||
|
||||
### 2. Relocate branch
|
||||
|
||||
**OPTION NODE**
|
||||
|
||||
Node `n_option_relocate`, kind=`option`, status=`known`, label="Relocate to Manchester". Description preserves all three consequences: "save £2M/year, but two senior engineers will leave and delivery may be delayed by up to two months."
|
||||
|
||||
### 3. Stay-put branch
|
||||
|
||||
**OPTION NODE**
|
||||
|
||||
Node `n_option_stay`, kind=`option`, status=`known`, label="Stay in London (Status Quo)". Description preserves all three consequences: "retain two senior engineers and avoid delivery disruption, but continue paying an extra £2M/year."
|
||||
|
||||
### 4. Membership
|
||||
|
||||
**BOTH CORRECT**
|
||||
|
||||
Both option nodes link to `n_relocation_decision` via `contained_in` edges. Both have confidence=high. Both edges are explicitly typed and directional (from option → decision).
|
||||
|
||||
### 5. Consequence attribution
|
||||
|
||||
**BOTH BRANCHES CLEAR**
|
||||
|
||||
Relocate description: "save £2M/year, but two senior engineers will leave and delivery may be delayed by up to two months." — all three consequences attributable.
|
||||
|
||||
Stay-put description: "retain two senior engineers and avoid delivery disruption, but continue paying an extra £2M/year." — all three consequences attributable.
|
||||
|
||||
Branch ownership is unambiguous because each consequence set lives within a distinct option node that only one `contained_in` edge reaches.
|
||||
|
||||
## Graph-Only Recoverability
|
||||
|
||||
| Question | Answer |
|
||||
|----------|--------|
|
||||
| Can graph-only reasoning recover Relocate as an option? | YES |
|
||||
| Can graph-only reasoning recover Stay put as an option? | YES |
|
||||
| Can it tell both belong to the same decision? | YES — both have `contained_in` → `n_relocation_decision` |
|
||||
| Can it distinguish which consequences belong to which option? | YES — each consequence lives in a distinct option node's description, reached by a unique `contained_in` edge |
|
||||
|
||||
## Secondary Assessment
|
||||
|
||||
### Savings-realism node
|
||||
|
||||
**REMAINS OPEN** — `n_savings_realism` remains status=unknown, unchanged. The answer did not address savings realism so the engine correctly left it unresolved (no updatedNodes).
|
||||
|
||||
### Selected question
|
||||
|
||||
**GOOD** — "What evidence would clarify which option leaves us better off overall?" continues the comparison and investigates a consequence that could distinguish the options. It targets `n_relocation_decision` which is the correct decision context node.
|
||||
|
||||
## Classification: A — NATIVE TWO-OPTION STRUCTURE CONFIRMED
|
||||
|
||||
Two separate `option` nodes exist, both are linked via `contained_in` to the same unresolved decision context (`n_relocation_decision`), and both branches are graph-recoverable with consequences attributable to the correct branch.
|
||||
|
||||
### Why:
|
||||
|
||||
All five classification A requirements are met:
|
||||
1. **Two nodes with kind=option:** ✅ `n_option_relocate` and `n_option_stay`
|
||||
2. **One node representing the unresolved decision context:** ✅ `n_relocation_decision` (kind=unknown, status=unknown)
|
||||
3. **Two contained_in relationships:** ✅ Both edges explicitly typed and directional
|
||||
4. **Both contained_in relationships target that same decision node:** ✅ Both → `n_relocation_decision`
|
||||
5. **Both option branches recoverable from graph alone:** ✅ Each consequence set lives in a distinct option node reached by a unique edge
|
||||
|
||||
The model used the new vocabulary correctly, structurally, and completely for this case. The question "Which option leaves us better off overall?" naturally captures the user's intent ("I am deciding which of those two options leaves us better off overall").
|
||||
|
||||
## What the engine understood correctly:
|
||||
|
||||
1. **Dual-option decomposition:** The answer explicitly names two options and the model created two corresponding `option` nodes — one for each branch.
|
||||
2. **Shared decision context:** Both options are linked to a single unresolved unknown node representing the decision question, not two separate decision nodes.
|
||||
3. **Containment semantics:** The model correctly used `contained_in` as the membership relationship from option → decision (not `causes`, `depends_on`, or other existing edge types).
|
||||
4. **Consequence attribution per branch:** Each option node's description carries its own complete set of consequences — no cross-contamination or collapse.
|
||||
5. **Decision-question alignment:** The selected question "Which option leaves us better off overall?" mirrors the user's stated intent and targets the correct decision context node.
|
||||
|
||||
## What it still flattened or omitted:
|
||||
|
||||
1. **No savings-realism update** — expected; the answer did not address it, so no mutation was needed.
|
||||
2. **No dedicated consequence nodes** — consequences remain embedded in option descriptions rather than as separate graph nodes. This is acceptable per the experiment scope ("Do not require every consequence to become its own node").
|
||||
3. **n_savings_realism still open** — correct behavior but means the investigation has diverged into two parallel threads (savings realism + relocation decision) without cross-linkage.
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The live model CAN create native two-option graph structure when the user explicitly presents two alternatives.
|
||||
2. Both options survive as first-class `option` nodes with structural membership (`contained_in`) to a shared decision context.
|
||||
3. Consequence ownership is structurally unambiguous via option node separation — downstream graph-only reasoning can recover both branches and their distinct consequences.
|
||||
4. The production prompt, after 60A.3's vocabulary additions, successfully steers the model toward using `option` + `contained_in` for dual-option decisions without any code changes beyond the schema addition.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability** — one run only; cold-start variance may produce different outcomes on repeated runs.
|
||||
2. **Cross-domain generalisation** — single domain case only (relocation decision).
|
||||
3. **Three-or-more options** — does not test whether the model scales option creation beyond two.
|
||||
4. **Baseline vs action discrimination** — both options have status=known and confidence=high; the model did not distinguish "active choice" from "status quo."
|
||||
5. **Downstream decision scoring** — this experiment stops at structural representation; it does not test whether the engine can now use these option nodes for comparison, weighting, or recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Validator changed during experiment: NO
|
||||
## Harness changed during experiment: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,201 @@
|
||||
# Experiment 60A.5 — Option-Specific Consequence Structure Confirmation
|
||||
|
||||
**Branch:** `feature/decision-options-v0.25`
|
||||
**Starting HEAD:** 3db6f40 (experiment: validate native option structure live)
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Complete
|
||||
**Type:** LIVE RUN — Bounded single-call experiment to verify whether known consequences for two alternatives become independently recoverable graph structure attached to the correct option.
|
||||
**Following:** 60A.4 which confirmed native two-option structure with contained_in edges.
|
||||
|
||||
## Objective
|
||||
|
||||
When the user explicitly separates known consequences for two alternatives ("If we relocate... If we stay put..."), does the live engine create consequence structure that remains attributable to the correct option — without converting known material into new unresolved unknowns?
|
||||
|
||||
**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
**Pre-existing uncertainty:** `n_savings_realism` (status=unknown)
|
||||
|
||||
## Fixed Answer (verbatim, exact)
|
||||
|
||||
> There are two options.
|
||||
>
|
||||
> If we relocate, we save £2 million per year, two senior engineers will definitely leave, and delivery will be delayed by no more than two months.
|
||||
>
|
||||
> If we stay put, we retain both senior engineers, avoid the relocation delay, and continue paying the extra £2 million every year.
|
||||
>
|
||||
> Those consequences are known. What I still do not know is which option leaves us better off overall.
|
||||
|
||||
## Configured Model
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A strong result should preserve: one shared decision-context unknown; option: relocate; option: stay put; and create independently recoverable consequence/evidence structure associated with the correct option.
|
||||
|
||||
Known consequences ≠ unresolved decision. The engine should not turn them into new unknowns merely because they matter to the decision.
|
||||
|
||||
## Execution
|
||||
|
||||
One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`.
|
||||
|
||||
**Call accounting:** startCalls=0, updateCalls=2 (1 rejection at proposal_validation due to null confidence fields + 1 successful inference), totalCalls=2. Only the successful run carries assessable inference results.
|
||||
|
||||
## Results — Successful Run
|
||||
|
||||
### UPDATE
|
||||
|
||||
**HTTP status:** 200
|
||||
**Stage:** update_applied (implied by success path)
|
||||
**Validation errors:** none
|
||||
|
||||
#### Answer Meaning
|
||||
|
||||
Not explicitly printed by harness for this path, but proposal mutations are available below.
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [{nodeId:"n_savings_realism", previousStatus:"unknown", newStatus:"resolved", previousValue:null, newValue:"£2m/year", reason:"User explicitly states the £2m annual saving consequence is known."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{id:"opt_relocate", label:"Relocate to Manchester", description:"Move the engineering team to Manchester. Consequences: saves £2m/year, loses two senior engineers, delays delivery by up to two months.", kind:"option", status:"provisional", confidence:"high"},
|
||||
{id:"opt_stay", label:"Stay in London", description:"Remain in the current office. Consequences: retain both senior engineers, avoid relocation delay, continue paying the extra £2m/year.", kind:"option", status:"provisional", confidence:"high"},
|
||||
{id:"n_overall_tradeoff", label:"Which option leaves us better off overall?", description:"Uncertainty regarding the net trade-off between relocation savings and retention/delay costs compared to staying put, needed to decide on the best path forward.", kind:"unknown", status:"unknown", confidence:"high"}
|
||||
]
|
||||
|
||||
addedEdges: [
|
||||
{fromNodeId:"opt_relocate", toNodeId:"n_overall_tradeoff", relationship:"contained_in"},
|
||||
{fromNodeId:"opt_stay", toNodeId:"n_overall_tradeoff", relationship:"contained_in"},
|
||||
{fromNodeId:"n_overall_tradeoff", toNodeId:"n_savings_realism", relationship:"depends_on"}
|
||||
]
|
||||
```
|
||||
|
||||
#### structuralActionRequired
|
||||
|
||||
null (not populated by model)
|
||||
|
||||
#### selectedQuestion
|
||||
|
||||
**Question:** "What evidence would clarify which option leaves us better off overall?"
|
||||
**nodeId:** n_overall_tradeoff
|
||||
|
||||
### Resulting persistent graph (5 nodes, 4 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| n_savings_realism | unknown | resolved | Are the projected office savings from relocation realistic? |
|
||||
| opt_relocate | option | provisional | Relocate to Manchester |
|
||||
| opt_stay | option | provisional | Stay in London |
|
||||
| n_overall_tradeoff | unknown | unknown | Which option leaves us better off overall? |
|
||||
|
||||
| Edge | From | To | Relationship |
|
||||
|------|------|----|-------------|
|
||||
| e-sr-to-state | n_savings_realism | n_relocation_state | depends_on |
|
||||
| opt-rel-to-trad | opt_relocate | n_overall_tradeoff | contained_in |
|
||||
| opt-stay-to-trad | opt_stay | n_overall_tradeoff | contained_in |
|
||||
| trad-to-savings | n_overall_tradeoff | n_savings_realism | depends_on |
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Decision structure: NATIVE TWO-OPTION STRUCTURE PRESERVED
|
||||
|
||||
Both option nodes survive with kind=option and are linked via contained_in to the same unresolved decision context (n_overall_tradeoff). Two minor differences from 60A.4:
|
||||
- Option statuses are provisional instead of known (both have confidence=high, so ambiguity is low)
|
||||
- Node IDs use lowercase abbreviations (opt_relocate/opt_stay vs n_option_relocate/n_option_stay)
|
||||
|
||||
### 2. Relocate consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
|
||||
|
||||
| Consequence | Present? | In graph? |
|
||||
|-------------|----------|-----------|
|
||||
| £2m/year saving | YES | "saves £2m/year" in opt_relocate.description |
|
||||
| two senior engineers leave | YES | "loses two senior engineers" in opt_relocate.description |
|
||||
| <= two months delivery delay | YES | "delays delivery by up to two months" in opt_relocate.description |
|
||||
|
||||
### 3. Stay-put consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
|
||||
|
||||
| Consequence | Present? | In graph? |
|
||||
|-------------|----------|-----------|
|
||||
| retain both engineers | YES | "retain both senior engineers" in opt_stay.description |
|
||||
| avoid relocation delay | YES | "avoid relocation delay" in opt_stay.description |
|
||||
| continue paying extra £2m/year | YES | "continue paying the extra £2m/year" in opt_stay.description |
|
||||
|
||||
### 4. Epistemic correctness: CORRECT
|
||||
|
||||
- Known consequences remain known (embedded in option descriptions, not unresolved)
|
||||
- n_savings_realism correctly resolved with newValue="£2m/year"
|
||||
- No consequences incorrectly converted to new unknowns
|
||||
- Only one new unknown created for the decision question — correct epistemic state
|
||||
|
||||
### 5. Option attribution: CLEAR FOR BOTH OPTIONS
|
||||
|
||||
Graph makes it possible to tell which option each consequence belongs to:
|
||||
- opt_relocate consequences reachable via its own contained_in edge to n_overall_tradeoff
|
||||
- opt_stay consequences reachable via its own contained_in edge to n_overall_tradeoff
|
||||
- No cross-contamination or ambiguity
|
||||
|
||||
### 6. Relationship direction: SEMANTICALLY CLEAR
|
||||
|
||||
| From | To | Relationship | Assessment |
|
||||
|------|----|-------------|------------|
|
||||
| opt_relocate | n_overall_tradeoff | contained_in | Clear — relocation is a candidate for the decision |
|
||||
| opt_stay | n_overall_tradeoff | contained_in | Clear — staying put is a candidate for the decision |
|
||||
| n_overall_tradeoff | n_savings_realism | depends_on | Workable but slightly odd direction — the unknown "depends on" a resolved node (epistemically inverted) |
|
||||
|
||||
### 7. Graph-only recoverability
|
||||
|
||||
| Question | Answer |
|
||||
|----------|--------|
|
||||
| Recover Relocate option | YES — node kind=option, label="Relocate to Manchester" |
|
||||
| Recover Stay-put option | YES — node kind=option, label="Stay in London" |
|
||||
| Recover Relocate consequences | PARTIAL — present in opt_relocate.description (structured field on graph node) |
|
||||
| Recover Stay-put consequences | PARTIAL — present in opt_stay.description (structured field on graph node) |
|
||||
| Tell which consequence belongs to which option | YES — each description attached to a distinct option node reached by its own contained_in edge |
|
||||
|
||||
### 8. Selected question: GOOD
|
||||
|
||||
"What evidence would clarify which option leaves us better off overall?" targets n_overall_tradeoff, the correct decision context node. Aligns with user's stated unresolved issue. No penalty for asking about a genuinely decision-relevant comparison criterion.
|
||||
|
||||
## Classification: B — CONSEQUENCE STRUCTURE PARTIAL
|
||||
|
||||
Both option branches survive as structurally distinct nodes (kind=option) with correct containment relationships to a shared decision context. All known consequences for both options are present and correctly attributable. However, consequences remain embedded in option descriptions rather than as independent graph nodes with typed edges — a downstream reasoning step would need to parse opt_relocate.description vs opt_stay.description text to extract specific consequence values.
|
||||
|
||||
This is an improvement over 59B.4 (where do-nothing had no structural presence) but does not reach A-level because consequences are not first-class independently recoverable nodes.
|
||||
|
||||
## What the engine understood correctly:
|
||||
|
||||
1. **Dual-option decomposition:** Two distinct option nodes created with kind=option — one per branch
|
||||
2. **Shared decision context:** Both options linked to single n_overall_tradeoff via contained_in edges
|
||||
3. **Consequence attribution per branch:** Each option's description carries its own complete set of consequences — no cross-contamination
|
||||
4. **Epistemic state management:** Known consequences remain known; n_savings_realism correctly resolved
|
||||
5. **Decision-question alignment:** Selected question mirrors the user's stated unresolved issue
|
||||
|
||||
## What it still flattened or misclassified:
|
||||
|
||||
1. **Consequences in descriptions, not as separate nodes:** All six consequence facts embedded in description text rather than as independent graph nodes with typed edges
|
||||
2. **Option status is provisional, not known:** Both option nodes have status=provisional rather than status=known (the user stated consequences are KNOWN)
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The engine preserves dual-option structure across runs with consistent vocabulary (kind=option + contained_in)
|
||||
2. Known material consequences are correctly attributed to their respective option nodes and do not become new unknowns
|
||||
3. Consequence facts survive in structured graph fields (description on option nodes), enabling graph-only consequence recovery through node+edge traversal followed by description parsing
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability across repeated runs** — the first run failed at proposal_validation; the successful inference was on a second attempt
|
||||
2. **Cross-domain generalisation** — single domain case only
|
||||
3. **Whether consequence nodes can be created independently of descriptions** — tested described consequences, not independent extraction
|
||||
4. **Whether downstream reasoning steps can use these structures without text parsing** — description-embedded consequences require semantic parsing to extract individual facts
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Validator changed during experiment: NO
|
||||
## Harness changed during experiment: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,212 @@
|
||||
# Experiment 60A.6 — Option-Specific Consequence Structure: First-Call Confirmation
|
||||
|
||||
**Branch:** `feature/decision-options-v0.25`
|
||||
**Starting HEAD:** 56a04dd (experiment: test option-specific consequence structure)
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Complete
|
||||
**Type:** LIVE RUN — Hard one-call boundary repeat of 60A.5's reasoning, testing whether the first proposal succeeds and preserves option-attributed consequences.
|
||||
|
||||
## Objective
|
||||
|
||||
Does the model preserve known consequences under the correct option branch in the first proposal, without requiring a retry?
|
||||
|
||||
**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
**Pre-existing uncertainty:** `n_savings_realism` (status=unknown)
|
||||
|
||||
## Fixed Answer (verbatim, exact)
|
||||
|
||||
> There are two options.
|
||||
>
|
||||
> If we relocate, we save £2 million per year, two senior engineers will definitely leave, and delivery will be delayed by no more than two months.
|
||||
>
|
||||
> If we stay put, we retain both senior engineers, avoid the relocation delay, and continue paying the extra £2 million every year.
|
||||
>
|
||||
> Those consequences are known. What I still do not know is which option leaves us better off overall.
|
||||
|
||||
## Configured Model
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The first proposal will contain:
|
||||
- one shared unresolved decision-context unknown
|
||||
- option: relocate
|
||||
- option: stay put
|
||||
- contained_in membership for both options
|
||||
- known consequences attributable to the correct option
|
||||
|
||||
The primary question is not whether every consequence gets its own node. The primary question is:
|
||||
> **Can graph state preserve which known consequences belong to which option without collapsing or reopening them as uncertainty?**
|
||||
|
||||
## Execution
|
||||
|
||||
One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`.
|
||||
|
||||
**Call accounting:** startCalls=0, updateCalls=1, totalCalls=1. Retries=0. One-call hard boundary.
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE
|
||||
|
||||
**HTTP status:** 200
|
||||
**Stage:** update_applied (success path)
|
||||
**Validation errors:** none
|
||||
**Proposal applied:** YES
|
||||
|
||||
#### Answer Meaning
|
||||
|
||||
Not explicitly printed for this success path (harness did not emit am fields on the success route).
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [{nodeId:"n_savings_realism", previousStatus:"unknown", newStatus:"resolved", previousValue:null, newValue:null, reason:"User explicitly confirmed the projected £2M annual savings figure is a known consequence for both options."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{id:"n_relocation_decision", label:"Which option leaves us better off overall?", description:"Uncertainty about which option yields superior net value; needed to decide whether the relocation should proceed.", kind:"unknown", status:"unknown", confidence:"medium"},
|
||||
{id:"opt_relocate", label:"Relocate to Manchester", description:"Consequences: save £2M/year, lose 2 senior engineers, delay delivery by <= 2 months.", kind:"option", status:"known", confidence:"high"},
|
||||
{id:"opt_stay_put", label:"Stay in London (Status Quo)", description:"Consequences: retain both senior engineers, avoid relocation delay, continue paying extra £2M/year.", kind:"option", status:"known", confidence:"high"}
|
||||
]
|
||||
|
||||
addedEdges: [
|
||||
{fromNodeId:"opt_relocate", toNodeId:"n_relocation_decision", relationship:"contained_in"},
|
||||
{fromNodeId:"opt_stay_put", toNodeId:"n_relocation_decision", relationship:"contained_in"}
|
||||
]
|
||||
```
|
||||
|
||||
#### structuralActionRequired
|
||||
|
||||
null (not populated by model)
|
||||
|
||||
#### selectedQuestion
|
||||
|
||||
**Question:** "What outcome would demonstrate enough value to justify continuing?"
|
||||
**nodeId:** n_relocation_decision
|
||||
|
||||
### Resulting persistent graph (5 nodes, 3 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| n_savings_realism | unknown | resolved | Are the projected office savings from relocation realistic? |
|
||||
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
||||
| opt_relocate | **option** | **known** | Relocate to Manchester |
|
||||
| opt_stay_put | **option** | **known** | Stay in London (Status Quo) |
|
||||
|
||||
| Edge | From | To | Relationship |
|
||||
|------|------|----|-------------|
|
||||
| e-sr-to-state | n_savings_realism | n_relocation_state | depends_on |
|
||||
| e-opt-relocate-to-dec | opt_relocate | n_relocation_decision | **contained_in** |
|
||||
| e-opt-stay-to-dec | opt_stay_put | n_relocation_decision | **contained_in** |
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Decision structure: NATIVE TWO-OPTION STRUCTURE
|
||||
|
||||
Both `option` nodes survive with kind=option and are linked via contained_in to a single shared decision-context unknown (n_relocation_decision). Notable improvement over 60A.5: option statuses are now `known` (not provisional), matching the user's stated epistemic position that consequences are known facts.
|
||||
|
||||
### 2. Relocate consequences — OPTION-OWNED DESCRIPTION
|
||||
|
||||
| Consequence | Present? | In graph? |
|
||||
|-------------|----------|-----------|
|
||||
| £2M/year saving | YES | "save £2M/year" in opt_relocate.description |
|
||||
| two senior engineers leave | YES | "lose 2 senior engineers" in opt_relocate.description |
|
||||
| <= two months delivery delay | YES | "delay delivery by <= 2 months" in opt_relocate.description |
|
||||
|
||||
All three consequences present within opt_relocate.description. Status is `known` (first-class epistemic treatment). Consequences are not re-encoded as unknown nodes. However, they remain embedded in the description field rather than as independent graph nodes with typed edges — downstream reasoning would need to parse opt_relocate.description text to extract individual consequence values.
|
||||
|
||||
**Classification: OPTION-OWNED DESCRIPTION**
|
||||
|
||||
### 3. Stay-put consequences — OPTION-OWNED DESCRIPTION
|
||||
|
||||
| Consequence | Present? | In graph? |
|
||||
|-------------|----------|-----------|
|
||||
| retain both engineers | YES | "retain both senior engineers" in opt_stay_put.description |
|
||||
| avoid relocation delay | YES | "avoid relocation delay" in opt_stay_put.description |
|
||||
| continue paying extra £2M/year | YES | "continue paying extra £2M/year" in opt_stay_put.description |
|
||||
|
||||
All three consequences present within opt_stay_put.description. Status is `known`. Not re-encoded as unknowns. Same structural class as relocate — embedded in description, not as independent nodes.
|
||||
|
||||
**Classification: OPTION-OWNED DESCRIPTION**
|
||||
|
||||
### 4. Epistemic correctness: CORRECT
|
||||
|
||||
- Known consequences remain known (embedded in option descriptions with status=known)
|
||||
- n_savings_realism correctly resolved to "resolved"
|
||||
- No consequences incorrectly converted to new unknowns
|
||||
- Only one new unknown created for the decision question — correct epistemic state
|
||||
- Option statuses are `known` (improved over 60A.5's provisional)
|
||||
|
||||
### 5. Option attribution: CLEAR FOR BOTH
|
||||
|
||||
Graph makes it possible to tell which option each consequence belongs to:
|
||||
- opt_relocate consequences reachable via its own contained_in edge to n_relocation_decision
|
||||
- opt_stay_put consequences reachable via its own contained_in edge to n_relocation_decision
|
||||
- No cross-contamination or ambiguity
|
||||
|
||||
### 6. Graph-only recoverability
|
||||
|
||||
| Question | Answer |
|
||||
|----------|--------|
|
||||
| Recover Relocate option | YES — node kind=option, label="Relocate to Manchester" |
|
||||
| Recover Stay-put option | YES — node kind=option, label="Stay in London (Status Quo)" |
|
||||
| Recover Relocate consequences | PARTIAL — present in opt_relocate.description (structured field on graph node) |
|
||||
| Recover Stay-put consequences | PARTIAL — present in opt_stay_put.description (structured field on graph node) |
|
||||
| Tell which consequence belongs to which option | YES — each description attached to a distinct option node reached by its own contained_in edge |
|
||||
|
||||
### 7. Selected question: USEFUL
|
||||
|
||||
"What outcome would demonstrate enough value to justify continuing?" targets n_relocation_decision, the correct decision context node. Slightly less aligned with user's phrasing than 60A.5's "What evidence would clarify which option leaves us better off overall?" but still correctly targets the shared trade-off unknown.
|
||||
|
||||
### 8. First-call success: CONFIRMED
|
||||
|
||||
First call returned HTTP 200, no validation errors, full structural result. Hard one-call boundary verified — no retry needed.
|
||||
|
||||
## Classification: A — FIRST-CALL OPTION CONSEQUENCE STRUCTURE CONFIRMED
|
||||
|
||||
First call succeeds; two-option structure survives and consequences are structurally attributable to the correct branch. Options now carry status=known (improved over 60A.5). All six consequences present in correct option-owned descriptions with no cross-contamination, no epistemic reopening, and clear graph-only attribution via contained_in edges.
|
||||
|
||||
This is a step forward over 60A.5: option status corrected from provisional to known, confirming the model now respects the user's epistemic claim ("Those consequences are known") when attributing material facts to option nodes.
|
||||
|
||||
## What the engine understood correctly:
|
||||
|
||||
1. **Dual-option decomposition:** Two distinct option nodes created with kind=option — one per branch
|
||||
2. **Shared decision context:** Both options linked to single n_relocation_decision via contained_in edges
|
||||
3. **Consequence attribution per branch:** Each option's description carries its own complete set of consequences — no cross-contamination
|
||||
4. **Epistemic state management:** Known consequences remain known (status=known on both option nodes); n_savings_realism correctly resolved
|
||||
5. **Option epistemic status:** Both options now have status=known (improvement over 60A.5's provisional)
|
||||
6. **First-call success:** No validation rejection, no retry needed
|
||||
|
||||
## What it still flattened or misclassified:
|
||||
|
||||
1. **Consequences in descriptions, not as separate nodes:** All six consequence facts embedded in description text rather than as independent graph nodes with typed edges
|
||||
2. **newValue null on resolved node:** n_savings_realism's newValue is null rather than a summary value like "£2M/year" (the reason text captures the confirmation but the value field is empty)
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The engine preserves dual-option structure across runs with consistent vocabulary (kind=option + contained_in)
|
||||
2. Known material consequences are correctly attributed to their respective option nodes and do not become new unknowns — confirmed on first call (no retry dependency)
|
||||
3. Option status is now correctly known (not provisional), matching the user's epistemic position
|
||||
4. Consequence facts survive in structured graph fields (description on option nodes), enabling graph-only consequence recovery through node+edge traversal followed by description parsing
|
||||
5. First-call success without validation failure — the 60A.5 null-confidence rejection does not recur
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability across repeated runs** — one run only; cold-start variance may produce different outcomes on repeated runs
|
||||
2. **Cross-domain generalisation** — single domain case only
|
||||
3. **Whether consequence nodes can be created independently of descriptions** — the experiment tested what happens with described consequences, not whether they can be extracted as separate graph entities
|
||||
4. **Whether downstream reasoning steps can use these structures without text parsing** — description-embedded consequences require semantic parsing to extract individual facts
|
||||
5. **Whether newValue null on resolved nodes is consistently acceptable** — the resolved node carries no explicit value summary
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Validator changed during experiment: NO
|
||||
## Harness changed during experiment: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,93 @@
|
||||
# Experiment 60A.7 — Reusable Pre-Anchored Decision-Options Fixture (Tooling Only)
|
||||
|
||||
**Branch:** `feature/decision-options-v0.25`
|
||||
**Date:** 2026-08-13
|
||||
**Type:** TEST TOOLING ONLY — no production reasoning code changes, no live API calls, no Ollama calls
|
||||
|
||||
## Objective
|
||||
|
||||
Add test-only support for the reusable pre-anchored decision-options fixture committed at `tests/fixtures/pre-anchored-decision-options.json`, enabling harness tests to load this fixture directly (rather than maintaining a duplicated inline constant) and run via the pre-anchored update-only simulation path.
|
||||
|
||||
## Context
|
||||
|
||||
The previous experiment (60A.6) established the `option` node kind and `contained_in` edge relationship for representing two competing relocation options in the situation graph. The committed JSON fixture captures this persistent reasoning state:
|
||||
|
||||
- Two `option` nodes (`opt_relocate`, `opt_stay_put`)
|
||||
- One shared decision unknown (`n_relocation_decision`)
|
||||
- `contained_in` edges from each option to the decision node
|
||||
- Unresolved question derived from the decision unknown's label
|
||||
|
||||
The interrupted edit (60A.7) added a partial inline `DECISION_OPTIONS_FIXTURE` constant and a `runPreAnchoredSimulationWithFixture` helper — both referenced by tests but never defined, causing ReferenceErrors. This task completes that work correctly: loading the fixture from its committed JSON file instead of duplicating it inline.
|
||||
|
||||
## Work Performed
|
||||
|
||||
### 1. Test file (`tests/reproduce-multi-turn-investigation.harness.test.js`)
|
||||
|
||||
- **Added** `fs` and `path` imports for direct JSON fixture loading
|
||||
- **Added** `DECISION_OPTIONS_FIXTURE` constant loaded from `tests/fixtures/pre-anchored-decision-options.json` via `JSON.parse(fs.readFileSync(...))` — single source of truth, no duplication
|
||||
- **Added** `runPreAnchoredSimulationWithFixture()` helper function that:
|
||||
- Accepts an optional custom graph (defaults to the committed fixture)
|
||||
- Validates anchor integrity (at least one unresolved unknown node — generic, not savings-specific)
|
||||
- Blocks on missing ANSWER_2 before any API calls (mirrors production behaviour)
|
||||
- Returns `anchor_validation_failed` when graph is null/missing (zero calls)
|
||||
- Derives `previousQuestion` from the fixture's `unresolved_question` field
|
||||
- Sends exactly one Update with the exact fixture graph
|
||||
- Captures all hardened fields: `structuralActionRequired`, `answerMeaning`, `selectedQuestion`, persistent graph, proposal mutation details
|
||||
|
||||
### 2. Script (`scripts/reproduce-multi-turn-investigation.mjs`)
|
||||
|
||||
- **Fixed** hardcoded savings-realism anchor validation to use generic unresolved unknown check (supports any pre-anchored fixture, including decision-options)
|
||||
- **Renamed** internal variable from `savingsNode` → `anchorNode` for clarity
|
||||
- No changes to production reasoning code (`lib/graph/*`)
|
||||
|
||||
### 3. Committed fixture (`tests/fixtures/pre-anchored-decision-options.json`)
|
||||
|
||||
- Already committed during interrupted edit — no changes needed
|
||||
- Valid JSON, complete graph schema with option nodes and contained_in edges
|
||||
|
||||
## Test Results
|
||||
|
||||
```
|
||||
npx vitest run tests/reproduce-multi-turn-investigation.harness.test.js
|
||||
|
||||
✓ 63 tests passed (0 failed)
|
||||
- Core one-shot semantics: 7/7
|
||||
- Accepted-update capture hardening (57J.62): 7/7
|
||||
- structuralActionRequired capture (57J.72): 10/10
|
||||
- Pre-anchored update-only fixture (57J.74): 1/1
|
||||
- Decision-options fixture mode (60A.7): 17/17
|
||||
- Update-only harness tests (57J.78): 8/8
|
||||
- Normal Start→Update unchanged: 2/2
|
||||
- Pre-anchored validation: 3/3
|
||||
- No-extra-call guarantees: 4/4
|
||||
- Existing savings-realism mode still works: 1/1
|
||||
```
|
||||
|
||||
All existing tests remain passing — no regression in any previously validated path.
|
||||
|
||||
## Scope Boundary
|
||||
|
||||
**Permitted changes only:**
|
||||
- `scripts/reproduce-multi-turn-investigation.mjs` (tooling)
|
||||
- `tests/reproduce-multi-turn-investigation.harness.test.js` (test harness)
|
||||
- `tests/fixtures/pre-anchored-decision-options.json` (fixture data)
|
||||
- `docs/experiment-60a7.md` (this doc)
|
||||
- `docs/current-handoff.md` (handoff note)
|
||||
|
||||
**Not changed:**
|
||||
- `lib/graph/prompt-builder.js`
|
||||
- `lib/graph/utils.js`
|
||||
- `lib/graph/schema.js`
|
||||
- Any production reasoning code
|
||||
- Any Ollama or live API calls (0 of each)
|
||||
|
||||
## Classification: COMPLETE — TOOLING ONLY
|
||||
|
||||
Production reasoning code changed: NO
|
||||
Test harness modified: YES (additions only, no removals to existing tests)
|
||||
Fixture loaded from committed JSON in tests: YES
|
||||
Inline decision-options fixture duplicated in test file: NO
|
||||
|
||||
Ollama calls: 0
|
||||
Live API calls: 0
|
||||
Vitest run: 1 focused command (63/63 pass)
|
||||
@@ -0,0 +1,196 @@
|
||||
# Experiment 60A.8 — Downstream Option Evidence Update on Committed Fixture
|
||||
|
||||
**Branch:** `feature/decision-options-v0.25`
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Complete
|
||||
**Type:** LIVE RUN — Single bounded update to test whether the engine attaches new option-specific evidence to the correct existing option without rebuilding the decision.
|
||||
|
||||
## Objective
|
||||
|
||||
When new information applies specifically to the Relocate option ("£400,000 lost margin from two-month delivery delay"), does the engine attach that information to the existing Relocate branch while preserving the existing Stay-put option and shared decision context?
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A strong result should:
|
||||
- Preserve the existing decision-context unknown (n_relocation_decision)
|
||||
- Preserve the existing Relocate option identity (opt_relocate)
|
||||
- Preserve the existing Stay-put option identity (opt_stay_put)
|
||||
- Represent the £400k lost-margin information as belonging to Relocate
|
||||
- Avoid creating duplicate Relocate / Stay-put options
|
||||
- Keep the overall decision unresolved
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
||||
|
||||
Pre-existing structure (4 nodes, 2 edges):
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| opt_relocate | option | known | Relocate to Manchester |
|
||||
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
||||
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
||||
|
||||
Edges: opt_relocate → n_relocation_decision (contained_in); opt_stay_put → n_relocation_decision (contained_in).
|
||||
|
||||
## Configured Model
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
|
||||
## Execution
|
||||
|
||||
Host/model: qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. Hard one-call boundary.
|
||||
|
||||
**Fixed Answer (verbatim):**
|
||||
> We have now estimated that the two-month delivery delay from relocating would cost about £400,000 in lost margin. That cost applies only to the relocation option. I am still deciding whether relocating or staying put leaves us better off overall.
|
||||
|
||||
## Result
|
||||
|
||||
**HTTP status:** 200 — first call succeeds, no validation rejection.
|
||||
|
||||
### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
|
||||
addedNodes: [{
|
||||
id: "n_delay_cost",
|
||||
label: "Lost margin from relocation delay",
|
||||
description: "Estimated £400,000 lost margin due to two-month delivery delay if relocated.",
|
||||
kind: "metric",
|
||||
status: "known",
|
||||
confidence: "high",
|
||||
value: 400000,
|
||||
unit: "GBP",
|
||||
evidenceIds: [],
|
||||
dependsOn: [],
|
||||
affects: [],
|
||||
parentId: null,
|
||||
childIds: ["opt_relocate"]
|
||||
}]
|
||||
|
||||
addedEdges: [{
|
||||
id: "e-cost-to-opt-relocate",
|
||||
fromNodeId: "n_delay_cost",
|
||||
toNodeId: "opt_relocate",
|
||||
relationship: "measures",
|
||||
confidence: "high",
|
||||
description: "Quantifies the financial impact of the relocation option's delay."
|
||||
}]
|
||||
|
||||
selectedQuestion.nodeId: "n_relocation_decision"
|
||||
selectedQuestion: "What outcome would demonstrate enough value to justify continuing?"
|
||||
```
|
||||
|
||||
### Resulting persistent graph (5 nodes, 3 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| opt_relocate | option | **known** | Relocate to Manchester |
|
||||
| opt_stay_put | option | **known** | Stay in London (Status Quo) |
|
||||
| n_relocation_decision | unknown | **unknown** | Which option leaves us better off overall? |
|
||||
| n_delay_cost | **metric** | **known** | Lost margin from relocation delay |
|
||||
|
||||
| Edge | From | To | Relationship |
|
||||
|------|------|----|-------------|
|
||||
| e-opt-rel-to-dec | opt_relocate | n_relocation_decision | contained_in |
|
||||
| e-opt-stay-to-dec | opt_stay_put | n_relocation_decision | contained_in |
|
||||
| e-cost-to-opt-relocate | n_delay_cost | opt_relocate | measures |
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Decision identity: PRESERVED
|
||||
|
||||
The original `n_relocation_decision` node survived untouched — same id, label "Which option leaves us better off overall?", status=unknown. Exactly one decision-context unknown. No duplicate created.
|
||||
|
||||
### 2. Relocate identity: PRESERVED
|
||||
|
||||
`opt_relocate` survived unchanged — kind=option, status=known, label="Relocate to Manchester". Not updated, not replaced, not duplicated.
|
||||
|
||||
### 3. Stay-put identity: PRESERVED
|
||||
|
||||
`opt_stay_put` survived unchanged — kind=option, status=known, label="Stay in London (Status Quo)". Not updated, not replaced, not duplicated.
|
||||
|
||||
### 4. £400k consequence: FIRST-CLASS STRUCTURE
|
||||
|
||||
The engine created `n_delay_cost` as a dedicated metric node with:
|
||||
- **value:** 400000 (numeric, not prose)
|
||||
- **unit:** "GBP" (structured unit field)
|
||||
- **kind:** "metric"
|
||||
- **status:** "known"
|
||||
- **label:** "Lost margin from relocation delay"
|
||||
- **description:** "Estimated £400,000 lost margin due to two-month delivery delay if relocated."
|
||||
|
||||
This is first-class graph structure with typed edges and numeric value — not description-only or embedded text.
|
||||
|
||||
### 5. Option ownership: CLEARLY OWNED BY RELOCATE
|
||||
|
||||
The `measures` edge connects n_delay_cost → opt_relocate. The `childIds` field on n_delay_cost contains ["opt_relocate"]. Both the edge relationship and the child reference unambiguously tie this metric to the Relocate option, not Stay-put. Graph-only reasoning can determine: £400k belongs only to Relocate.
|
||||
|
||||
### 6. Existing contained_in structure: BOTH PRESERVED
|
||||
|
||||
Both pre-existing edges remain intact:
|
||||
- opt_relocate → n_relocation_decision (contained_in) ✓
|
||||
- opt_stay_put → n_relocation_decision (contained_in) ✓
|
||||
|
||||
No edges were removed or altered.
|
||||
|
||||
### 7. Decision state: CORRECTLY REMAINS UNRESOLVED
|
||||
|
||||
`n_relocation_decision.status` is still "unknown". `resolvedUnknownNodeIds` is empty. The user's continued indecision ("I am still deciding") was correctly preserved — the engine did not prematurely resolve the overall decision.
|
||||
|
||||
### 8. Duplication: NO DUPLICATION
|
||||
|
||||
| Entity | Count | Node IDs |
|
||||
|--------|-------|----------|
|
||||
| Relocate option | 1 | opt_relocate |
|
||||
| Stay-put option | 1 | opt_stay_put |
|
||||
| Overall decision | 1 | n_relocation_decision |
|
||||
|
||||
### 9. Selected question: GOOD
|
||||
|
||||
"What outcome would demonstrate enough value to justify continuing?" targets `n_relocation_decision`. This is a genuinely decision-relevant next question — it pursues the missing trade-off evaluation rather than recreating already-known structure. It acknowledges that the cost figure has been added but net-value comparison still requires assessment.
|
||||
|
||||
## Classification: A — EXISTING OPTION GRAPH UPDATED CORRECTLY
|
||||
|
||||
The engine preserved all existing option identities and decision context, created a first-class metric node for the £400k consequence correctly owned by Relocate via both edge relationship (`measures`) and child reference (`childIds: ["opt_relocate"]`), added no duplicates to any entity, and kept the overall decision unresolved. This is a strong confirmation that downstream option evidence attaches cleanly to existing options without rebuilding the decision.
|
||||
|
||||
## What the engine understood correctly:
|
||||
|
||||
1. **Evidence ownership:** The £400k lost-margin fact belongs to Relocate specifically — represented via a `measures` edge from metric → opt_relocate and childIds containing only "opt_relocate".
|
||||
2. **Non-resolution of overall decision:** Despite new evidence being added, the engine correctly kept n_relocation_decision unresolved. The user's continued indecision was respected.
|
||||
3. **No option duplication:** Existing opt_relocate and opt_stay_put survived untouched — no duplicate nodes created for either option.
|
||||
4. **First-class representation:** The consequence was not relegated to prose/description. It is a numeric metric node with value=400000, unit="GBP", kind="metric".
|
||||
5. **Edge topology preserved:** Both original contained_in edges remain intact alongside the new measures edge.
|
||||
|
||||
## What it did NOT do:
|
||||
|
||||
1. **Did not update existing option nodes** — opt_relocate was added-to via a child reference but its own node fields were not modified (updatedNodes=[])
|
||||
2. **Did not create dependent unknowns** — no new uncertainty nodes were generated from the consequence; the metric is stated as known
|
||||
3. **Did not resolve n_savings_realism** — there was no such node in this fixture (this was a clean decision-options context, not savings-realism)
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The engine can add first-class structural evidence (numeric metric nodes with typed edges) to an existing option branch in a single update call.
|
||||
2. Evidence ownership by the correct option is achievable via both edge relationships and child references — enabling graph-only reasonability.
|
||||
3. Downstream option evidence does not force premature resolution of the overall decision context.
|
||||
4. Existing option identities are preserved without duplication during evidence updates.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability across repeated runs** — one run only; cold-start variance may produce different outcomes on repeated runs.
|
||||
2. **Consequence directionality semantics** — `measures` edge goes from metric → opt_relocate; the semantic direction (cost as a property of the option vs. the option being measured by the cost) is correct but untested for alternative relationship types.
|
||||
3. **Cross-domain generalisation** — single domain case only.
|
||||
4. **Multiple consequences per option** — tested with one consequence fact; multiple concurrent facts on the same option were not tested.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls: 1
|
||||
## Direct API calls: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user