experiment: test decision-relative coherence

docs/current-handoff.md

@@ -58,7 +58,9 @@ Experiment 48 audited whether real graph updates populate usable unknown relatio
 
 Experiment 49 tested whether any sequence of real production updates creates two or more active unknowns referencing the same populated relationship anchor. Results: no shared anchor found in production update sequences (both Cases A and B returned separate_anchors or insufficient_data). Structural capability exists but triggering logic never produces coexisting anchors. Status closed.
 
-Experiment 50 tested whether shared edge topology from `buildInitialGraph` provides a usable coherence signal. Coherent and scattered inputs both produce identical edge topology — every unknown connects to the same summary node (kind=state) via depends_on edges, regardless of semantics. Conclusion: production edges provide only a generic shared anchor, not meaningful coherence. Status pending Rob's review.
+Experiment 50 tested whether shared edge topology from `buildInitialGraph` provides a usable coherence signal. Coherent and scattered inputs both produce identical edge topology — every unknown connects to the same summary node (kind=state) via depends_on edges, regardless of semantics. Initial shared edges are generic structural wiring, not coherence evidence. Closed (pending Rob's review).
+
+Experiment 51 tested whether decision-relative relevance distinguishes coherent from scattered unknowns better than graph topology does. Within its training vocabulary, the classifier classified all four coherent unknowns as relevant and three of four scattered unknowns as irrelevant — but one scattered question was incorrectly flagged due to identical phrasing. Outside its vocabulary (different domain or paraphrased language), the classifier could not generalise: all four coherent unknowns received `cannot_determine`. The decision target never provided semantic context, only a binary action-keyword gate. No production code changed; no active engine behaviour changed; 70 tests pass (45 new + 25 Exp 21 regression). Status pending Rob's review.
 
 ## 5. What Remains Open
 
@@ -112,7 +114,7 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38–50. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–51. Branch: `feature/user-workspace-ux-v0.7`.*
 
 ### Return-to-Work Note (Experiment 47)
 
@@ -122,4 +124,4 @@ Experiment 48 passively audited whether real graph updates populate usable unkno
 
 Experiment 49 tested whether production update sequences can produce a real shared anchor (two or more active unknowns sharing the same populated relationship node). Two sequential-update scenarios via `applyValidatedProposal` (Cases A and B in the new test file) consistently returned `separate_anchors` or `insufficient_data` — no coexisting active unknowns reference the same anchor. The structural capability exists (fields populate correctly via emergent reasoning), but the triggering logic never produces shared anchors within tested flows. Control cases (C–F, 20 tests) confirmed the diagnostic works correctly on controlled fixtures and all produced nodes pass schema validation. Total: 36 new tests, all passing. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/shared-anchor-production-path.test.js` for results, then `docs/design-evolution-log.md` Experiment 49 section.
 
-Experiment 50 tested whether shared graph edges from `buildInitialGraph` provide a meaningful coherence signal. Two inputs — one coherent (North West expansion), one scattered (business feels stuck) — passed through identical production edge-building path. Both produced the same edge topology: four unknowns, each connecting via depends_on to the summary node (kind=state). Node-level relationship fields remain empty in both cases. Conclusion: shared-edge connectivity is structural plumbing, not coherence evidence. No structural difference exists between coherent and scattered inputs at the production graph level. Status pending Rob's review. First file: `tests/graph/initial-edge-coherence.test.js`.
+Experiment 51 tested whether the existing passive decision-relevance classifier distinguishes coherent from scattered unknowns better than graph topology did (Exp 50). Within its training vocabulary (European market entry), the classifier correctly classified all four coherent unknowns as relevant to the decision and three of four scattered unknowns as irrelevant — but one scattered question was incorrectly flagged as relevant due to identical phrasing. Outside its vocabulary (different domain: community events, or plain-English paraphrases), the classifier could not generalise: all four coherent unknowns were classified as `cannot_determine`. The decision target never provided semantic context — only a binary action-keyword gate for Rule 1 firing. No production code changed; no active engine behaviour changed; all 70 tests pass (45 new + 25 Exp 21 regression). What remains uncertain: whether phrasing-aware pattern matching or genuine semantic understanding is needed for coherence detection. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relative-coherence.test.js` for the full test and results, then this handoff's Experiment 51 section.

docs/design-evolution-log.md

@@ -3362,3 +3362,164 @@ The test-only helper distinguishes generic summary nodes from specific anchors b
 ### Pending Rob's review. No production code or graph schema modified.
 
 **Commit:** pending (experiment: test initial graph edge coherence)
+
+---
+
+## Experiment 51 — Is Coherence Relative to the Decision, Rather Than the Graph Shape? (2026-08-07)
+
+Experiment 51 tested whether an explicit decision target provides a more useful coherence signal than raw graph structure. It used one known good signal for scope confusion: the existing passive `assessQuestionRelevanceToDecision` classifier, which judges an unknown against an explicit decision target using five relevance categories. No production code changed.
+
+### Hypothesis
+
+When an explicit decision target is supplied, coherent unknowns should all show meaningful relevance to that decision, while scattered unknowns should contain some classified as irrelevant. If this holds across multiple wordings and domains, decision-relative relevance may be a better coherence signal than graph topology.
+
+### Decision Target Used
+
+Domain 1: "Should we enter the European market with our SaaS analytics platform?"
+Domain 2: "Should we organise the community event outdoors this September?"
+
+### Coherent Unknown Set — Domain 1 (European Market)
+
+Four unknowns all contributing to one decision:
+- `demand`: "Whether to enter the European market for analytics tools"
+- `compliance`: "Whether our product is suitable for European compliance requirements"
+- `cost-benefit`: "Whether the cost of achieving compliance is justified by the potential market size"
+- `differentiation`: "Whether we have competitive differentiation against existing European players"
+
+### Scattered Unknown Set — Domain 1 (European Market)
+
+Four unknowns with mixed relevance:
+- `scat-demand`: "Whether we should enter the European market for analytics tools"
+- `scat-staff-conflict`: "Can two senior staff members resolve their ongoing disagreement?"
+- `scat-lease`: "Should the head office lease be renewed at the current rate next year?"
+- `scat-pricing`: "Does an existing unrelated product's pricing align with market willingness to pay?"
+
+### Coherent-set Relevance Results — Domain 1
+
+| Unknown | Classification | Reason (pattern matched) |
+|---|---|---|
+| demand | could_change_decision | DECISION_REVERSAL_PATTERNS ("Whether to enter") |
+| compliance | supports_decision | PRECONDITION_PATTERNS ("product is suitable for ... compliance requirements") |
+| cost-benefit | supports_decision | FEASIBILITY_PATTERNS ("cost of achieving compliance is justified") |
+| differentiation | supports_decision | SUPPORTING_CONTEXT_PATTERNS ("competitive differentiation against existing") |
+
+**All four coherent unknowns received meaningful decision-relative classifications (not `cannot_determine`). One `could_change_decision`, three `supports_decision`. Multiple distinct categories produced. Every result included a non-empty reason.**
+
+### Scattered-set Relevance Results — Domain 1
+
+| Unknown | Classification | Outcome |
+|---|---|---|
+| scat-demand | could_change_decision | Unintended: matches DECISION_REVERSAL_PATTERNS ("enter") |
+| scat-staff-conflict | cannot_determine | Correctly rejected (no pattern match) |
+| scat-lease | cannot_determine | Correctly rejected (no pattern match) |
+| scat-pricing | cannot_determine | Correctly rejected (no pattern match) |
+
+**Three of four scattered unknowns were correctly identified as irrelevant (`cannot_determine`). One — `scat-demand` — matched because its phrasing happens to contain the same keyword pattern ("enter") as the coherent demand question. This is an expected behaviour: the classifier matches phrasing, not intent.**
+
+### Unrelated Questions Correctly Rejected
+
+- Staff disagreement: `cannot_determine`
+- Head office lease renewal: `cannot_determine`
+- Unrelated product pricing: `cannot_determine`
+
+### Unrelated Questions Incorrectly Treated as Relevant
+
+- "Whether we should enter the European market for analytics tools" — matched DECISION_REVERSAL_PATTERNS because it contains "Whether to/should enter". This is a phrasing match, not a coherence signal. The scattered set's first item deliberately uses the same action keyword as the coherent domain to test whether the classifier can distinguish genuine coherence from pattern matching. It cannot.
+
+### Second-domain Decision Target Used
+
+"Should we organise the community event outdoors this September?"
+
+### Second-domain Results — Coherent Set
+
+| Unknown | Classification | Outcome |
+|---|---|---|
+| evt-weather | cannot_determine | Failed: "weather risk" not in demand keywords |
+| evt-insurance | cannot_determine | Failed: no precondition pattern match |
+| evt-capacity | cannot_determine | Failed: generic capacity language |
+| evt-accessibility | cannot_determine | Failed: no compliance/mandatory keyword match |
+
+**All four coherent unknowns received `cannot_determine`. The classifier could not generalise to this domain because none of the phrasing matched its trained keyword patterns.**
+
+### Second-domain Results — Scattered Set
+
+| Unknown | Classification | Outcome |
+|---|---|---|
+| scat-evt-weather | cannot_determine | Consistent with coherent weather result |
+| scat-board-chairs | cannot_determine or unlikely_to_change_decision | Correctly rejected |
+| scat-volunteer | cannot_determine or unlikely_to_change_decision | Correctly rejected |
+| scat-local-park | cannot_determinate or unlikely_to_change_decision | Correctly rejected |
+
+### Coherent Paraphrase Result
+
+Original: "Whether sufficient customer demand exists in the North West"
+Paraphrase: "Would enough people there actually want what we offer?"
+
+The paraphrase produced `cannot_determine`. The classifier recognised the original phrasing because it contains "sufficient ... demand" (matching DECISION_REVERSAL_PATTERNS), but the plain-English paraphrase avoids that keyword and was not classified as relevant. **This demonstrates that the classifier recognises familiar words rather than understanding decision relevance.**
+
+### Unrelated Paraphrase Result
+
+Original: "What benchmarks do other SaaS companies use for market sizing"
+Paraphrase: "Which analytics firms set the industry standard?"
+
+Both produced irrelevant classifications (original matched INCIDENTAL_PATTERNS via "benchmark"; paraphrase also did not match any relevant pattern). The unrelated classification was preserved across paraphrasing, which is a positive signal.
+
+### Language Sensitivity Observed
+
+**Extreme.** The classifier is entirely keyword-matching with zero semantic understanding:
+- Within vocabulary (European market entry): works as expected by Exp 21 test patterns
+- Different domain (community event): all four coherent unknowns classified as `cannot_determine` despite being genuinely relevant to the decision
+- Paraphrased language: coherent original classified as relevant, paraphrase as `cannot_determine` — same underlying question, different results
+- The decision target is only used for a binary action-keyword gate ("enter/launch/build/stop/abandon" present or not). It never provides semantic context for matching the unknown against the decision.
+
+### Comparison with Experiment 50 Graph-topology Result
+
+Both experiments reached the same fundamental conclusion about their respective signals: **neither graph topology nor decision-relative keyword matching can reliably distinguish coherent from scattered breadth.**
+- Exp 50: every unknown connects to the same generic `state` node regardless of semantics
+- Exp 51: classification depends on phrasing keywords, not on whether the unknown actually matters to the stated decision
+
+### Experiment Conclusion
+
+**Decision-relative relevance is promising but language-sensitive.** Within its training vocabulary (European market entry scenarios matching Exp 21 patterns), the classifier produces meaningful distinctions between coherent and scattered unknown sets. However, it fails completely outside that vocabulary — both in different domains and when rephrased. The decision target never provides semantic context; it only gates whether Rule 1 fires via a binary action-keyword check. This is not coherence detection. It is keyword pattern matching dressed as decision relevance.
+
+### Questionable or Unsupported Findings
+
+The classifier's behaviour within its training vocabulary may be coincidental rather than principled. The five pattern rules (DECISION_REVERSAL, PRECONDITION, FEASIBILITY, SUPPORTING_CONTEXT, INCIDENTAL) were written to cover known market-entry scenarios and may not generalise even within the same domain. The test confirms they work for those specific cases only.
+
+### Focused Test Result
+
+| Test File | Tests | Passed |
+|---|---|---|
+| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
+| `question-decision-relevance.test.js` (Exp 21 regression) | 25 | 25 |
+
+### Regression / Validation Result
+
+All existing Exp 21 tests pass. The classifier's output for known patterns is unchanged: `could_change_decision`, `supports_decision`, `unlikely_to_change_decision`, and `cannot_determine` all produce identically as before. No production behaviour changed.
+
+### Documentation Updated
+
+- `docs/design-evolution-log.md`: Experiment 50 closed; Experiment 51 added
+- `docs/current-handoff.md`: Return-to-work note updated
+
+### Confirmation Production Decision-Relevance Classifier Remained Unchanged
+
+The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass) and confirming five categories produce identically. The test file includes explicit assertions that known patterns return their original classifications unchanged.
+
+### Confirmation Assessor and Behaviour Selection Remained Unchanged
+
+No assessor files were loaded or modified. No Behaviour Selection files were loaded or modified. The experiment uses only the decision-relevance classifier directly.
+
+### Confirmation Graph Schema and Construction Remained Unchanged
+
+No schema or builder files were loaded or modified. The experiment tests classifier output, not graph topology.
+
+### Confirmation Existing Fixtures Remained Unchanged
+
+No fixtures were loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`.
+
+### Confirmation Active Engine Behaviour Remained Unchanged
+
+The decision-relevance classifier has no callers outside its own module (verified in Exp 28 implementation-verification). No active user-facing behaviour changed.
+
+**Status: Pending Rob's review.**