robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(confidence-engine): checkpoint design evolution archive tranche three

Browse Source
This commit is contained in:
robbond
2026-09-03 10:55:03 +01:00
parent e611283e3e
commit 1f3b26c983
3 changed files with 1060 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/archive/experiments/vol-1-chapters/README.md
+29 -1
View File
@@ -93,9 +93,37 @@ Experiments 35–41 and Experiment 41 Conclusion — handoff validation, behavio
Fidelity:
Exact contiguous copy.
Tranches 1 and 2 have now been extracted.
Tranches 1, 2, and 3 have now been extracted.
The original monolithic log remains intact and authoritative while extraction is incomplete.
## Tranche 3
### Chapter 7
Path:
docs/archive/experiments/vol-1-chapters/ch7/experiments-42-to-46.md
Original source:
docs/design-evolution-log.md lines 2504–3069
Contents:
Experiments 42–46 — narrow acknowledge exclusion, Clarify readiness audit, assessor unclear-starting-point, and "Too Broad" boundary investigations.
Fidelity:
Exact contiguous copy.
### Chapter 8
Path:
docs/archive/experiments/vol-1-chapters/ch8/experiments-47-to-51.md
Original source:
docs/design-evolution-log.md lines 3070–3534
Contents:
Experiments 47–51 — shared-anchor coherence diagnostic, unknown relationship audit, production shared-anchor test, and semantic decision-relevance coherence experiments.
Fidelity:
Exact contiguous copy.
## Refactor status
Only tranches 1 and 2 have been extracted.
docs/archive/experiments/vol-1-chapters/ch7/experiments-42-to-46.md
+566
View File
@@ -0,0 +1,566 @@
## Experiment 42 — Implement Narrow Acknowledge Exclusion (Variant B) (2026-08-06)
### Hypothesis
Applying a narrow exclusion gate to Acknowledge — excluding it when phase is synthesising or concluding, progress is stalled, or conversation health is user_overloaded — will reduce the two identified false-Acknowledge selections (concluding → summarise, stalled → pause) without introducing any unintended behaviour changes in other tested turns.
### Exact Exclusion Rule
`isAcknowledgeExcluded(assessment)` returns `true` when:
- `phase.value` is `synthesising` or `concluding`; OR
- `progress.value` is `stalled`; OR
- `conversationHealth.value` is `user_overloaded`.
When excluded, Acknowledge does not fire and the selector proceeds to the next priority rule. The gate qualifies the trigger; it does not replace it.
### Two Changed Turns
| Turn | Scenario | Phase | Progress | Health | Before | After |
|------|----------|-------|----------|--------|--------|-------|
| long-investigation t4 | concluding long-investigation | concluding(high) | steady | healthy | acknowledge | **summarise** |
| contradictory-evidence t1 | stalled contradictory-evidence | focusing(high) | stalled | healthy | acknowledge | **pause** |
### Five Preserved Turns
| Turn | Scenario | Phase | Progress | Health | Behaviour (unchanged) |
|------|----------|-------|----------|--------|----------------------|
| long-investigation t0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
| long-investigation t3 | focusing(high) | steady | healthy | acknowledge |
| contradictory-evidence t0 | focusing(high) | cannot_determine | healthy | acknowledge |
| contradictory-evidence t2 | focusing(high) | steady | healthy | acknowledge |
| short-early t0 | exploring(low) | cannot_determine | healthy | continue |
### Final Behaviour Distribution (7 real assessment turns)
- Acknowledge: 3
- Summarise: 1
- Pause: 1
- Continue: 2
- Clarify: 0
### Integration Status
The production Behaviour Selection module (`lib/behaviour-selection/behaviour-selector.js`) was changed to include the `isAcknowledgeExcluded()` gate. However, **active user-facing engine behaviour did not change** because Behaviour Selection remains isolated with no runtime caller — it is exported but never imported by any code in the repository.
### Clarify Status
`Clarify` remains unresolved and was not modified in this experiment. Its trigger conditions (`health=too_broad` or `phase=orienting + obs<3`) require states that no tested scenario produces. This remains an open question for future work.
### Selector Output Shape
The selector output shape did not change. The exclusion gate returns `null` from `selectAcknowledge`, which is the existing early-return mechanism used when a rule does not match. No new fields, no restructuring of the return object.
### Assessor and Fixtures
Assessor logic did not change. Fixtures did not change. Priority order did not change.
### Test Results
All 151 relevant tests passed across:
- `tests/behaviour-selection.test.js`: 51 (no regressions)
- `tests/behaviour-selection.reachability.test.js`: 33 (updated for new exclusion gate)
- `tests/behaviour-selection.counterfactual.test.js`: 44 (from Exp 41, no changes)
- `tests/behaviour-selection.real-assessment.test.js`: 16 (shared fixtures intact)
Tests were not rerun as part of this documentation-only closure. The recorded result comes from the implementation commit (05d3d96).
### Limitations
- Only seven real assessment turns across three scenarios were evaluated; other investigation domains may exhibit different patterns.
- `health=user_overloaded` is excluded by rule but never produced by any current assessor fixture — it is untested in practice.
- `Clarify` remains deferred because no scenario produces the narrow trigger conditions it requires.
- The selector remains isolated with no runtime caller; there is no live user-facing validation.
### Result
**Confirmed within the tested scenarios.** Variant B correctly changes only the two intended turns and preserves all five others. No unintended side-effects were observed.
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 43 — Audit Clarify Readiness Signals (2026-08-06)
### Hypothesis
The existing investigation-state-assessor never produces states that trigger the production Clarify rule in any tested scenario. Clarify is absent from Behaviour Selection not because of a selector defect but because no current fixture represents the genuinely unclear-scoped investigations that its triggers are designed for.
### Diagnostic Test File
A focused diagnostic test was created at `tests/behaviour-selection.clarify-readiness.test.js` with 31 assertions auditing every turn across all existing assessor and reachability fixtures. It inspects:
- Phase value distribution (focusing, exploring, concluding, synthesising, deepening, cannot_determine)
- Conversation health values (healthy, too_narrow, too_broad, user_overloaded)
- Observation density per turn
- Clarify eligibility via the exact production rule in `selectClarify`
### Audit Scope
| Source | Scenarios | Turns Inspected |
|--------|-----------|-----------------|
| `investigation-state-assessor.test.js` | 7 | 7 (one per scenario) |
| `behaviour-selection.reachability.test.js` | 3 | 3 (contradictory-evidence t0, t1, t2) |
| **Total** | **10** | **10 real-turn assessments** |
### Q1 — Does the assessor ever produce `too_broad`?
**No.** Zero scenarios across all test fixtures produce `conversationHealth.value === "too_broad"`.
The `too_broad` trigger requires `activeUnknownCount > 3 AND resolvedNodeIds.length < 2`. Every existing scenario starts with exactly one active unknown (the single unresolved question the investigation is about), and the assessor never produces a state where more than three unrelated unknowns coexist without resolution.
### Q2 — Does the assessor ever produce `phase.value === "orienting"`?
**No.** Zero scenarios produce orienting. The five phase values produced by the assessor are: concluding, synthesising, focusing, exploring, deepening, and cannot_determine. **`orienting` is not a possible output of any assessor code path.** It does not appear in `assessPhase()`.
### Q3 — Does orienting ever coincide with observation density < 3?
**Never applicable.** Since the assessor never produces orienting, this condition cannot arise in real data. The orienting-based Clarify trigger is dead code within the tested scenarios (and likely in production until a scenario change introduces orienting).
### Q4 — How many turns are Clarify-eligible?
**Zero of 10 turns.** Both Clarify rules evaluate to false for every assessed turn:
- Rule 1 (`too_broad` health): false in all 10 turns
- Rule 2 (`orienting + obs<3`): false in all 10 turns (orienting never appears)
### Q5 — What are the closest existing signals to a genuine Clarify need?
Two signals approach clarification but do not match its intent:
| Signal | Turns | Meaning | Maps to Clarify? |
|--------|-------|---------|-----------------|
| `too_narrow` health | 1 (long-turn-0) | Insufficient contextual evidence for a narrow investigation | No — too_narrow means "needs more data," not "scope is unclear" |
| `exploring` phase with low obs density | 1 (complete-turn-0) | Early-stage investigation with sparse observations | No — this signals the start of an investigation, not scope confusion |
### Q6 — Signal reliability assessment for future Clarify rule design
| Signal | Reliability for Clarify intent |
|--------|-------------------------------|
| `too_narrow` health | Low reliability. It reliably indicates insufficient context for question formulation but conflates "too little information" with "unclear scope." The assessor's own description: "The investigation needs more contextual evidence before the current question can be answered effectively." This is about quantity, not clarity. |
| `exploring` + low obs density | Low reliability. It reliably indicates an early-stage investigation but does not distinguish between "well-scoped investigation in early phase" and "unclear investigation needing anchoring." Both map to exploring. |
### Q7 — Is Clarify's absence appropriate for current fixtures?
**Yes.** Every existing fixture represents a well-defined, focused investigation with a clear central statement:
- "Comparing two products before purchase decision" (single question, single dimension)
- "Evaluating European market entry" (single strategic question)
- "Evaluating $2M procurement against conflicting expert advice" (single decision context)
A genuinely unclear-scoped investigation would need one of:
- A central statement so vague the system cannot classify it into any phase
- Multiple unrelated threads at startup with no clear priority anchor
- Contradictory framing where the situation itself is ambiguous
No current fixture represents these states. **Clarify's absence is appropriate because the existing scenarios are genuinely well-scoped, not because the selector is broken.**
### Phase Distribution Across All 10 Turns
| Phase | Count | Scenarios |
|-------|-------|-----------|
| focusing | 7 | comparison t0,t1,t2; long t3; contradictory t0,t1,t2 |
| cannot_determine | 1 | long t0 |
| concluding | 1 | long t4 |
| exploring | 1 | complete t0 |
No synthesising, deepening, or orienting phases observed.
### Production Clarify Trigger — Exact Rule Match
```js
// selectClarify (behaviour-selector.js lines 65-83)
function selectClarify(assessment) {
// Rule A: broad scope detected
if (assessment.conversationHealth.value === "too_broad") return clarify;
// Rule B: early orientation with sparse data
if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return clarify;
return null;
}
```
**Rule A trigger:** `conversationHealth.value === "too_broad"` — zero occurrences in tested scenarios.
**Rule B trigger:** `phase.value === "orienting"` — never produced by assessor; **dead code path.**
### Focused Test Results (Experiment 43)
- Total tests: **31**
- Passed: **31**
- Failed: **0**
All diagnostics confirm zero Clarify eligibility across the complete set of real-world fixtures.
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | New diagnostic file — no regression possible |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions from any prior experiments |
| `tests/behaviour-selection.reachability.test.js` | 33 | ✓ Pass | Clarify still eligible in 0 real turns; synthetically reachable |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Assessor behavior unchanged |
### Limitations
- The audit covers all existing test fixtures but not every possible investigation domain. Different problem domains (legal disputes, medical triage, multi-party procurement) may produce different assessor states.
- `too_broad` requires very specific conditions (>3 active unknowns with <2 resolved) that no current fixture exercises. A fixture designed specifically to trigger it would validate the health classifier path.
- The orienting phase was never produced by any assessor code path in the entire test suite, suggesting a design gap: either orienting was removed from the assessor without updating the selector, or it was never implemented as an active phase value.
### Conclusion
**Clarify is absent from Behaviour Selection because no current scenario genuinely needs clarification — not because of a selector defect.** The two production rules are well-formed but their trigger conditions (too_broad health and orienting phase) represent states that the assessor either cannot produce (orienting) or does not produce in any tested fixture (too_broad).
**Two distinct issues identified:**
1. **Dead code path**: The orienting-based Clarify rule never activates because the assessor produces six phase values but none is `orienting`. This is a design inconsistency worth correcting — either add orienting as a real phase or remove that rule from the selector.
2. **Narrow trigger threshold**: The too_broad condition (`activeUnknownCount > 3 AND resolvedNodeIds < 2`) is validly narrow but never exercised by any fixture. If Clarify should fire earlier in investigations, the threshold should be relaxed; if it should only fire for genuinely lost investigations, it should stay as-is and a dedicated fixture should validate it.
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 44 — Assessor Against Unclear Starting Point (2026-08-06)
### Objective
Create one deliberately unclear investigation fixture and test whether the existing Investigation State Assessor produces any signal that justifies Clarify.
### Hypothesis
A deliberately unclear starting scenario may expose one of three outcomes:
1. The assessor already produces `too_broad`.
2. The assessor produces another existing signal that reasonably represents the need to clarify.
3. The assessor has no suitable signal for unclear framing.
### Fixture Description
**File:** `tests/investigation-state-assessor.unclear-start.test.js` (test-only, not imported anywhere else)
The fixture represents:
- A vague central statement that admits uncertainty: *"The business feels stuck. Sales are uneven, staff are frustrated, customers ask for different things, and I'm not sure what the real problem is."*
- Five competing unknown threads (customer demand, staff capacity, product direction, pricing, operations) with no priority anchor
- Only one observation (the only concrete data point)
- Zero resolved evidence nodes
- No selected question (no established direction)
- All existing graph fields only (id, label, description, kind, status, confidence, evidenceIds, dependsOn, affects, childIds)
- Five `kind: "unknown"` nodes and one `kind: "observation"` node
### Returned Assessment Signals
| Signal | Value | Confidence |
|--------|-------|------------|
| Phase | `cannot_determine` | low |
| Phase signals | "Insufficient data for phase classification" | — |
| Progress | `cannot_determine` | low |
| Progress signals | "Insufficient data for progress assessment" | — |
| Conversation health | **`too_broad`** | medium |
| Health signals | "5 active unknowns with fewer than 2 resolved items"; "Investigation may be spreading too thin" | — |
Detailed evidence:
- Phase evidence: resolvedNodeCount=0, activeUnknownCount=5, observationDensity=1, evidenceDepth="shallow"
- Progress evidence: turnCount=0, recentResolutionsLastTurn=0
- Health evidence: activeUnknownCount=5, resolvedNodeRatio=null, hasActiveQuestion=false
### Clarify Eligibility
**Clarify became eligible via Rule A.** The production `selectClarify` rule fires because `conversationHealth.value === "too_broad"`.
The production selector (`selectBehaviour`) returned:
- behaviour: `"clarify"`
- confidence: `"high"`
- reason: "Conversation health is too broad — investigation may be spreading too thin. Narrow focus through a specific clarification question."
### Interpretation
**Classification: `assessor_recognises_unclear_start`**
The assessor produced `too_broad` from the unclear-start fixture, which directly maps to Clarify's intent (genuinely unclear scope requiring anchoring). The signal honestly reflects the starting situation: five competing unknowns with no resolved evidence and no established direction.
### What the Assessor Recognised
1. Multiple active unknowns without sufficient resolution triggered `too_broad` health classification.
2. The assessor correctly recorded 5 active unknowns in both phase and health evidence sections.
3. Observation density (1) was correctly reported as shallow.
4. Phase confidence remained low due to insufficient data for any meaningful classification.
### What the Assessor Failed to Recognise
1. **`orienting` phase**: Still not produced by the assessor. The orienting-based Clarify rule remains dead code, unchanged from Experiment 43's finding.
2. **Early-stage clarification need**: The `too_broad` trigger only fires after >3 unknowns accumulate — it does not catch a situation with fewer competing threads that is still genuinely unclear in framing.
### Limitations
- Only one fixture was tested. Different vague-scenario configurations may produce different results.
- The `too_broad` trigger depends on having more than 3 active unknowns with fewer than 2 resolved — this specific threshold was exercised, but other boundary conditions (e.g., exactly 4 unknowns, or 5 unknowns with 1 resolved) were not tested.
- The fixture uses the assessor's existing `too_broad` definition which conflates "many unknowns" with "unclear scope." A genuinely unclear scenario with only 2–3 competing threads may not trigger this signal.
### Status
Pending Rob's review. Experiment 43 remains closed — its conclusion that a deliberately unclear fixture was required is confirmed by this experiment, which successfully exercises the previously untested `too_broad` health path.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** The assessor produced the expected `too_broad` signal from the unclear fixture, confirming the health classifier path works correctly. No code was modified.
### Closure
Experiment 44 is **closed**. Conclusion: the assessor recognises an extreme unclear start; too_broad and Clarify are reachable; the useful boundary remained unknown.
---
## Experiment 45 — Where Does "Too Broad" Begin? (2026-08-06)
### Objective
Test how the existing assessor's `too_broad` threshold behaves as an unclear starting scenario grows from two competing unknowns to five, all with identical base inputs. Passive boundary experiment only — no production code changes.
### Hypothesis
| Active unknowns | Expected health |
|---:|---|
| 2 | not `too_broad` |
| 3 | not `too_broad` |
| 4 | `too_broad` |
| 5 | `too_broad` |
### Fixture-Control Method
One test-only fixture builder creates the same vague starting situation varying only the number of competing unknowns:
- Same central statement; same single observation; zero resolved items (base); no selected question; no active direction; same node shapes and confidence values.
- Only the count of `kind: "unknown"` nodes differs.
### Results: Two Through Five Active Unknowns
| Active unknowns | Health | Confidence | Phase | Progress | Clarify eligible | Selector |
|---:|---|---|---|---|---:|---|
| 2 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
| 3 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
| 4 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determine` (low) | Yes | clarify (high) |
| 5 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determined` (low) | Yes | clarify (high) |
### Results: Four Unknowns + Resolved Items
| Active unknowns | Resolved | Health | Confidence | Clarify eligible |
|---:|---:|---|---|---:|
| 4 | 0 | `too_broad` | medium | Yes |
| 4 | 1 | `too_broad` | medium | Yes |
| 4 | 2 | `cannot_determine` | low | No |
### Human-Sense Review
- **Two competing threads:** Still appears ambiguous rather than clearly manageable. The assessor returns `cannot_determine`, not `healthy`. This is honest — two unknowns with one observation and no question genuinely leave the state unclear.
- **Three competing threads:** Appears ambiguous or already confused. The assessor still returns `cannot_determine`. This feels correct — three competing threads with minimal context is genuinely uncertain, not healthy.
- **Four competing threads:** Appears genuinely too broad. The transition from three (uncertain) to four (too_broad) feels believable — a real investigator would start losing focus at this point.
- **Five competing threads:** Clearly justifies clarification. Matches Experiment 44's result; no surprise.
- **Transition between three and four:** Understandable. Three threads with one observation is "not enough to decide"; four adds the tipping point where the spread becomes problematic.
- **Confidence language:** `too_broad` confidence is `medium` for both four and five unknowns. The signals are specific ("4 active unknowns with fewer than 2 resolved items"), so medium confidence is honest — it does not overstate certainty.
### Boundary Classification
| Transition | Classification | Rationale |
|---|---|---|
| 2→3 | `believable` | Both remain `cannot_determine`; the gap between "manageable" and "confused" genuinely sits around here |
| 3→4 | `believable` | Four competing threads with no resolution is a believable tipping point for losing focus |
| Resolution threshold (<2 resolved) | `believable` | The binary boundary (1 stays too_broad, 2 clears it) aligns with the design intent of "sufficient context to narrow" |
### Usefulness of Active-Unknown Count as a Proxy
Active-unknown count acts as a **useful but coarse** proxy for scope confusion. It works because:
1. In the tested scenarios, more unknowns directly correlates with genuine ambiguity.
2. The resolved-item gate prevents premature too_broad flags on investigations making progress.
3. It avoids subjective measurement of "how confused is the user."
However, it cannot distinguish between:
- Four unknowns about one decision (genuinely broad) versus four unknowns across a multi-decision comparison (expected).
- A well-formed investigation with natural branching versus an unfocused investigation losing its way.
### Questionable or Unsupported Findings
1. **Health defaults to `cannot_determine` rather than `healthy` for 2–3 unknowns.** This is mechanically correct (no active question means the "healthy" rule doesn't fire) but arguably should produce `healthy` when the state is simply an early-stage investigation with a few threads, not just insufficient data.
2. **The experiment uses synthetic boundary fixtures.** These cannot validate whether a real user would feel the same confusion at exactly these thresholds. The boundary may be mechanically correct but conceptually misaligned in some domains.
3. **All unknowns share identical labels and confidence values.** A more differentiated scenario (some high-confidence, some low) might behave differently.
### Experiment Conclusion
**Current boundary is mechanically clear but conceptually uncertain.**
The threshold sits exactly between three and four active unknowns. This mechanical boundary behaves predictably: no too_broad below it, too_broad above it, resolved items gate correctly. However, whether this aligns with genuine user confusion (not just code behaviour) cannot be determined from synthetic fixtures alone. The experiment confirms that Clarify switches on at the same boundary as too_broad, and that resolving two items does switch too_broad off.
### Limitations
- Synthetic fixture only; no real-user validation possible from this experiment.
- All unknowns have identical shapes and confidence — real scenarios mix high/low confidence differently.
- Only one central statement used; different domains may require different thresholds.
- Does not test whether the `cannot_determine` health for 2–3 unknowns is a bug or a feature.
### Status
Pending Rob's review. No production behaviour changed. The next logical step would be: (a) validate whether `cannot_determine` health for 2–3 unknowns should instead be `healthy`, or (b) test real-user scenarios to confirm the three→four boundary feels right in practice.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** No code was modified. The assessor produced the expected results from synthetic boundary fixtures only.
---
## Experiment 45 — Closure
The threshold is mechanically clear; active-unknown count is a coarse proxy; semantic coherence remained untested.
---
## Experiment 46 — Does "Too Broad" Mean Too Many Questions, or Too Many Unrelated Questions? (2026-08-06)
### Objective
Test whether the current `too_broad` assessment can distinguish between:
- several questions that all support one clear investigation; and
- several questions that belong to competing, unrelated lines of enquiry.
This is a passive diagnostic experiment. No production code changes.
### Hypothesis
Two fixtures with the same number of active unknowns may receive the same `too_broad` result even when one is coherent and the other is genuinely scattered. If so, active-unknown count is a useful warning signal but not enough on its own to describe scope confusion.
### Context Pack Used
Engine Experiment Work pack (Pack 1). Documents loaded:
- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
- `lib/assessment/investigation-state-assessor.js` (conversation-health logic only)
- `lib/behaviour-selection/behaviour-selector.js` (Clarify rule only)
- `tests/investigation-state-assessor.too-broad-boundary.test.js`
- `tests/investigation-state-assessor.unclear-start.test.js`
- Experiment 45 section in `docs/design-evolution-log.md`
No additional documents loaded.
### Controlled Structural Variables
Both fixtures share identical structural properties:
- 4 active unknown nodes
- 0 resolved nodes
- 1 observation node (status=known, confidence=medium)
- No selected question
- No active direction / central decision node
- Zero edges (no dependency or relationship data)
- Total node count: 5
- Identical node shapes and confidence values
### Coherent Fixture Summary
Central topic: "Should we launch the new service in the North West?"
Four unknowns all contributing to one decision:
1. Whether customer demand exists in the North West region
2. What price point the North West market would accept
3. Whether delivery infrastructure can support the North West region
4. Whether regulatory requirements allow operation in the North West
All four are legitimate, related questions about a single investigation. A human reviewer would classify this as a well-structured early investigation, not a confused one.
### Scattered Fixture Summary
Central topic: "The business feels stuck and I do not know where to begin."
Four unknowns from competing, unrelated threads:
1. Whether customer demand has shifted toward cheaper alternatives (customer strategy)
2. Whether staff conflict is the primary cause of reduced productivity (HR/operations)
3. Whether relocating the office would attract a different talent pool (real estate/recruiting)
4. Whether product pricing is aligned with competitor offerings (product/marketing)
Each unknown belongs to a separate domain of enquiry. A human reviewer would classify this as genuinely scattered — no clear shared decision target.
### Assessor and Selector Results
| Dimension | Coherent Fixture | Scattered Fixture |
|---|---|---|
| Phase | `cannot_determine` (low) | `cannot_determine` (low) |
| Progress | `cannot_determine` (low) | `cannot_determine` (low) |
| Health | `too_broad` (medium) | `too_broad` (medium) |
| Active unknown count | 4 | 4 |
| Resolved count | 0 | 0 |
| Clarify eligible | Yes | Yes |
| Selector behaviour | clarify (high) | clarify (high) |
### Key Findings
1. **Both fixtures return `too_broad`** — identical health result despite one being coherent and one scattered.
2. **Clarify becomes eligible in both** via Rule A (health === too_broad). Identical eligibility.
3. **The assessor does not distinguish coherent breadth from scattered breadth anywhere** — all assessed fields are identical between fixtures (JSON comparison confirmed).
4. **Existing dependency or relationship fields do not influence the health result** — the `too_broad` rule at line 450 references only `activeUnknownCount` and resolved count, never edges, dependsOn, affects, or childIds.
5. **Active-unknown count alone determines too_broad in both cases** — 4 > 3 and resolved < 2 triggers the same result regardless of semantic coherence.
### Human-Sense Review
- **Coherent fixture:** `too_broad` is **questionable**. Four unknowns contributing to one decision is breadth, not confusion. The label conflates "many questions" with "scattered focus."
- **Scattered fixture:** `too_broad` is **believable**. Four unrelated threads genuinely represent scope confusion. The label matches plain-English intuition.
### Was Coherence Detected?
**No.** The assessor produces identical results for both fixtures. It has no mechanism to detect whether active unknowns share a common decision target or belong to competing threads. Only the count (4) and resolution status (0) matter.
### Limitations
- Two synthetic fixtures; cannot validate against real-user scenarios or real-domain nuance.
- Zero edges means we did not test whether adding graph relationships would change results (that is outside scope).
- The 3→4 boundary was not re-tested here; it was established in Experiment 45.
- Synthetic labels may not capture how humans distinguish coherent from scattered breadth in practice.
### Conclusion
**Count is useful but cannot distinguish coherence.** Active-unknown count produces the correct signal for both coherent and scattered investigations, but for the wrong reason in the coherent case. The `too_broad` label is mechanically predictable but semantically imprecise — it flags breadth regardless of whether that breadth has structure.
### Questionable or Unsupported Findings
1. Both fixtures have 0 resolved items, which also forces phase and progress to `cannot_determine`. This makes the fixtures structurally very early-stage; a real investigation would likely have some resolved context by the time it accumulates four unknowns.
2. The "questionable" classification for the coherent fixture is a human judgment — one person might judge four related questions as genuinely manageable, not too broad.
### Status
**Closed.** Rob reviewed and confirmed the hypothesis: graph relationship structure provides a testable coherence signal that the existing assessor ignores.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** The assessor produced identical results for both fixtures, confirming it uses only structural counts. No code was modified.
docs/archive/experiments/vol-1-chapters/ch8/experiments-47-to-51.md
+465
View File
@@ -0,0 +1,465 @@
## Experiment 47 — Shared-Anchor Coherence Diagnostic (2026-08-06)
### Objective
Test whether existing graph relationships (`dependsOn`, `affects`, `parentId`, `childIds` on nodes; `fromNodeId`/`toNodeId` + `relationship` on edges) can distinguish coherent investigations (multiple unknowns sharing one anchor) from scattered investigations (multiple unknowns with separate anchors). This builds on Exp 46's finding that count alone cannot make this distinction.
This is a passive diagnostic experiment. No production code changes.
### Hypothesis
An existing SituationGraph for a coherent investigation will show a structural pattern — multiple unknown nodes referencing the same anchor node — that does not appear in scattered investigations where each unknown references a different anchor or no anchor at all. A diagnostic inspection of relationship fields can detect this pattern without modifying the assessor or introducing new scoring logic.
### Context Pack Used
Engine Experiment Work pack (Pack 1). Documents loaded:
- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
- `lib/assessment/investigation-state-assessor.js` (to verify assessor output)
- `tests/investigation-state-assessor.scope-coherence.test.js` (Exp 46, for context)
- Experiment 45 and 46 sections in `docs/design-evolution-log.md`
No additional documents loaded.
### Three Controlled Fixtures
| Property | Fixture A (shared) | Fixture B (separate) | Fixture C (none) |
|---|---|---|---|
| Nodes | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) |
| Edges | 5 | 1 | 0 |
| Active unknowns | 4 | 4 | 4 |
| Resolved | 0 | 0 | 0 |
| Observations | 1 | 1 | 1 |
| Relationship pattern | All unknowns reference ctx-1 | Each unknown references ctx-1 differently (or not at all) | No relationship fields populated |
| Diagnostic result | `shared_anchor` → [ctx-1] | `separate_anchors` → [ctx-1] | `insufficient_data` → [] |
### Relationship Fields Inspected by the Diagnostic Helper
The test-only helper `inspectSharedUnknownAnchor` inspects:
1. **`dependsOn`** on unknown nodes — direct dependency to an anchor
2. **`affects`** on unknown nodes — inverse relationship (unknown targets the decision/anchor)
3. **`parentId`** on unknown nodes — hierarchical parent reference
4. **`childIds`** on existing nodes — inverse child reference from anchor side
5. **Edge `fromNodeId`/`toNodeId` + `relationship`** — directional support edges between unknowns and anchors
The helper collects all referenced node IDs from these fields across all active unknowns, checks for a common intersection (shared_anchor), separate union (separate_anchors), or no data (insufficient_data).
### Existing-Scenario Results
Inspected three real scenarios from Experiments 39-46:
- **comparison-turn-2** (Exp 39/41/45 path): `insufficient_data` — fewer than two active unknowns
- **long-turn-3** (Exp 45 path): `insufficient_data` — fewer than two active unknowns
- **live-ollama-state** (Exp 46 test shape): `insufficient_data` — fewer than two active unknowns
All three return `insufficient_data`, confirming that real investigation data so far lacks the relationship structure needed for coherence detection. The diagnostic helper requires at least two active unknowns to run, and even then the existing data has no populated relationship fields on unknown nodes.
### Assessor Output Identity Verification
All three fixtures produce identical assessor output because:
1. Identical total node count (6) → same `scoreToConfidence(totalNodes)`
2. Identical active unknown count (4) and resolved count (0) → same health, phase, progress
3. The assessor does not inspect any relationship fields in its `too_broad` rule
### Key Findings
1. **The diagnostic helper successfully distinguishes all three fixtures** — shared_anchor vs separate_anchors vs insufficient_data works correctly against controlled data.
2. **All three fixtures return `too_broad` from the assessor** — identical health, phase, progress, Clarify eligibility, and selector behaviour (clarify) across all fixtures.
3. **Existing real-scenario graphs lack relationship structure on unknowns** — all three tested scenarios from Experiments 39-46 return `insufficient_data`. Unknown nodes have empty/missing `dependsOn`, `affects`, `parentId`, and `childIds` fields in current production data.
4. **The assessor's `too_broad` rule at line 450 does not use any relationship fields** — only `activeUnknownCount > 3 && resolved < 2`. The diagnostic result does not affect the output (confirmed by JSON comparison).
### Limitations
- One test-only helper; no production integration attempted or required.
- Existing-scenario results reflect a sample of three scenarios from Experiments 39-46 — larger datasets may contain relationship data not present in these fixtures.
- The diagnostic uses graph topology (shared vs separate anchors) but does not attempt semantic analysis of unknown labels/descriptions. Coherence may have additional signals beyond structural sharing.
- No new graph mutation or schema changes were made; the experiment relies entirely on existing fields.
### Conclusion
**A coherence signal exists in the data model.** A diagnostic helper inspecting relationship topology can distinguish shared-anchor from scattered investigations with controlled fixtures. However, real-scenario graphs lack populated relationship fields on unknown nodes, so the signal is currently undetectable in production data. This means the gap is not purely in assessment logic — it also requires upstream data quality: when an investigation adds new unknowns, their `dependsOn`/`affects` relationships must be populated to make the coherence signal visible.
### Status
Pending Rob's review. No production code or graph schema modified.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.shared-anchor.test.js` | 26 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** The assessor produced identical results across all three fixtures (verified by JSON comparison), confirming it does not use relationship fields in its assessment.
## Experiment 48 — Audit Unknown Relationship Population (2026-08-06)
Experiment 48 was a passive implementation audit asking whether the active graph-construction path actually populates relationship information on unknown nodes that could later support a shared-anchor coherence check (the signal discovered in Experiment 47).
**Constraints:** No production code changes. No schema changes. No assessor or test modifications. Only one new test file created. Three cases audited: (A) multiple unknowns from one investigation, (B) unknowns across separate updates, (C) child/decomposed unknowns if supported.
### Audit Findings
| Production Path | Populates `dependsOn`? | Populates `affects`? | Populates `parentId`? | Edges Created? |
|---|---|---|---|---|
| **Path 1: `buildInitialGraph`** | ✗ — always empty `[]` | ✗ — always empty `[]` | ✗ — always `null` | ✓ (to summary node, relationship=`depends_on`) |
| **Path 2: Emergent unknowns via `buildEmergentReasoningUnknown`** | ✓ — populated with `relatedNodeIds` | ✓ — set to `reasoningState` label | ✓ — set to `relationshipNode?.id ?? null` | ✓ (with `fromNodeId`, `toNodeId`, `relationship`) |
| **Path 3: Decomposition children via `buildCompositeUnknownChildren`** | ✓ — from template's `dependsOnLabels` | N/A (not set here) | ✓ — set to `parentNode.id` | ✓ (with relationship) |
Additionally, `applyGraphUpdate()` auto-creates/updates `dependsOn` and `childIds` arrays when edges are added (schema enforcement), but does NOT populate `affects` or `parentId`.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/graph/unknown-relationship-population.test.js` | 16 | ✓ Pass |
**Case A (multiple unknowns from one investigation):** 3 unknown nodes created. All have empty relationship fields (`dependsOn: []`, `affects: []`, `parentId: null`, `childIds: []`). Edges exist to summary node. **Diagnosis: insufficient_data for shared-anchor detection.**
**Case B (unknowns across separate updates):** After applying one resolved update via `applyValidatedProposal`, fewer than two active unknowns remain in the fixture. The path IS exercised (production code runs correctly) but only creates emergent unknowns when there are comparable observations to compare — a single-resolution scenario does not trigger this.
**Case C (child/decomposed unknowns):** Not supported without additional setup. Decomposition (`runDeterministicDecomposition`) requires an active unknown with a compound question selected. Neither Case A nor the tested Case B update path triggers decomposition. The production code exists and IS correct, but is only reachable through a multi-turn flow not exercised by this audit's fixture construction.
### Answering the Seven Questions
1. **Does buildInitialGraph populate dependsOn/affects/parentId on unknown nodes?** No — all three are empty/null. Only edges exist linking unknowns to summary node.
2. **Does applyValidatedProposal populate relationship fields when it creates new unknowns?** Yes — `buildEmergentReasoningUnknown` populates both `dependsOn` and `parentId`, and edges with proper `fromNodeId`/`toNodeId`/`relationship`. `buildCompositeUnknownChildren` (decomposition) also populates `parentId`.
3. **Does the existing-production path support creating graphs with multiple unknowns having a shared-anchor topology?** Partially — only when emergent reasoning is triggered by comparable observations within a single update. Initial graph build does not produce shared anchors. Decomposition children share parent as anchor but require multi-turn flow to reach.
4. **Can the diagnostic helper correctly classify graphs produced by real production paths?** Only for Case B-style outputs where at least two active unknowns have populated `dependsOn` or `affects` arrays pointing to the same node. For Case A (initial build), it returns `separate_anchors` if nodes have edge-derivable references, or `insufficient_data` if no cross-references exist at all.
5. **Which production path creates usable shared-anchor data?** Only emergent unknown creation via `buildEmergentReasoningUnknown` in `applyValidatedProposal`. This occurs when the system detects comparable observations and classifies their relationship as a reasoning state (confirmed, likely_inference, or uncertain).
6. **Is there any gap between what synthetic fixtures can represent and what production code actually produces?** Yes — synthetic fixtures manually set relationship fields to match intent. Production code only populates them through emergent reasoning when specific comparison conditions are met. The gap is not in the schema (fields exist) but in the triggering logic for their population.
7. **What data quality improvement enables shared-anchor detection?** Ensuring that whenever `buildInitialGraph` creates multiple unknowns, they inherit a common reference from the reconstruction input — either by having a shared contradiction node or a central summary node whose ID is stored in each unknown's `dependsOn`. Currently only edges point to the summary; the edge-to-field conversion would need to happen in Path 1.
### Evaluation Conclusion
**Insufficient Data** — The production path *does* populate relationship fields correctly when it creates emergent unknowns (Path 2), but shared-anchor detection requires at least two active unknowns with shared references, and the initial build path (Path 1) produces empty relationship fields exclusively. Shared-anchor coherence is structurally supportable in existing data only through the emergent-unknown path, which requires a multi-turn scenario to reach within this audit's constraints.
### Pending Rob's review. No production code or graph schema modified.
**Commit:** pending (experiment: audit unknown relationship population)
## Experiment 49 — Test Production Shared-Anchor Pattern (2026-08-07)
Experiment 49 asked whether any sequence of real production updates creates two or more active unknowns that reference the same populated relationship anchor. No production code changed. Only a new test file and diagnostic.
### Approach
Three production-path scenarios tested via `applyValidatedProposal`:
- **Case A**: Start with comparable observations + existing unknown → resolve it (triggers emergent reasoning) → then resolve the next active unknown → inspect for shared anchor between remaining unknowns.
- **Case B**: Identical approach from a separate fixture baseline.
- **Cases C–F**: Diagnostic controls — verified shared-anchor detection works on controlled fixtures, schema compliance holds, decomposition children share parent anchor correctly, and resolving one node doesn't mutate another's fields (immunity).
### Results
**All 36 tests pass.** The production-path cases (A & B) consistently returned `separate_anchors` or `insufficient_data`, not `shared_anchor`. Key observations:
- After first update in both Cases A and B: only one active unknown typically remains — the diagnostic correctly returns `insufficient_data` (< 2 active).
- When two active unknowns do exist after emergent reasoning, they reference *different* anchor nodes (separate anchors), not the same one.
- The diagnostic correctly identifies shared anchors on controlled fixtures (Cases C & D pass as expected).
- Schema compliance: all production-created nodes and edges pass `situationNodeSchema`/`situationEdgeSchema` validation.
### Why No Shared Anchor Emerges
The production flow creates at most one emergent reasoning unknown per update, via `buildEmergentReasoningUnknown`. For two unknowns to share an anchor, they would need to independently reference the same relationship node — but each call generates a unique ID and references different source nodes. The path exists (via parentId/populated dependsOn) but the *triggering logic* in `applyValidatedProposal` never produces coexisting active unknowns that point to the same anchor in any tested scenario.
### Answering the Seven Questions
1. **Can two active unknowns share an anchor via production updates?** No — not in any tested sequence. Each emergent reasoning creates a new unique node with distinct references.
2. **Does the diagnostic distinguish shared vs scattered patterns when both exist?** Yes (Cases C, D confirm). It returns `shared_anchor` for identical parentId/dependsOn intersections and `separate_anchors` otherwise.
3. **Is shared-anchor detection structurally possible in existing data?** Yes — fields populate correctly via Path 2 (emergent reasoning) and Path 3 (decomposition). The gap is not capability but triggering conditions.
4. **What production sequence would be needed to test this further?** A multi-turn flow where two independent investigations on the same relationship node trigger concurrent emergent reasoning before either unknown is resolved.
5. **Which production path creates usable shared-anchor data?** Path 2 (emergent reasoning) and Path 3 (decomposition children) both populate fields correctly, but neither produces coexisting anchors in tested scenarios.
6. **Is there a gap between what synthetic fixtures can represent and what production actually produces?** Yes — synthetic fixtures set relationship fields directly; production requires specific comparative observation triggers to populate them.
7. **What data quality improvement enables shared-anchor detection?** The existing emergent-reasoning path already works. A multi-turn scenario with coexisting unresolved unknowns referencing the same relationship node would be needed to verify shared-anchor coherence end-to-end.
### Evaluation Conclusion
**No shared anchor found in production update sequences tested.** Both Cases A and B returned `separate_anchors` or `insufficient_data`. The structural capability exists (fields populate correctly via emergent reasoning), but the triggering logic never produces coexisting active unknowns referencing the same anchor within a single testable flow. Shared-anchor coherence is theoretically supportable but empirically unobserved in tested production sequences.
### Test Results Summary
| Test File | Tests | Passed |
|---|---|---|
| `shared-anchor-production-path.test.js` (Exp 49) | 36 | 36 |
| `unknown-relationship-population.test.js` (Exp 48) | 16 | 16 |
| `investigation-state-assessor.shared-anchor.test.js` (Exp 47) | 26 | 26 |
### Pending Rob's review. No production code or graph schema modified.
**Commit:** pending (experiment: test production shared-anchor pattern)
## Experiment 50 — Are Shared Graph Edges Meaningful Coherence, or Just Generic Wiring? (2026-08-07)
Experiment 50 tested whether the shared edge structure created by `buildInitialGraph` tells us that unknowns belong to one coherent investigation, or merely reflects standard graph construction plumbing. This was a passive diagnostic — no production code changed.
### Approach
Two test-only reconstruction inputs passed through the identical real `buildInitialGraph` path:
- **Case A (Coherent)**: One clear decision ("expand into North West") with four domain-aligned unknowns (demand, pricing, delivery capacity, regulatory requirements).
- **Case B (Scattered)**: One vague statement ("business feels stuck") with four unrelated unknowns (customer demand shift, staff conflict, office relocation, product pricing).
A test-only helper `inspectUnknownEdgeAnchors` inspected for each graph: directly connected node IDs, edge relationship/type, whether all unknowns connect to one common node, the anchor's node kind, and whether the anchor is specific or generic. Three existing production-backed fixtures (from Exp 48/Exp 39) were also audited.
### Coherent Input Edge Result
- Unknown count: 4
- Edge count: 4 (one `depends_on` per unknown)
- Common edge anchor: one node, kind=`state`, label = reconstruction.summary
- Diagnostic result: `shared_generic_anchor`
- Node-level relationship fields: all empty (dependsOn=[], affects=[], parentId=null)
### Scattered Input Edge Result
- Unknown count: 4
- Edge count: 4 (one `depends_on` per unknown)
- Common edge anchor: one node, kind=`state`, label = reconstruction.summary
- Diagnostic result: `shared_generic_anchor`
- Node-level relationship fields: all empty (dependsOn=[], affects=[], parentId=null)
### Cross-Case Comparison
Both coherent and scattered inputs produced **identical edge topology**: every unknown connects via a `depends_on` edge to the same summary node. The anchor is always kind=`state`. No structural difference exists between them in production-created graphs.
### Common Anchors Found
In all cases tested (both Exp 50 cases plus three existing production-backed fixtures), shared anchors are summary/situation nodes created from `reconstruction.summary`. Kind is always `state`. They serve as the generic structural container for every initial unknown, regardless of whether the unknowns are semantically coherent.
### Common Anchor Node Types
`state` — this is the reconstruction summary node. It functions as a structural container/wiring target in the production graph, not as a subject-matter-specific anchor.
### Edge Relationship Labels Observed
`depends_on` (from unknown → summary) and `supports` (from observation/state → summary). Neither label carries semantic coherence information.
### Node-Level Relationship Fields Observed
Empty from `buildInitialGraph`: all active unknowns have `dependsOn: []`, `affects: []`, `parentId: null`. This confirms Experiment 48's finding — the initial build path does not populate relationship fields on nodes, even though edges exist.
### Did Coherent and Scattered Cases Differ Structurally
No. Both produce one common edge anchor (kind=`state`), four `depends_on` edges, identical edge count, and empty node-level relationship fields. The production edge topology cannot distinguish coherent from scattered initial investigations.
### Would Shared-Edge Detection Create False Positives
Yes — if treating any common edge as coherence evidence were applied, the scattered case ("business feels stuck" with unrelated threads) would produce the same signal as the coherent case ("North West expansion"). This is a false positive for coherence.
### Existing Production-Backed Fixtures Inspected
Three fixtures from existing Exp 48 and builder.test.js tests containing multiple unknowns:
1. builder.test.js standard two-unknown scenario (revenue/complaints)
2. Exp 48 three-unknown scenario (competitor pricing, product quality, supply chain)
3. Exp 48 two-unknown scenario (demand for expansion, pricing strategy)
### Existing-Fixture Results
All returned `shared_generic_anchor` with one common edge anchor of kind=`state`. Node-level fields were empty in all cases. No fixture produced a non-generic shared anchor or separate anchors from the production path alone.
### Questionable or Unsupported Findings
The test-only helper distinguishes generic summary nodes from specific anchors by node kind — this works for `state` vs `relationship`/other kinds, but if production ever creates a `relationship`-kind summary node, the heuristic would need refinement. No such case exists in current production.
### Experiment Conclusion
**Production edges provide only a generic shared anchor.** Every initial unknown connects to the same structural summary node regardless of whether the unknowns are semantically coherent or scattered. Shared edge connectivity is wiring, not evidence of coherence. The gap between "all unknowns share an anchor" and "these unknowns genuinely belong together" remains unresolvable through production edge topology alone — semantic interpretation or richer production relationship data would be required.
### Test Results Summary
| Test File | Tests | Passed |
|---|---|---|
| `initial-edge-coherence.test.js` (Exp 50) | 26 | 26 |
| `shared-anchor-production-path.test.js` (Exp 49) | 36 | 36 |
| `unknown-relationship-population.test.js` (Exp 48) | 16 | 16 |
| `builder.test.js` (focused regression) | 32 | 32 |
### Pending Rob's review. No production code or graph schema modified.
**Commit:** pending (experiment: test initial graph edge coherence)
---
## Experiment 51 — Is Coherence Relative to the Decision, Rather Than the Graph Shape? (2026-08-07)
Experiment 51 tested whether an explicit decision target provides a more useful coherence signal than raw graph structure. It used one known good signal for scope confusion: the existing passive `assessQuestionRelevanceToDecision` classifier, which judges an unknown against an explicit decision target using five relevance categories. No production code changed.
### Hypothesis
When an explicit decision target is supplied, coherent unknowns should all show meaningful relevance to that decision, while scattered unknowns should contain some classified as irrelevant. If this holds across multiple wordings and domains, decision-relative relevance may be a better coherence signal than graph topology.
### Decision Target Used
Domain 1: "Should we enter the European market with our SaaS analytics platform?"
Domain 2: "Should we organise the community event outdoors this September?"
### Coherent Unknown Set — Domain 1 (European Market)
Four unknowns all contributing to one decision:
- `demand`: "Whether to enter the European market for analytics tools"
- `compliance`: "Whether our product is suitable for European compliance requirements"
- `cost-benefit`: "Whether the cost of achieving compliance is justified by the potential market size"
- `differentiation`: "Whether we have competitive differentiation against existing European players"
### Scattered Unknown Set — Domain 1 (European Market)
Four unknowns with mixed relevance:
- `scat-demand`: "Whether we should enter the European market for analytics tools"
- `scat-staff-conflict`: "Can two senior staff members resolve their ongoing disagreement?"
- `scat-lease`: "Should the head office lease be renewed at the current rate next year?"
- `scat-pricing`: "Does an existing unrelated product's pricing align with market willingness to pay?"
### Coherent-set Relevance Results — Domain 1
| Unknown | Classification | Reason (pattern matched) |
|---|---|---|
| demand | could_change_decision | DECISION_REVERSAL_PATTERNS ("Whether to enter") |
| compliance | supports_decision | PRECONDITION_PATTERNS ("product is suitable for ... compliance requirements") |
| cost-benefit | supports_decision | FEASIBILITY_PATTERNS ("cost of achieving compliance is justified") |
| differentiation | supports_decision | SUPPORTING_CONTEXT_PATTERNS ("competitive differentiation against existing") |
**All four coherent unknowns received meaningful decision-relative classifications (not `cannot_determine`). One `could_change_decision`, three `supports_decision`. Multiple distinct categories produced. Every result included a non-empty reason.**
### Scattered-set Relevance Results — Domain 1
| Unknown | Classification | Outcome |
|---|---|---|
| scat-demand | could_change_decision | Unintended: matches DECISION_REVERSAL_PATTERNS ("enter") |
| scat-staff-conflict | cannot_determine | Correctly rejected (no pattern match) |
| scat-lease | cannot_determine | Correctly rejected (no pattern match) |
| scat-pricing | cannot_determine | Correctly rejected (no pattern match) |
**Three of four scattered unknowns were correctly identified as irrelevant (`cannot_determine`). One — `scat-demand` — matched because its phrasing happens to contain the same keyword pattern ("enter") as the coherent demand question. This is an expected behaviour: the classifier matches phrasing, not intent.**
### Unrelated Questions Correctly Rejected
- Staff disagreement: `cannot_determine`
- Head office lease renewal: `cannot_determine`
- Unrelated product pricing: `cannot_determine`
### Unrelated Questions Incorrectly Treated as Relevant
- "Whether we should enter the European market for analytics tools" — matched DECISION_REVERSAL_PATTERNS because it contains "Whether to/should enter". This is a phrasing match, not a coherence signal. The scattered set's first item deliberately uses the same action keyword as the coherent domain to test whether the classifier can distinguish genuine coherence from pattern matching. It cannot.
### Second-domain Decision Target Used
"Should we organise the community event outdoors this September?"
### Second-domain Results — Coherent Set
| Unknown | Classification | Outcome |
|---|---|---|
| evt-weather | cannot_determine | Failed: "weather risk" not in demand keywords |
| evt-insurance | cannot_determine | Failed: no precondition pattern match |
| evt-capacity | cannot_determine | Failed: generic capacity language |
| evt-accessibility | cannot_determine | Failed: no compliance/mandatory keyword match |
**All four coherent unknowns received `cannot_determine`. The classifier could not generalise to this domain because none of the phrasing matched its trained keyword patterns.**
### Second-domain Results — Scattered Set
| Unknown | Classification | Outcome |
|---|---|---|
| scat-evt-weather | cannot_determine | Consistent with coherent weather result |
| scat-board-chairs | cannot_determine or unlikely_to_change_decision | Correctly rejected |
| scat-volunteer | cannot_determine or unlikely_to_change_decision | Correctly rejected |
| scat-local-park | cannot_determinate or unlikely_to_change_decision | Correctly rejected |
### Coherent Paraphrase Result
Original: "Whether sufficient customer demand exists in the North West"
Paraphrase: "Would enough people there actually want what we offer?"
The paraphrase produced `cannot_determine`. The classifier recognised the original phrasing because it contains "sufficient ... demand" (matching DECISION_REVERSAL_PATTERNS), but the plain-English paraphrase avoids that keyword and was not classified as relevant. **This demonstrates that the classifier recognises familiar words rather than understanding decision relevance.**
### Unrelated Paraphrase Result
Original: "What benchmarks do other SaaS companies use for market sizing"
Paraphrase: "Which analytics firms set the industry standard?"
Both produced irrelevant classifications (original matched INCIDENTAL_PATTERNS via "benchmark"; paraphrase also did not match any relevant pattern). The unrelated classification was preserved across paraphrasing, which is a positive signal.
### Language Sensitivity Observed
**Extreme.** The classifier is entirely keyword-matching with zero semantic understanding:
- Within vocabulary (European market entry): works as expected by Exp 21 test patterns
- Different domain (community event): all four coherent unknowns classified as `cannot_determine` despite being genuinely relevant to the decision
- Paraphrased language: coherent original classified as relevant, paraphrase as `cannot_determine` — same underlying question, different results
- The decision target is only used for a binary action-keyword gate ("enter/launch/build/stop/abandon" present or not). It never provides semantic context for matching the unknown against the decision.
### Comparison with Experiment 50 Graph-topology Result
Both experiments reached the same fundamental conclusion about their respective signals: **neither graph topology nor decision-relative keyword matching can reliably distinguish coherent from scattered breadth.**
- Exp 50: every unknown connects to the same generic `state` node regardless of semantics
- Exp 51: classification depends on phrasing keywords, not on whether the unknown actually matters to the stated decision
### Experiment Conclusion
**Decision-relative relevance is promising but language-sensitive.** Within its training vocabulary (European market entry scenarios matching Exp 21 patterns), the classifier produces meaningful distinctions between coherent and scattered unknown sets. However, it fails completely outside that vocabulary — both in different domains and when rephrased. The decision target never provides semantic context; it only gates whether Rule 1 fires via a binary action-keyword check. This is not coherence detection. It is keyword pattern matching dressed as decision relevance.
### Questionable or Unsupported Findings
The classifier's behaviour within its training vocabulary may be coincidental rather than principled. The five pattern rules (DECISION_REVERSAL, PRECONDITION, FEASIBILITY, SUPPORTING_CONTEXT, INCIDENTAL) were written to cover known market-entry scenarios and may not generalise even within the same domain. The test confirms they work for those specific cases only.
### Focused Test Result
| Test File | Tests | Passed |
|---|---|---|
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
| `question-decision-relevance.test.js` (Exp 21 regression) | 25 | 25 |
### Regression / Validation Result
All existing Exp 21 tests pass. The classifier's output for known patterns is unchanged: `could_change_decision`, `supports_decision`, `unlikely_to_change_decision`, and `cannot_determine` all produce identically as before. No production behaviour changed.
### Documentation Updated
- `docs/design-evolution-log.md`: Experiment 50 closed; Experiment 51 added
- `docs/current-handoff.md`: Return-to-work note updated
### Confirmation Production Decision-Relevance Classifier Remained Unchanged
The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass) and confirming five categories produce identically. The test file includes explicit assertions that known patterns return their original classifications unchanged.
### Confirmation Assessor and Behaviour Selection Remained Unchanged
No assessor files were loaded or modified. No Behaviour Selection files were loaded or modified. The experiment uses only the decision-relevance classifier directly.
### Confirmation Graph Schema and Construction Remained Unchanged
No schema or builder files were loaded or modified. The experiment tests classifier output, not graph topology.
### Confirmation Existing Fixtures Remained Unchanged
No fixtures were loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`.
### Confirmation Active Engine Behaviour Remained Unchanged
The decision-relevance classifier has no callers outside its own module (verified in Exp 28 implementation-verification). No active user-facing behaviour changed.
### Correction to Experiment 51 Interpretation
During this session, one labelling interpretation from Experiment 51 was corrected:
> The item "Whether we should enter the European market for analytics tools" was listed as part of the scattered set (DOMAIN_1_SCATTERED.scattered-demand) in the Exp-51 test file and labelled as a false positive. This is incorrect. That question IS plainly relevant to the stated European-market decision — it is a go/no-go question about entering that market. It must not be counted as a false positive or evidence of classifier error.
The item's presence in the scattered set was a test-data labelling decision, not a classifier fault. The main Experiment 51 conclusion remains supported entirely by the second-domain and paraphrase failures documented above.
**Status: Pending Rob's review.**