experiment: test shared-anchor coherence signal

Experiment 47: created a test-only diagnostic helper that inspects existing
graph relationship fields (dependsOn, affects, parentId, childIds on nodes;
fromNodeId/toNodeId + relationship on edges) to distinguish coherent
investigations (multiple unknowns sharing one anchor) from scattered ones.

Three controlled fixtures confirm the helper works: shared_anchor vs
separate_anchors vs insufficient_data — all with identical structural counts
(6 nodes, 4 active unknowns). All three produce identical too_broad output
from the existing assessor, confirming no production code changes needed.

Existing-scenario inspection (3 real scenarios from Exp 39-46) all return
insufficient_data — current data lacks populated relationship fields on
unknown nodes. This means the gap is not purely in assessment logic but also
in upstream data quality.

Closed Experiment 46. Updated design-evolution-log and handoff.
This commit is contained in:
2026-08-06 19:31:43 +01:00
parent 0de2ffb4be
commit 40ef3e108f
3 changed files with 640 additions and 6 deletions
+10 -5
View File
@@ -48,14 +48,19 @@ Experiment 43 audited Clarify readiness across all 10 real assessment turns in e
Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned `too_broad` conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 23 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.
Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns — two and three returned cannot_determine; four and five returned too_broad. Clarify eligibility followed the same boundary. Resolved-item gate works correctly: one resolved item stays too_broad, two resolves it. The boundary appears mechanically clear but conceptually uncertain — synthetic fixtures cannot confirm whether three-to-four feels right to real users. No production code changed. What remains open: whether health should default to healthy (not cannot_determine) for 23 unknowns with no question; whether the threshold needs widening for real-world use. Status pending Rob's review.
Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns — two and three returned cannot_determine; four and five returned too_broad. Clarify eligibility followed the same boundary. Resolved-item gate works correctly: one resolved item stays too_broad, two resolves it. The boundary appears mechanically clear but conceptually uncertain — synthetic fixtures cannot confirm whether three-to-four feels right to real users. No production code changed. What remains open: whether health should default to healthy (not cannot_determine) for 23 unknowns with no question; whether the threshold needs widening for real-world use. Status closed.
Experiment 46 compared two four-unknown investigations with identical structural counts — one coherent (four unknowns contributing to one decision) and one scattered (four unrelated threads). Both returned too_broad with Clarify eligible, confirming the assessor cannot distinguish semantic coherence from scatter using active-unknown count alone. No production behaviour changed. Status closed.
Experiment 47 created a test-only diagnostic helper (`inspectSharedUnknownAnchor`) that inspects existing graph relationship fields to distinguish shared-anchor investigations from scattered ones. Three controlled fixtures (shared/separate/none anchors, all with identical structural counts) confirmed the helper correctly distinguishes all three patterns. Inspecting three real scenarios from Experiments 39-46 returned insufficient_data for all — existing data lacks populated relationship fields on unknown nodes. The assessor remains unchanged. Status pending Rob's review.
## 5. What Remains Open
- The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
- Health defaults to `cannot_determine` rather than `healthy` for 23 unknowns (no active question present); whether this is a bug or feature needs review;
- Whether the `too_broad` threshold needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states;
- Existing-scenario graphs lack populated relationship fields on unknown nodes — coherence detection requires upstream data quality improvement (populating dependsOn/affects when adding unknowns).
### When This Knowledge-Management Phase Is Complete
@@ -101,8 +106,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43, 44, 45, 46. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 3847. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 45)
### Return-to-Work Note (Experiment 47)
Experiment 46 compared two four-unknown investigations with identical structural counts — one coherent (four unknowns contributing to one decision) and one scattered (four unrelated threads). Both returned too_broad with Clarify eligible, confirming the assessor cannot distinguish semantic coherence from scatter using active-unknown count alone. No production behaviour changed. What remains open: whether a coherence signal should be added, and what threshold would feel right in real-user data. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/investigation-state-assessor.scope-coherence.test.js` for results, then `docs/design-evolution-log.md` Experiment 46 section for full analysis.
Experiment 47 created a test-only diagnostic helper (`inspectSharedUnknownAnchor`) that inspects existing relationship fields (dependsOn, affects, parentId, childIds on nodes; fromNodeId/toNodeId + relationship on edges) to distinguish shared-anchor investigations from scattered ones. Three controlled fixtures (shared/none/separate anchors, all with identical structural counts of 6 nodes and 4 active unknowns) confirmed the helper correctly distinguishes all three patterns. Inspecting three real scenarios from Experiments 39-46 returned insufficient_data for all — existing data lacks populated relationship fields on unknown nodes. The assessor remains unchanged (produces identical too_broad output across all fixtures). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/investigation-state-assessor.shared-anchor.test.js` for results, then `docs/design-evolution-log.md` Experiment 47 section for full analysis.