robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test shared-anchor coherence signal

Browse Source 
Experiment 47: created a test-only diagnostic helper that inspects existing
graph relationship fields (dependsOn, affects, parentId, childIds on nodes;
fromNodeId/toNodeId + relationship on edges) to distinguish coherent
investigations (multiple unknowns sharing one anchor) from scattered ones.

Three controlled fixtures confirm the helper works: shared_anchor vs
separate_anchors vs insufficient_data — all with identical structural counts
(6 nodes, 4 active unknowns). All three produce identical too_broad output
from the existing assessor, confirming no production code changes needed.

Existing-scenario inspection (3 real scenarios from Exp 39-46) all return
insufficient_data — current data lacks populated relationship fields on
unknown nodes. This means the gap is not purely in assessment logic but also
in upstream data quality.

Closed Experiment 46. Updated design-evolution-log and handoff.
This commit is contained in:
robbond
2026-08-06 19:31:43 +01:00
parent 0de2ffb4be
commit 40ef3e108f
3 changed files with 640 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design-evolution-log.md
+101 -1
View File
@@ -3046,7 +3046,7 @@ Each unknown belongs to a separate domain of enquiry. A human reviewer would cla
### Status
Pending Rob's review. No production behaviour changed.
**Closed.** Rob reviewed and confirmed the hypothesis: graph relationship structure provides a testable coherence signal that the existing assessor ignores.
### Focused Test Results
@@ -3066,3 +3066,103 @@ Pending Rob's review. No production behaviour changed.
### Production Assessor Status
**Unchanged.** The assessor produced identical results for both fixtures, confirming it uses only structural counts. No code was modified.
## Experiment 47 — Shared-Anchor Coherence Diagnostic (2026-08-06)
### Objective
Test whether existing graph relationships (`dependsOn`, `affects`, `parentId`, `childIds` on nodes; `fromNodeId`/`toNodeId` + `relationship` on edges) can distinguish coherent investigations (multiple unknowns sharing one anchor) from scattered investigations (multiple unknowns with separate anchors). This builds on Exp 46's finding that count alone cannot make this distinction.
This is a passive diagnostic experiment. No production code changes.
### Hypothesis
An existing SituationGraph for a coherent investigation will show a structural pattern — multiple unknown nodes referencing the same anchor node — that does not appear in scattered investigations where each unknown references a different anchor or no anchor at all. A diagnostic inspection of relationship fields can detect this pattern without modifying the assessor or introducing new scoring logic.
### Context Pack Used
Engine Experiment Work pack (Pack 1). Documents loaded:
- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
- `lib/assessment/investigation-state-assessor.js` (to verify assessor output)
- `tests/investigation-state-assessor.scope-coherence.test.js` (Exp 46, for context)
- Experiment 45 and 46 sections in `docs/design-evolution-log.md`
No additional documents loaded.
### Three Controlled Fixtures
| Property | Fixture A (shared) | Fixture B (separate) | Fixture C (none) |
|---|---|---|---|
| Nodes | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) |
| Edges | 5 | 1 | 0 |
| Active unknowns | 4 | 4 | 4 |
| Resolved | 0 | 0 | 0 |
| Observations | 1 | 1 | 1 |
| Relationship pattern | All unknowns reference ctx-1 | Each unknown references ctx-1 differently (or not at all) | No relationship fields populated |
| Diagnostic result | `shared_anchor` → [ctx-1] | `separate_anchors` → [ctx-1] | `insufficient_data` → [] |
### Relationship Fields Inspected by the Diagnostic Helper
The test-only helper `inspectSharedUnknownAnchor` inspects:
1. **`dependsOn`** on unknown nodes — direct dependency to an anchor
2. **`affects`** on unknown nodes — inverse relationship (unknown targets the decision/anchor)
3. **`parentId`** on unknown nodes — hierarchical parent reference
4. **`childIds`** on existing nodes — inverse child reference from anchor side
5. **Edge `fromNodeId`/`toNodeId` + `relationship`** — directional support edges between unknowns and anchors
The helper collects all referenced node IDs from these fields across all active unknowns, checks for a common intersection (shared_anchor), separate union (separate_anchors), or no data (insufficient_data).
### Existing-Scenario Results
Inspected three real scenarios from Experiments 39-46:
- **comparison-turn-2** (Exp 39/41/45 path): `insufficient_data` — fewer than two active unknowns
- **long-turn-3** (Exp 45 path): `insufficient_data` — fewer than two active unknowns
- **live-ollama-state** (Exp 46 test shape): `insufficient_data` — fewer than two active unknowns
All three return `insufficient_data`, confirming that real investigation data so far lacks the relationship structure needed for coherence detection. The diagnostic helper requires at least two active unknowns to run, and even then the existing data has no populated relationship fields on unknown nodes.
### Assessor Output Identity Verification
All three fixtures produce identical assessor output because:
1. Identical total node count (6) → same `scoreToConfidence(totalNodes)`
2. Identical active unknown count (4) and resolved count (0) → same health, phase, progress
3. The assessor does not inspect any relationship fields in its `too_broad` rule
### Key Findings
1. **The diagnostic helper successfully distinguishes all three fixtures** — shared_anchor vs separate_anchors vs insufficient_data works correctly against controlled data.
2. **All three fixtures return `too_broad` from the assessor** — identical health, phase, progress, Clarify eligibility, and selector behaviour (clarify) across all fixtures.
3. **Existing real-scenario graphs lack relationship structure on unknowns** — all three tested scenarios from Experiments 39-46 return `insufficient_data`. Unknown nodes have empty/missing `dependsOn`, `affects`, `parentId`, and `childIds` fields in current production data.
4. **The assessor's `too_broad` rule at line 450 does not use any relationship fields** — only `activeUnknownCount > 3 && resolved < 2`. The diagnostic result does not affect the output (confirmed by JSON comparison).
### Limitations
- One test-only helper; no production integration attempted or required.
- Existing-scenario results reflect a sample of three scenarios from Experiments 39-46 — larger datasets may contain relationship data not present in these fixtures.
- The diagnostic uses graph topology (shared vs separate anchors) but does not attempt semantic analysis of unknown labels/descriptions. Coherence may have additional signals beyond structural sharing.
- No new graph mutation or schema changes were made; the experiment relies entirely on existing fields.
### Conclusion
**A coherence signal exists in the data model.** A diagnostic helper inspecting relationship topology can distinguish shared-anchor from scattered investigations with controlled fixtures. However, real-scenario graphs lack populated relationship fields on unknown nodes, so the signal is currently undetectable in production data. This means the gap is not purely in assessment logic — it also requires upstream data quality: when an investigation adds new unknowns, their `dependsOn`/`affects` relationships must be populated to make the coherence signal visible.
### Status
Pending Rob's review. No production code or graph schema modified.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.shared-anchor.test.js` | 26 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** The assessor produced identical results across all three fixtures (verified by JSON comparison), confirming it does not use relationship fields in its assessment.