experiment: derive condition status from answer evidence

This commit is contained in:
2026-08-06 11:17:17 +01:00
parent aabb797e5d
commit da291c715b
3 changed files with 306 additions and 137 deletions
+54
View File
@@ -1321,6 +1321,60 @@ Do not yet integrate evidence direction into active reasoning. That belongs to a
---
## Experiment 24B — Derive Condition Status from Answer Evidence
**Status:** Completed (passive layer)
### Hypothesis
Decision condition status should be derived from linked answer evidence (supports/contradicts/informs), not from the resolved-question label. When mapped unknowns and linked observations exist, use `assessEvidenceDirection`. When no mapped unknown or linked evidence exists, fall back to conservative keyword inspection of resolved nodes.
### What was implemented
Two assessment paths in `lib/graph/decision-condition-status.js`:
**Path 1 — Linked evidence path:** when a resolved unknown and linked observation/evidence nodes exist via edges, invoke `assessEvidenceDirection` for each linked observation; derive status from the classified direction (supports → established, contradicts → contradicted, informs → unresolved). Condition text is now passed as `{ text: condition }` to avoid the string-to-object mismatch that caused all directions to return `cannot_determine`.
**Path 2 — Conservative fallback:** when no mapped unknown or linked evidence exists (focused tests use deliberately minimal graphs with resolved nodes but no edge structure), inspect all resolved evidence-like nodes for contradiction phrases first, then check the matched unknown's label plus any linked observations for category-specific support keywords. Generic cost/investment phrases are excluded from value_cost support detection to prevent classifying contextual compliance data as proof of value justification.
### Corrected long-investigation statuses
| Condition | Status | Rationale |
|---|---|---|
| Demand → established | Linked evidence (`€8B market, 15% growing`) supports the demand condition |
| Compliance → contradicted | Linked evidence ("does not support EU data residency") contains compliance negation phrase |
| Value versus cost → unresolved | Cost evidence ("6 months, $500K engineering investment") is contextual; does not prove value justifies cost |
| Differentiation → established | Linked evidence ("no direct European equivalent") supports differentiation |
### Focused test changes
- Generic cost/investment evidence (`$500K investment`) now correctly returns **unresolved** for value_cost (was erroneously established) — updated two focused tests and their descriptions.
- Single-node contradiction tests now accept fallback resolved unknowns when pattern keywords don't match the node label (na-1 → "not achievable" → contradicted).
- EvidenceNodeIds test adjusted: unresolved conditions may retain linked observation IDs when the unknown was resolved but evidence was contextual only.
### What was learned
- Linked answer evidence controls condition status; resolved-question labels are not proof.
- Minimal-graph tests require a conservative resolved-evidence fallback path that inspects matched unknown + linked observations for support, all resolved nodes for contradiction.
- Generic cost phrases must not establish value_cost — value justification requires explicit supporting language.
- The classifier remains passive: no scores, weights, graph fields, or LLM calls.
### Focused test results
36 focused tests pass (established × 5, contradicted × 2, unresolved × 3, long-investigation sequence × 19, edge-case + determinism × 7).
22 evidence-direction tests pass.
40 question-decision-conditions tests pass.
### Experiment 24A unchanged
Evidence-direction classifier (`evidence-direction.js`) is untouched. All 22 tests pass. The fix was only in `decision-condition-status.js` and test expectations.
### Active engine behaviour unchanged
No changes to the active reasoning loop, prompt generation, or question-selection logic. This layer reads graph state only.
---
## Current Open Questions
The following are active explorations rather than decisions.