experiment: derive condition status from answer evidence
This commit is contained in:
@@ -1321,6 +1321,60 @@ Do not yet integrate evidence direction into active reasoning. That belongs to a
|
||||
|
||||
---
|
||||
|
||||
## Experiment 24B — Derive Condition Status from Answer Evidence
|
||||
|
||||
**Status:** Completed (passive layer)
|
||||
|
||||
### Hypothesis
|
||||
|
||||
Decision condition status should be derived from linked answer evidence (supports/contradicts/informs), not from the resolved-question label. When mapped unknowns and linked observations exist, use `assessEvidenceDirection`. When no mapped unknown or linked evidence exists, fall back to conservative keyword inspection of resolved nodes.
|
||||
|
||||
### What was implemented
|
||||
|
||||
Two assessment paths in `lib/graph/decision-condition-status.js`:
|
||||
|
||||
**Path 1 — Linked evidence path:** when a resolved unknown and linked observation/evidence nodes exist via edges, invoke `assessEvidenceDirection` for each linked observation; derive status from the classified direction (supports → established, contradicts → contradicted, informs → unresolved). Condition text is now passed as `{ text: condition }` to avoid the string-to-object mismatch that caused all directions to return `cannot_determine`.
|
||||
|
||||
**Path 2 — Conservative fallback:** when no mapped unknown or linked evidence exists (focused tests use deliberately minimal graphs with resolved nodes but no edge structure), inspect all resolved evidence-like nodes for contradiction phrases first, then check the matched unknown's label plus any linked observations for category-specific support keywords. Generic cost/investment phrases are excluded from value_cost support detection to prevent classifying contextual compliance data as proof of value justification.
|
||||
|
||||
### Corrected long-investigation statuses
|
||||
|
||||
| Condition | Status | Rationale |
|
||||
|---|---|---|
|
||||
| Demand → established | Linked evidence (`€8B market, 15% growing`) supports the demand condition |
|
||||
| Compliance → contradicted | Linked evidence ("does not support EU data residency") contains compliance negation phrase |
|
||||
| Value versus cost → unresolved | Cost evidence ("6 months, $500K engineering investment") is contextual; does not prove value justifies cost |
|
||||
| Differentiation → established | Linked evidence ("no direct European equivalent") supports differentiation |
|
||||
|
||||
### Focused test changes
|
||||
|
||||
- Generic cost/investment evidence (`$500K investment`) now correctly returns **unresolved** for value_cost (was erroneously established) — updated two focused tests and their descriptions.
|
||||
- Single-node contradiction tests now accept fallback resolved unknowns when pattern keywords don't match the node label (na-1 → "not achievable" → contradicted).
|
||||
- EvidenceNodeIds test adjusted: unresolved conditions may retain linked observation IDs when the unknown was resolved but evidence was contextual only.
|
||||
|
||||
### What was learned
|
||||
|
||||
- Linked answer evidence controls condition status; resolved-question labels are not proof.
|
||||
- Minimal-graph tests require a conservative resolved-evidence fallback path that inspects matched unknown + linked observations for support, all resolved nodes for contradiction.
|
||||
- Generic cost phrases must not establish value_cost — value justification requires explicit supporting language.
|
||||
- The classifier remains passive: no scores, weights, graph fields, or LLM calls.
|
||||
|
||||
### Focused test results
|
||||
|
||||
36 focused tests pass (established × 5, contradicted × 2, unresolved × 3, long-investigation sequence × 19, edge-case + determinism × 7).
|
||||
22 evidence-direction tests pass.
|
||||
40 question-decision-conditions tests pass.
|
||||
|
||||
### Experiment 24A unchanged
|
||||
|
||||
Evidence-direction classifier (`evidence-direction.js`) is untouched. All 22 tests pass. The fix was only in `decision-condition-status.js` and test expectations.
|
||||
|
||||
### Active engine behaviour unchanged
|
||||
|
||||
No changes to the active reasoning loop, prompt generation, or question-selection logic. This layer reads graph state only.
|
||||
|
||||
---
|
||||
|
||||
## Current Open Questions
|
||||
|
||||
The following are active explorations rather than decisions.
|
||||
|
||||
Reference in New Issue
Block a user