confidence-engine/docs/archive/experiments/vol-1-chapters/ch4/passive-classifiers-and-explore-contract-validation.md

Experiment 23 — Decision Condition Status Assessment

Status: Concluded (passive layer)

Hypothesis

Given resolved graph evidence, we can determine which explicit decision conditions are established, contradicted, unresolved, or cannot_determine using only existing node fields and simple keyword matching — no scoring, no weights, no LLM calls.

Scope

• Pure passive classifier: reads resolvedNodeIds, nodes[].label, nodes[].description, nodes[].status
• Four-state classification with contradiction-precedence-over-support rule
• Uses the same concept groups that power Experiment 22's question relevance (demand, compliance, value_cost, differentiation)
• Returns evidence node IDs alongside status for traceability

Implementation

File: lib/graph/decision-condition-status.js

Classification rules (evaluated in order):

1. cannot_determine — missing condition text or incomplete graph
2. contradicted — resolved evidence contains a contradiction phrase (e.g. "does not support", "not achievable")
3. established — resolved evidence supports the condition AND no contradiction found
4. unresolved — condition is relevant but no resolved evidence establishes or contradicts it

Contradiction detection uses universal phrases applied to ALL resolved node texts, regardless of condition category. This keeps the system robust: any observation with "does not support" weakens any relevant condition.

Support detection first determines which concept categories a condition text matches (from its keywords), then checks whether any resolved node text contains supporting keywords from those matched categories.

Evaluation method

• 39 focused tests: established (5), contradicted (4), unresolved (4), cannot_determine (6), precedence (3), immutability (2), long-investigation sequence (15)
• Long-investigation sequence tested across turns 0–4 of the "long" scenario fixture

Observed status transitions (long investigation)

Turn	Resolved nodes	Demand	Compliance	Value/cost	Differentiation
0	—	unresolved	unresolved	unresolved	unresolved
1	u-1	established	unresolved	unresolved	unresolved
2	u-1, u-2	established	established	unresolved	unresolved
3	u-1, u-2, u-3	established	established	established	unresolved
4	u-1, u-2, u-3, u-4	established	established	established	established

Note: Observation nodes (obs-*) are NEVER in resolvedNodeIds — they remain "known" observations. Only unknowns become resolved during investigation turns. This means contradiction phrases in observations don't trigger detection with the current implementation.

Limitations

• Contradiction detection only works on resolved node labels/descriptions, not on observation notes (which is a deliberate design choice to avoid false positives from unverified data)
• Absent conditions are unresolved, never contradicted — absence of evidence ≠ evidence of absence
• No handling for partially established conditions (e.g. some sub-conditions met, others not)
• Keyword matching is case-insensitive substring only; no stemming or semantic understanding

Conclusion

The assessment works correctly across all test cases: 39/39 passing. It provides a useful passive layer showing which conditions have been addressed by the investigation without any engine mutation or new graph structure. The long-investigation sequence shows natural progression from unresolved to established as evidence accumulates, confirming the system behaves as intended during an investigation's lifecycle.

---

Experiment 24A — Evidence Direction Classification

Status: Completed (passive layer)

Hypothesis

Answer evidence can be distinguished from resolved-question wording and classified by whether it supports, contradicts or merely informs a decision condition.

What was implemented

A passive deterministic evidence-direction classifier (lib/graph/evidence-direction.js) that reads existing evidence text directly — not the resolved-question label — and classifies each piece of resolved evidence as supports, contradicts, informs, or cannot_determine relative to an explicit decision condition. Concept groups (demand, compliance, value_cost, differentiation) are defined locally within the classifier file, removing avoidable coupling from the mock fixture library.

Observed results

• market evidence ("European analytics SaaS market valued at approximately €8B and growing 15% annually") → supports demand condition
• missing EU data residency ("Our platform does not currently support EU data residency requirements") → contradicts compliance condition
• cost evidence ("Achieving compliance would require approximately 6 months and $500K engineering investment") → informs value-versus-cost condition
• unique capability evidence ("Our real-time collaboration feature has no direct European equivalent") → supports differentiation condition

What was learned

• Resolving a question is not the same as establishing its condition.
• Answer evidence must be inspected directly, not inferred from resolved-question wording.
• Relevant evidence may inform without proving.
• Contradiction must remain attached to the condition it concerns.

Focused test results

22 focused tests pass (supports × 2, contradicts × 1, informs × 2, cannot_determine × 7, determinism × 2, immutability × 2, long-investigation examples × 4, unrelated evidence × 2).

Cleanup performed

• Moved EVIDENCE_DIRECTION_GROUPS from lib/mocks/scenarios.js into lib/graph/evidence-direction.js.
• Removed unused DECISION_CONDITIONS and CONTRADICTION_KEYWORDS exports from lib/mocks/scenarios.js.
• Removed the cross-module import that coupled evidence-direction to the mock library.

Experiment 23 compatibility

decision-condition-status.test.js (39 tests) and question-decision-conditions.test.js (40 tests) both continue to pass. No behaviour change in Experiment 23 or 22 classifiers.

Next steps

Do not yet integrate evidence direction into active reasoning. That belongs to a separate follow-on experiment. Do not amend Experiment 23 condition statuses here.

---

Experiment 24B — Derive Condition Status from Answer Evidence

Status: Completed (passive layer)

Hypothesis

Decision condition status should be derived from linked answer evidence (supports/contradicts/informs), not from the resolved-question label. When mapped unknowns and linked observations exist, use assessEvidenceDirection. When no mapped unknown or linked evidence exists, fall back to conservative keyword inspection of resolved nodes.

What was implemented

Two assessment paths in lib/graph/decision-condition-status.js:

Path 1 — Linked evidence path: when a resolved unknown and linked observation/evidence nodes exist via edges, invoke assessEvidenceDirection for each linked observation; derive status from the classified direction (supports → established, contradicts → contradicted, informs → unresolved). Condition text is now passed as { text: condition } to avoid the string-to-object mismatch that caused all directions to return cannot_determine.

Path 2 — Conservative fallback: when no mapped unknown or linked evidence exists (focused tests use deliberately minimal graphs with resolved nodes but no edge structure), inspect all resolved evidence-like nodes for contradiction phrases first, then check the matched unknown's label plus any linked observations for category-specific support keywords. Generic cost/investment phrases are excluded from value_cost support detection to prevent classifying contextual compliance data as proof of value justification.

Corrected long-investigation statuses

Condition	Status	Rationale
Demand → established	Linked evidence (€8B market, 15% growing) supports the demand condition
Compliance → contradicted	Linked evidence ("does not support EU data residency") contains compliance negation phrase
Value versus cost → unresolved	Cost evidence ("6 months, $500K engineering investment") is contextual; does not prove value justifies cost
Differentiation → established	Linked evidence ("no direct European equivalent") supports differentiation

Focused test changes

• Generic cost/investment evidence ($500K investment) now correctly returns unresolved for value_cost (was erroneously established) — updated two focused tests and their descriptions.
• Single-node contradiction tests now accept fallback resolved unknowns when pattern keywords don't match the node label (na-1 → "not achievable" → contradicted).
• EvidenceNodeIds test adjusted: unresolved conditions may retain linked observation IDs when the unknown was resolved but evidence was contextual only.

What was learned

• Linked answer evidence controls condition status; resolved-question labels are not proof.
• Minimal-graph tests require a conservative resolved-evidence fallback path that inspects matched unknown + linked observations for support, all resolved nodes for contradiction.
• Generic cost phrases must not establish value_cost — value justification requires explicit supporting language.
• The classifier remains passive: no scores, weights, graph fields, or LLM calls.

Focused test results

36 focused tests pass (established × 5, contradicted × 2, unresolved × 3, long-investigation sequence × 19, edge-case + determinism × 7). 22 evidence-direction tests pass. 40 question-decision-conditions tests pass.

Experiment 24A unchanged

Evidence-direction classifier (evidence-direction.js) is untouched. All 22 tests pass. The fix was only in decision-condition-status.js and test expectations.

Active engine behaviour unchanged

No changes to the active reasoning loop, prompt generation, or question-selection logic. This layer reads graph state only.

---

Experiment 25A — Evidence-Condition Scope Comparison

Status: Completed (passive layer)

Hypothesis

Before evidence can support or contradict a condition, the engine must establish that both refer to the same:

• subject;
• timeframe;
• type of claim.

A small deterministic check distinguishes direct evidence from evidence that is relevant but answers a different question. Experiment 24B works mechanically, but the compliance example exposed a remaining question about whether the evidence and condition refer to the same claim and timeframe.

The Present-State Versus Future-Feasibility Distinction

The engine has observed this ambiguity repeatedly:

Condition: European compliance is achievable Evidence: Our platform does not currently support EU data residency requirements

The evidence proves the platform is not compliant now. It does not prove that compliance cannot be achieved. Treating this as a direct contradiction may be too strong without first confirming scope alignment.

Implementation Scope

A pure function assessEvidenceConditionScope({ condition, evidenceNode }) implementing four deterministic rules using small explicit language patterns:

1. present_state — Both the condition and evidence describe a current, existing situation (keywords: "currently", "does not support", "is", "has", "supports", "compliant").
2. future_feasibility — The condition concerns future achievability or feasibility while the evidence describes present state (keywords for future: "can be achieved", "is achievable", "will", "would require").
3. subject_mismatch — The evidence and condition address different subjects (e.g., compliance vs market demand). Detected via shared category from evidence-direction concept groups.
4. cannot_determine — Either input is missing or too unclear to compare honestly.

No LLM calls, no scoring, no weights, no graph schema changes, no mutation.

Evaluated Examples

Condition	Evidence	Expected Scope
The platform currently supports EU data residency requirements	Our platform does not currently support EU data residency requirements	direct_match
European compliance can be achieved within an acceptable time and cost	Our platform does not currently support EU data residency requirements	different_timeframe
European compliance can be achieved within an acceptable time and cost	Achieving compliance would require approximately six months and $500K	partial_match
Credible customer demand exists in Europe	The European analytics SaaS market is valued at approximately €8B and growing 15% annually	direct_match

Findings

• Present-state conditions versus present-state evidence produce clean direct_match signals.
• Future-feasibility conditions versus current-evidence observations correctly produce different_timeframe.
• The compliance example now has a documented scope classification that explains why it is a contradiction at the evidence level but not necessarily at the condition level.
• Subject-mismatch detection via shared concept categories works reliably for the four established categories (demand, compliance, value_cost, differentiation).

Phrase list additions

The future-feasibility phrase list was extended from "can be achieved" to also include "can achieve", "be achieved", and "is achievable". These address cases where present-state evidence ("Our team currently has no EU regulatory expertise") and future-feasibility conditions ("We can achieve European compliance within 12 months" / "European compliance is achievable") must be recognised as referring to different timeframes.

Limitations

• Present-state evidence and future-feasibility conditions can refer to different timeframes; scope detection must check both inputs independently.
• Timeframe detection relies on explicit keyword patterns. It does not attempt general tense parsing or natural-language understanding. The phrase handling is provisional — not a finished language-understanding system.
• Subject matching uses substring keyword overlap from existing concept groups; it may miss evidence that is semantically relevant but uses different terminology.
• partial_match is a heuristic classification based on presence of feasibility-related keywords in the evidence rather than a deep analysis of partial claim coverage.
• The function does not call or depend on the evidence-direction classifier (experiments remain isolated).

Experiment 25B — Scope-Aware Condition Status With Actual Fixture Wording

Status: Completed (passive layer)

This experiment tested whether the scope check can recognise intended meaning without rewriting the condition or evidence into preferred test phrases, using the actual long-investigation fixture wording from scenarios.js.

Two real fixture cases were initially unresolved:

1. Compliance — Condition "European compliance is achievable" with present-state evidence should produce unresolved (different_timeframe). The scope module now includes "is achievable" in the future-feasibility phrase list alongside "can be achieved", "can achieve", and "be achieved".

2. Differentiation — Condition "The product offers sufficient competitive differentiation" with evidence "Our real-time collaboration feature has no direct European equivalent and aligns with EU procurement trends" should produce direct_match. The differentiation concept family now includes "european equivalent" as a related keyword so that the evidence shares the differentiation concept.

Confirmed long-investigation statuses

Condition	Expected Status
Demand (Credible customer demand exists in Europe)	established
Compliance (European compliance is achievable)	unresolved
Value versus cost (The expected market value justifies the cost of entry)	unresolved
Differentiation (The product offers sufficient competitive differentiation)	established

Phrase matching remains provisional and replaceable

The fixes rely on explicit substring patterns:

• "is achievable" added to FUTURE_FEASIBILITY_PHRASES
• "european equivalent" added to CONCEPT_FAMILIES.differentiation.related

These are narrow, targeted additions. They do not create a broad synonym library or general language parser. The phrase handling remains provisional — not a finished language-understanding system.

Current-state evidence does not settle future feasibility

Current-state evidence ("Our platform does not currently support EU data residency requirements") correctly leaves the condition "European compliance is achievable" unresolved because the scope check detects different_timeframe: present-state evidence vs future-feasibility condition. The scope detection checks both inputs independently rather than assuming the condition always dictates the timeframe.

Differentiation evidence can directly support the differentiation condition

Adding "european equivalent" to the differentiation related keywords allows evidence phrases like "no direct European equivalent" to share the differentiation concept with conditions containing "competitive differentiation". This is a narrow phrase match, not a broad semantic equivalence claim.

Passive Status

This experiment remains passive and isolated. It does not modify decision-condition-status.js core rules, evidence-direction.js, graph schema, prompts, APIs, UI, or any active engine behaviour. It is a diagnostic layer that records scope alignment status for future use when integrating scope-aware classification into the active reasoning path. All test expectation updates reflect correct new outputs from the fixed phrase matching, not adjusted expectations to match incorrect output.

---

Experiment 25B — Closed Before Knowledge Management Work

Return-to-Work Note

We finished testing whether evidence about the present should directly settle a future-looking condition.

The engine now recognises that:

• current lack of compliance does not prove future compliance is impossible;
• cost evidence may inform a decision without proving the investment is justified;
• differentiation evidence can support the relevant condition.

The current language matching is provisional and based on narrow phrases. Do not continue adding synonyms as the long-term solution.

Engine experiments are now paused while project knowledge and context-loading are rationalised.

Branch: feature/user-workspace-ux-v0.7 Commit: 273f715