confidence-engine/docs/v0.6-comparability-experiment.md

v0.6 Comparability Experiment

Hypothesis

The engine should confirm that observations are comparable before treating their difference as a contradiction that needs explanatory follow-up.

Fixtures

1. Revenue increased by 18%, but cash in the bank fell over the same period.
2. Complaints increased. Production increased.
3. Average delivery time decreased by 25%, but order cancellations increased.
4. Customer satisfaction increased, but complaints increased.
5. Temperature increased. Ice melted.
6. Sales doubled. Sales doubled.

Results

• The first four scenarios repeated the same failure pattern: contradiction-level investigation could begin before comparability was established.
• A deterministic comparability gate corrected that by producing one comparison question first.
• Confirmed comparability did not by itself imply contradiction.
• Temperature increased / Ice melted was reclassified as a compatible relationship, so no contradiction question was asked.
• Sales doubled / Sales doubled was reclassified as duplicate observations, so no follow-up question was asked.

Relationship classification stage

After comparability assessment, observations now pass through a deterministic relationship classification stage:

• contradictory
• compatible
• potentially_related
• duplicate
• insufficient_information

Whether comparability should become a permanent reasoning stage

Yes, in minimal deterministic form.

The repeated pattern appeared in four scenarios, so a small pre-contradiction comparability assessment is justified.

Two-step experiment result

A comparison question is useful only if its answer advances the reasoning stage rather than merely adding more text.

In the revenue-versus-cash scenario, the first question now confirms whether the figures are comparable, and the answer resolves that existing uncertainty instead of creating a parallel note. After that update, the engine progresses from comparability assessment to cautious relationship assessment and can select one broad non-expert follow-up question.

Every justified next question should correspond to an explicit unresolved graph node.

The earlier fallback-only path has now been removed from the normal successful progression. After comparability is resolved and a further investigation question is justified, the engine creates or reuses an explicit unresolved reasoning unknown and lets deterministic selection and question formulation proceed through the standard graph pipeline. A fallback is now only acceptable as an explicit failure case, not as the normal source of the next question.