docs: archive historical Confidence Engine evidence
This commit is contained in:
@@ -0,0 +1,211 @@
|
||||
# v0.6 Atomicity Experiment
|
||||
|
||||
## Hypothesis
|
||||
|
||||
After deterministic unknown selection, the engine should assess whether the selected unknown is already atomic or is still too composite to ask directly.
|
||||
|
||||
If the unknown is atomic, the engine should proceed exactly as before.
|
||||
|
||||
If the unknown is composite, the engine should not ask that parent unknown directly. Instead, it should decompose it into a small set of explicit child unknowns representing broad, independent candidate dimensions that a non-expert could understand.
|
||||
|
||||
## Constraints
|
||||
|
||||
- No graph redesign
|
||||
- No persistence
|
||||
- No UI redesign
|
||||
- No selection-weight tuning
|
||||
- No Ollama calls in unit tests
|
||||
|
||||
## Deterministic rule introduced
|
||||
|
||||
Atomicity assessment is **not** a new investigation strategy.
|
||||
|
||||
It runs in the graph update path at this seam:
|
||||
|
||||
```text
|
||||
unknown selection -> atomicity assessment -> optional decomposition -> deterministic reselection -> question formulation
|
||||
```
|
||||
|
||||
The implementation uses deterministic text and graph-shape checks:
|
||||
|
||||
- focused unknowns like denominator / threshold / definition / baseline / evidence remain **atomic**
|
||||
- broad relationship-explanation unknowns and broad “possible causes / what changed / explanation for why X but Y” unknowns become **composite**
|
||||
|
||||
## Decomposition behavior
|
||||
|
||||
When a selected unknown is composite:
|
||||
|
||||
1. The parent unknown remains unresolved.
|
||||
2. Between 2 and 5 child unknowns are created or reused deterministically.
|
||||
3. Children become explicit graph nodes.
|
||||
4. Children link back to the parent with existing `depends_on` edges.
|
||||
5. Children inherit the same “why it matters” discipline in their descriptions.
|
||||
6. Deterministic selection reruns across the updated graph.
|
||||
|
||||
For the current relationship-explanation experiment, the broad child dimensions are:
|
||||
|
||||
- Whether the two observations reflect different timing
|
||||
- How the two observations were measured
|
||||
- Change affecting signal A more than signal B
|
||||
- Change affecting signal B more than signal A
|
||||
- One-off event during the period
|
||||
|
||||
These are intentionally non-jargon and broad enough to generalise across scenarios like:
|
||||
|
||||
- Revenue up / Cash down
|
||||
- Customer satisfaction up / Complaints up
|
||||
- Delivery time down / Cancellations up
|
||||
- Traffic up / Sales flat
|
||||
- Production up / Defects up
|
||||
|
||||
## Diagnostics added
|
||||
|
||||
The orchestrator now reports:
|
||||
|
||||
- `atomicityAssessment`
|
||||
- `atomicityDecisionReason`
|
||||
- `decompositionDepth`
|
||||
- `decompositionAttempted`
|
||||
- `decompositionAccepted`
|
||||
- `decompositionStoppedReason`
|
||||
- `proposedChildCount`
|
||||
- `acceptedChildCount`
|
||||
- `rejectedChildren`
|
||||
- `selectedChildNodeId`
|
||||
- `childQualitySummary`
|
||||
- `propagationPerformed`
|
||||
- `resolvedChildNodeId`
|
||||
- `parentNodeId`
|
||||
- `parentStatusBefore`
|
||||
- `parentStatusAfter`
|
||||
- `parentConfidenceBefore`
|
||||
- `parentConfidenceAfter`
|
||||
- `affectedAncestorIds`
|
||||
- `nextSelectedSibling`
|
||||
- `parentResolved`
|
||||
- `decompositionPerformed`
|
||||
- `childUnknownCount`
|
||||
- `childNodeIds`
|
||||
- `atomicityReason`
|
||||
|
||||
This sits alongside the existing explicit-emergent-unknown diagnostics.
|
||||
|
||||
## Observed outcome
|
||||
|
||||
The experiment was useful.
|
||||
|
||||
Before this change, the engine could select a broad explanation unknown and ask it directly.
|
||||
|
||||
After this change:
|
||||
|
||||
- the broad explanation parent remains explicit in the graph
|
||||
- the engine decomposes it into child unknowns first
|
||||
- the next asked question is backed by a more focused child unknown
|
||||
- repeated updates reuse the same decomposition children deterministically
|
||||
- child-quality checks reject compound or duplicate children before they enter the graph
|
||||
- decomposition stops deterministically once a selected child is directly answerable
|
||||
- resolving one child does not resolve the parent immediately
|
||||
- resolved child evidence now propagates upward to the parent and ancestor chain deterministically
|
||||
- parent status and confidence change conservatively after child resolution
|
||||
- the next sibling becomes eligible for normal deterministic selection without recreating the resolved child
|
||||
|
||||
In the revenue-versus-cash case, the selected next question becomes:
|
||||
|
||||
> What evidence would clarify how the two observations were measured?
|
||||
|
||||
rather than asking the full broad explanation node directly.
|
||||
|
||||
## Upward propagation and reconstruction
|
||||
|
||||
Recursive reasoning is complete only when decomposition and reconstruction are both deterministic.
|
||||
|
||||
Confidence must not outrun completeness or evidence.
|
||||
|
||||
For this experiment, reconstruction now behaves as follows:
|
||||
|
||||
- when a child unknown resolves, that child keeps its own resolved status and answer evidence
|
||||
- the parent is updated, but remains unresolved unless the deterministic completion rule is satisfied
|
||||
- only the ancestor chain connected to that child is updated
|
||||
- unrelated branches remain unchanged
|
||||
- the deterministic selector then chooses the next justified unresolved sibling or related follow-up
|
||||
|
||||
For the current conservative completion rule:
|
||||
|
||||
- **one resolved child** → parent becomes `provisional` with higher confidence, but remains unresolved
|
||||
- **all direct child unknowns resolved** → parent resolves deterministically with `high` confidence
|
||||
|
||||
The confidence model is now explicitly separated into:
|
||||
|
||||
- **evidence confidence**: how trustworthy the currently attached support is
|
||||
- **completeness**: whether the required direct child structure is empty, partial, or complete
|
||||
- **conclusion confidence**: how strongly the current parent state is justified given both evidence and completeness
|
||||
|
||||
Deterministic propagation rules now enforce:
|
||||
|
||||
- one resolved child may raise evidence confidence
|
||||
- unresolved direct children cap conclusion confidence
|
||||
- contradictory direct children block high conclusion confidence
|
||||
- duplicate evidence does not increase confidence
|
||||
- status changes do not raise confidence on their own
|
||||
- parent resolution still requires the separate completion rule
|
||||
|
||||
## Cross-branch corroboration
|
||||
|
||||
The next confidence experiment adds deterministic branch interaction checks without changing the graph model.
|
||||
|
||||
The engine now distinguishes between:
|
||||
|
||||
- **multiple evidence**: more than one branch exists
|
||||
- **independent corroboration**: distinct resolved branches support the same parent without sharing the same evidence key
|
||||
- **duplicate evidence**: the same evidence key appears through multiple branches and must not be double-counted
|
||||
- **conflicting evidence**: branches support incompatible positions, such as `recognised correctly` vs `recognised incorrectly`
|
||||
|
||||
Deterministic branch rules:
|
||||
|
||||
- corroboration only counts when branches are distinct and their evidence sources differ
|
||||
- duplicate evidence groups never count as corroboration
|
||||
- conflicts cap conclusion confidence and prevent a higher confidence upgrade
|
||||
- independent branches remain interaction-neutral
|
||||
|
||||
Additional diagnostics now expose:
|
||||
|
||||
- `corroboratingBranchCount`
|
||||
- `conflictingBranchCount`
|
||||
- `duplicateEvidenceCount`
|
||||
- `independentBranchCount`
|
||||
- `interactionSummary`
|
||||
- `confidenceAdjustmentReason`
|
||||
|
||||
Observed effect:
|
||||
|
||||
- independent corroboration can raise `evidenceConfidence`
|
||||
- duplicate evidence produces no extra confidence increase
|
||||
- conflicting evidence lowers or caps `conclusionConfidence`
|
||||
- completeness rules still dominate whether a parent may become highly justified
|
||||
|
||||
Example progression:
|
||||
|
||||
- parent before: `unknown`, `medium`
|
||||
- after resolving `How the two observations were measured`: parent becomes `provisional`, `medium`
|
||||
- evidence confidence becomes `high`, completeness becomes `partial`, conclusion confidence becomes `medium`
|
||||
- next sibling becomes selectable and the engine moves on without recreating the resolved child
|
||||
|
||||
## Interpretation
|
||||
|
||||
This supports the idea that recursive decomposition is a fundamental part of graph-backed questioning, not just a prompt refinement.
|
||||
|
||||
The main remaining limitation is that sibling selection still inherits the existing deterministic scorer. That means some domains may advance to a justified sibling that is not the intuitively expected next child, even though the propagation itself remains deterministic and graph-valid.
|
||||
|
||||
## Validation run
|
||||
|
||||
Covered by:
|
||||
|
||||
- `tests/graph/atomicity-assessment.test.js`
|
||||
- `tests/graph/decomposition-quality.test.js`
|
||||
- `tests/graph/upward-propagation.test.js`
|
||||
- `tests/graph/apply-proposal.test.js`
|
||||
- `tests/graph/orchestrator.test.js`
|
||||
- `tests/graph/question-formulator.test.js`
|
||||
- `tests/ui/scenario-form.test.jsx`
|
||||
|
||||
And then by the broader requested validation pass with lint and build.
|
||||
Reference in New Issue
Block a user