212 lines
8.1 KiB
Markdown
212 lines
8.1 KiB
Markdown
# v0.6 Atomicity Experiment
|
|
|
|
## Hypothesis
|
|
|
|
After deterministic unknown selection, the engine should assess whether the selected unknown is already atomic or is still too composite to ask directly.
|
|
|
|
If the unknown is atomic, the engine should proceed exactly as before.
|
|
|
|
If the unknown is composite, the engine should not ask that parent unknown directly. Instead, it should decompose it into a small set of explicit child unknowns representing broad, independent candidate dimensions that a non-expert could understand.
|
|
|
|
## Constraints
|
|
|
|
- No graph redesign
|
|
- No persistence
|
|
- No UI redesign
|
|
- No selection-weight tuning
|
|
- No Ollama calls in unit tests
|
|
|
|
## Deterministic rule introduced
|
|
|
|
Atomicity assessment is **not** a new investigation strategy.
|
|
|
|
It runs in the graph update path at this seam:
|
|
|
|
```text
|
|
unknown selection -> atomicity assessment -> optional decomposition -> deterministic reselection -> question formulation
|
|
```
|
|
|
|
The implementation uses deterministic text and graph-shape checks:
|
|
|
|
- focused unknowns like denominator / threshold / definition / baseline / evidence remain **atomic**
|
|
- broad relationship-explanation unknowns and broad “possible causes / what changed / explanation for why X but Y” unknowns become **composite**
|
|
|
|
## Decomposition behavior
|
|
|
|
When a selected unknown is composite:
|
|
|
|
1. The parent unknown remains unresolved.
|
|
2. Between 2 and 5 child unknowns are created or reused deterministically.
|
|
3. Children become explicit graph nodes.
|
|
4. Children link back to the parent with existing `depends_on` edges.
|
|
5. Children inherit the same “why it matters” discipline in their descriptions.
|
|
6. Deterministic selection reruns across the updated graph.
|
|
|
|
For the current relationship-explanation experiment, the broad child dimensions are:
|
|
|
|
- Whether the two observations reflect different timing
|
|
- How the two observations were measured
|
|
- Change affecting signal A more than signal B
|
|
- Change affecting signal B more than signal A
|
|
- One-off event during the period
|
|
|
|
These are intentionally non-jargon and broad enough to generalise across scenarios like:
|
|
|
|
- Revenue up / Cash down
|
|
- Customer satisfaction up / Complaints up
|
|
- Delivery time down / Cancellations up
|
|
- Traffic up / Sales flat
|
|
- Production up / Defects up
|
|
|
|
## Diagnostics added
|
|
|
|
The orchestrator now reports:
|
|
|
|
- `atomicityAssessment`
|
|
- `atomicityDecisionReason`
|
|
- `decompositionDepth`
|
|
- `decompositionAttempted`
|
|
- `decompositionAccepted`
|
|
- `decompositionStoppedReason`
|
|
- `proposedChildCount`
|
|
- `acceptedChildCount`
|
|
- `rejectedChildren`
|
|
- `selectedChildNodeId`
|
|
- `childQualitySummary`
|
|
- `propagationPerformed`
|
|
- `resolvedChildNodeId`
|
|
- `parentNodeId`
|
|
- `parentStatusBefore`
|
|
- `parentStatusAfter`
|
|
- `parentConfidenceBefore`
|
|
- `parentConfidenceAfter`
|
|
- `affectedAncestorIds`
|
|
- `nextSelectedSibling`
|
|
- `parentResolved`
|
|
- `decompositionPerformed`
|
|
- `childUnknownCount`
|
|
- `childNodeIds`
|
|
- `atomicityReason`
|
|
|
|
This sits alongside the existing explicit-emergent-unknown diagnostics.
|
|
|
|
## Observed outcome
|
|
|
|
The experiment was useful.
|
|
|
|
Before this change, the engine could select a broad explanation unknown and ask it directly.
|
|
|
|
After this change:
|
|
|
|
- the broad explanation parent remains explicit in the graph
|
|
- the engine decomposes it into child unknowns first
|
|
- the next asked question is backed by a more focused child unknown
|
|
- repeated updates reuse the same decomposition children deterministically
|
|
- child-quality checks reject compound or duplicate children before they enter the graph
|
|
- decomposition stops deterministically once a selected child is directly answerable
|
|
- resolving one child does not resolve the parent immediately
|
|
- resolved child evidence now propagates upward to the parent and ancestor chain deterministically
|
|
- parent status and confidence change conservatively after child resolution
|
|
- the next sibling becomes eligible for normal deterministic selection without recreating the resolved child
|
|
|
|
In the revenue-versus-cash case, the selected next question becomes:
|
|
|
|
> What evidence would clarify how the two observations were measured?
|
|
|
|
rather than asking the full broad explanation node directly.
|
|
|
|
## Upward propagation and reconstruction
|
|
|
|
Recursive reasoning is complete only when decomposition and reconstruction are both deterministic.
|
|
|
|
Confidence must not outrun completeness or evidence.
|
|
|
|
For this experiment, reconstruction now behaves as follows:
|
|
|
|
- when a child unknown resolves, that child keeps its own resolved status and answer evidence
|
|
- the parent is updated, but remains unresolved unless the deterministic completion rule is satisfied
|
|
- only the ancestor chain connected to that child is updated
|
|
- unrelated branches remain unchanged
|
|
- the deterministic selector then chooses the next justified unresolved sibling or related follow-up
|
|
|
|
For the current conservative completion rule:
|
|
|
|
- **one resolved child** → parent becomes `provisional` with higher confidence, but remains unresolved
|
|
- **all direct child unknowns resolved** → parent resolves deterministically with `high` confidence
|
|
|
|
The confidence model is now explicitly separated into:
|
|
|
|
- **evidence confidence**: how trustworthy the currently attached support is
|
|
- **completeness**: whether the required direct child structure is empty, partial, or complete
|
|
- **conclusion confidence**: how strongly the current parent state is justified given both evidence and completeness
|
|
|
|
Deterministic propagation rules now enforce:
|
|
|
|
- one resolved child may raise evidence confidence
|
|
- unresolved direct children cap conclusion confidence
|
|
- contradictory direct children block high conclusion confidence
|
|
- duplicate evidence does not increase confidence
|
|
- status changes do not raise confidence on their own
|
|
- parent resolution still requires the separate completion rule
|
|
|
|
## Cross-branch corroboration
|
|
|
|
The next confidence experiment adds deterministic branch interaction checks without changing the graph model.
|
|
|
|
The engine now distinguishes between:
|
|
|
|
- **multiple evidence**: more than one branch exists
|
|
- **independent corroboration**: distinct resolved branches support the same parent without sharing the same evidence key
|
|
- **duplicate evidence**: the same evidence key appears through multiple branches and must not be double-counted
|
|
- **conflicting evidence**: branches support incompatible positions, such as `recognised correctly` vs `recognised incorrectly`
|
|
|
|
Deterministic branch rules:
|
|
|
|
- corroboration only counts when branches are distinct and their evidence sources differ
|
|
- duplicate evidence groups never count as corroboration
|
|
- conflicts cap conclusion confidence and prevent a higher confidence upgrade
|
|
- independent branches remain interaction-neutral
|
|
|
|
Additional diagnostics now expose:
|
|
|
|
- `corroboratingBranchCount`
|
|
- `conflictingBranchCount`
|
|
- `duplicateEvidenceCount`
|
|
- `independentBranchCount`
|
|
- `interactionSummary`
|
|
- `confidenceAdjustmentReason`
|
|
|
|
Observed effect:
|
|
|
|
- independent corroboration can raise `evidenceConfidence`
|
|
- duplicate evidence produces no extra confidence increase
|
|
- conflicting evidence lowers or caps `conclusionConfidence`
|
|
- completeness rules still dominate whether a parent may become highly justified
|
|
|
|
Example progression:
|
|
|
|
- parent before: `unknown`, `medium`
|
|
- after resolving `How the two observations were measured`: parent becomes `provisional`, `medium`
|
|
- evidence confidence becomes `high`, completeness becomes `partial`, conclusion confidence becomes `medium`
|
|
- next sibling becomes selectable and the engine moves on without recreating the resolved child
|
|
|
|
## Interpretation
|
|
|
|
This supports the idea that recursive decomposition is a fundamental part of graph-backed questioning, not just a prompt refinement.
|
|
|
|
The main remaining limitation is that sibling selection still inherits the existing deterministic scorer. That means some domains may advance to a justified sibling that is not the intuitively expected next child, even though the propagation itself remains deterministic and graph-valid.
|
|
|
|
## Validation run
|
|
|
|
Covered by:
|
|
|
|
- `tests/graph/atomicity-assessment.test.js`
|
|
- `tests/graph/decomposition-quality.test.js`
|
|
- `tests/graph/upward-propagation.test.js`
|
|
- `tests/graph/apply-proposal.test.js`
|
|
- `tests/graph/orchestrator.test.js`
|
|
- `tests/graph/question-formulator.test.js`
|
|
- `tests/ui/scenario-form.test.jsx`
|
|
|
|
And then by the broader requested validation pass with lint and build.
|