docs: archive historical Confidence Engine evidence
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
# v0.5 Question Priority Generalisation
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The current deterministic unknown selector and graph-context question formulator should generalise across several decision types by selecting a foundational unknown before downstream implementation or pricing leaves.
|
||||
|
||||
## Scenarios
|
||||
|
||||
1. Should we hire another engineer?
|
||||
2. Should we replace the delivery vans?
|
||||
3. Should we launch in another country?
|
||||
4. Should we continue a project that is over budget?
|
||||
5. Should we introduce a paid support tier?
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Selected unknown | Strategy | Pass/Fail |
|
||||
| ---------------------------- | --------------------------- | -------------------- | --------- |
|
||||
| Hire another engineer | `hire-success-criteria` | `decision criterion` | Pass |
|
||||
| Replace the delivery vans | `van-reliability-threshold` | `decision criterion` | Pass |
|
||||
| Launch in another country | `country-value-threshold` | `actor/customer` | Pass |
|
||||
| Continue over-budget project | `project-benefit-threshold` | `decision criterion` | Pass |
|
||||
| Introduce paid support tier | `support-value-threshold` | `actor/customer` | Pass |
|
||||
|
||||
## Repeated failure patterns
|
||||
|
||||
Two repeated structural formulation failures appeared before the final pass:
|
||||
|
||||
1. **Constraint language in surrounding graph context outranked node-local decision-threshold language** in more than one case.
|
||||
2. **Baseline language in surrounding graph context outranked node-local threshold language** in more than one case.
|
||||
|
||||
Both failures affected formulation strategy, not deterministic unknown selection.
|
||||
|
||||
## Code change made
|
||||
|
||||
A small deterministic change was made in `lib/graph/question-formulator.js`:
|
||||
|
||||
- prefer node-local `definition` language before broader criterion inference
|
||||
- prefer node-local `decision criterion` language before context-only `constraint` inference
|
||||
- only treat `baseline` or `constraint` as primary when the selected node itself carries that language, otherwise allow them as fallback strategies later
|
||||
|
||||
No architecture, UI, persistence, prompt, scoring, additional model turns, or provider calls were added.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
- In two passing cases, the selector chose a threshold-style foundational node while the formulator still used an `actor/customer` strategy because related context strongly referenced customers or recipients.
|
||||
- This experiment is fixture-driven and deterministic; it is useful for regression protection, not scientific validation.
|
||||
- The suite exercises the production path without model calls, but it does not prove behaviour over arbitrary real-world graph structures.
|
||||
@@ -0,0 +1,211 @@
|
||||
# v0.6 Atomicity Experiment
|
||||
|
||||
## Hypothesis
|
||||
|
||||
After deterministic unknown selection, the engine should assess whether the selected unknown is already atomic or is still too composite to ask directly.
|
||||
|
||||
If the unknown is atomic, the engine should proceed exactly as before.
|
||||
|
||||
If the unknown is composite, the engine should not ask that parent unknown directly. Instead, it should decompose it into a small set of explicit child unknowns representing broad, independent candidate dimensions that a non-expert could understand.
|
||||
|
||||
## Constraints
|
||||
|
||||
- No graph redesign
|
||||
- No persistence
|
||||
- No UI redesign
|
||||
- No selection-weight tuning
|
||||
- No Ollama calls in unit tests
|
||||
|
||||
## Deterministic rule introduced
|
||||
|
||||
Atomicity assessment is **not** a new investigation strategy.
|
||||
|
||||
It runs in the graph update path at this seam:
|
||||
|
||||
```text
|
||||
unknown selection -> atomicity assessment -> optional decomposition -> deterministic reselection -> question formulation
|
||||
```
|
||||
|
||||
The implementation uses deterministic text and graph-shape checks:
|
||||
|
||||
- focused unknowns like denominator / threshold / definition / baseline / evidence remain **atomic**
|
||||
- broad relationship-explanation unknowns and broad “possible causes / what changed / explanation for why X but Y” unknowns become **composite**
|
||||
|
||||
## Decomposition behavior
|
||||
|
||||
When a selected unknown is composite:
|
||||
|
||||
1. The parent unknown remains unresolved.
|
||||
2. Between 2 and 5 child unknowns are created or reused deterministically.
|
||||
3. Children become explicit graph nodes.
|
||||
4. Children link back to the parent with existing `depends_on` edges.
|
||||
5. Children inherit the same “why it matters” discipline in their descriptions.
|
||||
6. Deterministic selection reruns across the updated graph.
|
||||
|
||||
For the current relationship-explanation experiment, the broad child dimensions are:
|
||||
|
||||
- Whether the two observations reflect different timing
|
||||
- How the two observations were measured
|
||||
- Change affecting signal A more than signal B
|
||||
- Change affecting signal B more than signal A
|
||||
- One-off event during the period
|
||||
|
||||
These are intentionally non-jargon and broad enough to generalise across scenarios like:
|
||||
|
||||
- Revenue up / Cash down
|
||||
- Customer satisfaction up / Complaints up
|
||||
- Delivery time down / Cancellations up
|
||||
- Traffic up / Sales flat
|
||||
- Production up / Defects up
|
||||
|
||||
## Diagnostics added
|
||||
|
||||
The orchestrator now reports:
|
||||
|
||||
- `atomicityAssessment`
|
||||
- `atomicityDecisionReason`
|
||||
- `decompositionDepth`
|
||||
- `decompositionAttempted`
|
||||
- `decompositionAccepted`
|
||||
- `decompositionStoppedReason`
|
||||
- `proposedChildCount`
|
||||
- `acceptedChildCount`
|
||||
- `rejectedChildren`
|
||||
- `selectedChildNodeId`
|
||||
- `childQualitySummary`
|
||||
- `propagationPerformed`
|
||||
- `resolvedChildNodeId`
|
||||
- `parentNodeId`
|
||||
- `parentStatusBefore`
|
||||
- `parentStatusAfter`
|
||||
- `parentConfidenceBefore`
|
||||
- `parentConfidenceAfter`
|
||||
- `affectedAncestorIds`
|
||||
- `nextSelectedSibling`
|
||||
- `parentResolved`
|
||||
- `decompositionPerformed`
|
||||
- `childUnknownCount`
|
||||
- `childNodeIds`
|
||||
- `atomicityReason`
|
||||
|
||||
This sits alongside the existing explicit-emergent-unknown diagnostics.
|
||||
|
||||
## Observed outcome
|
||||
|
||||
The experiment was useful.
|
||||
|
||||
Before this change, the engine could select a broad explanation unknown and ask it directly.
|
||||
|
||||
After this change:
|
||||
|
||||
- the broad explanation parent remains explicit in the graph
|
||||
- the engine decomposes it into child unknowns first
|
||||
- the next asked question is backed by a more focused child unknown
|
||||
- repeated updates reuse the same decomposition children deterministically
|
||||
- child-quality checks reject compound or duplicate children before they enter the graph
|
||||
- decomposition stops deterministically once a selected child is directly answerable
|
||||
- resolving one child does not resolve the parent immediately
|
||||
- resolved child evidence now propagates upward to the parent and ancestor chain deterministically
|
||||
- parent status and confidence change conservatively after child resolution
|
||||
- the next sibling becomes eligible for normal deterministic selection without recreating the resolved child
|
||||
|
||||
In the revenue-versus-cash case, the selected next question becomes:
|
||||
|
||||
> What evidence would clarify how the two observations were measured?
|
||||
|
||||
rather than asking the full broad explanation node directly.
|
||||
|
||||
## Upward propagation and reconstruction
|
||||
|
||||
Recursive reasoning is complete only when decomposition and reconstruction are both deterministic.
|
||||
|
||||
Confidence must not outrun completeness or evidence.
|
||||
|
||||
For this experiment, reconstruction now behaves as follows:
|
||||
|
||||
- when a child unknown resolves, that child keeps its own resolved status and answer evidence
|
||||
- the parent is updated, but remains unresolved unless the deterministic completion rule is satisfied
|
||||
- only the ancestor chain connected to that child is updated
|
||||
- unrelated branches remain unchanged
|
||||
- the deterministic selector then chooses the next justified unresolved sibling or related follow-up
|
||||
|
||||
For the current conservative completion rule:
|
||||
|
||||
- **one resolved child** → parent becomes `provisional` with higher confidence, but remains unresolved
|
||||
- **all direct child unknowns resolved** → parent resolves deterministically with `high` confidence
|
||||
|
||||
The confidence model is now explicitly separated into:
|
||||
|
||||
- **evidence confidence**: how trustworthy the currently attached support is
|
||||
- **completeness**: whether the required direct child structure is empty, partial, or complete
|
||||
- **conclusion confidence**: how strongly the current parent state is justified given both evidence and completeness
|
||||
|
||||
Deterministic propagation rules now enforce:
|
||||
|
||||
- one resolved child may raise evidence confidence
|
||||
- unresolved direct children cap conclusion confidence
|
||||
- contradictory direct children block high conclusion confidence
|
||||
- duplicate evidence does not increase confidence
|
||||
- status changes do not raise confidence on their own
|
||||
- parent resolution still requires the separate completion rule
|
||||
|
||||
## Cross-branch corroboration
|
||||
|
||||
The next confidence experiment adds deterministic branch interaction checks without changing the graph model.
|
||||
|
||||
The engine now distinguishes between:
|
||||
|
||||
- **multiple evidence**: more than one branch exists
|
||||
- **independent corroboration**: distinct resolved branches support the same parent without sharing the same evidence key
|
||||
- **duplicate evidence**: the same evidence key appears through multiple branches and must not be double-counted
|
||||
- **conflicting evidence**: branches support incompatible positions, such as `recognised correctly` vs `recognised incorrectly`
|
||||
|
||||
Deterministic branch rules:
|
||||
|
||||
- corroboration only counts when branches are distinct and their evidence sources differ
|
||||
- duplicate evidence groups never count as corroboration
|
||||
- conflicts cap conclusion confidence and prevent a higher confidence upgrade
|
||||
- independent branches remain interaction-neutral
|
||||
|
||||
Additional diagnostics now expose:
|
||||
|
||||
- `corroboratingBranchCount`
|
||||
- `conflictingBranchCount`
|
||||
- `duplicateEvidenceCount`
|
||||
- `independentBranchCount`
|
||||
- `interactionSummary`
|
||||
- `confidenceAdjustmentReason`
|
||||
|
||||
Observed effect:
|
||||
|
||||
- independent corroboration can raise `evidenceConfidence`
|
||||
- duplicate evidence produces no extra confidence increase
|
||||
- conflicting evidence lowers or caps `conclusionConfidence`
|
||||
- completeness rules still dominate whether a parent may become highly justified
|
||||
|
||||
Example progression:
|
||||
|
||||
- parent before: `unknown`, `medium`
|
||||
- after resolving `How the two observations were measured`: parent becomes `provisional`, `medium`
|
||||
- evidence confidence becomes `high`, completeness becomes `partial`, conclusion confidence becomes `medium`
|
||||
- next sibling becomes selectable and the engine moves on without recreating the resolved child
|
||||
|
||||
## Interpretation
|
||||
|
||||
This supports the idea that recursive decomposition is a fundamental part of graph-backed questioning, not just a prompt refinement.
|
||||
|
||||
The main remaining limitation is that sibling selection still inherits the existing deterministic scorer. That means some domains may advance to a justified sibling that is not the intuitively expected next child, even though the propagation itself remains deterministic and graph-valid.
|
||||
|
||||
## Validation run
|
||||
|
||||
Covered by:
|
||||
|
||||
- `tests/graph/atomicity-assessment.test.js`
|
||||
- `tests/graph/decomposition-quality.test.js`
|
||||
- `tests/graph/upward-propagation.test.js`
|
||||
- `tests/graph/apply-proposal.test.js`
|
||||
- `tests/graph/orchestrator.test.js`
|
||||
- `tests/graph/question-formulator.test.js`
|
||||
- `tests/ui/scenario-form.test.jsx`
|
||||
|
||||
And then by the broader requested validation pass with lint and build.
|
||||
@@ -0,0 +1,48 @@
|
||||
# v0.6 Comparability Experiment
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The engine should confirm that observations are comparable before treating their difference as a contradiction that needs explanatory follow-up.
|
||||
|
||||
## Fixtures
|
||||
|
||||
1. Revenue increased by 18%, but cash in the bank fell over the same period.
|
||||
2. Complaints increased. Production increased.
|
||||
3. Average delivery time decreased by 25%, but order cancellations increased.
|
||||
4. Customer satisfaction increased, but complaints increased.
|
||||
5. Temperature increased. Ice melted.
|
||||
6. Sales doubled. Sales doubled.
|
||||
|
||||
## Results
|
||||
|
||||
- The first four scenarios repeated the same failure pattern: contradiction-level investigation could begin before comparability was established.
|
||||
- A deterministic comparability gate corrected that by producing one comparison question first.
|
||||
- Confirmed comparability did not by itself imply contradiction.
|
||||
- Temperature increased / Ice melted was reclassified as a compatible relationship, so no contradiction question was asked.
|
||||
- Sales doubled / Sales doubled was reclassified as duplicate observations, so no follow-up question was asked.
|
||||
|
||||
## Relationship classification stage
|
||||
|
||||
After comparability assessment, observations now pass through a deterministic relationship classification stage:
|
||||
|
||||
- `contradictory`
|
||||
- `compatible`
|
||||
- `potentially_related`
|
||||
- `duplicate`
|
||||
- `insufficient_information`
|
||||
|
||||
## Whether comparability should become a permanent reasoning stage
|
||||
|
||||
Yes, in minimal deterministic form.
|
||||
|
||||
The repeated pattern appeared in four scenarios, so a small pre-contradiction comparability assessment is justified.
|
||||
|
||||
## Two-step experiment result
|
||||
|
||||
A comparison question is useful only if its answer advances the reasoning stage rather than merely adding more text.
|
||||
|
||||
In the revenue-versus-cash scenario, the first question now confirms whether the figures are comparable, and the answer resolves that existing uncertainty instead of creating a parallel note. After that update, the engine progresses from comparability assessment to cautious relationship assessment and can select one broad non-expert follow-up question.
|
||||
|
||||
Every justified next question should correspond to an explicit unresolved graph node.
|
||||
|
||||
The earlier fallback-only path has now been removed from the normal successful progression. After comparability is resolved and a further investigation question is justified, the engine creates or reuses an explicit unresolved reasoning unknown and lets deterministic selection and question formulation proceed through the standard graph pipeline. A fallback is now only acceptable as an explicit failure case, not as the normal source of the next question.
|
||||
@@ -0,0 +1,89 @@
|
||||
# v0.6 Release Notes
|
||||
|
||||
## Purpose
|
||||
|
||||
v0.6 turns the engine into a deterministic recursive reasoning system that keeps next questions, decomposition, propagation, and confidence updates explicitly grounded in the situation graph.
|
||||
|
||||
## Capabilities added
|
||||
|
||||
- deterministic unknown selection explanations
|
||||
- explicit ambiguity handling instead of silent tie-breaking
|
||||
- comparability assessment before relationship reasoning
|
||||
- relationship classification after comparability
|
||||
- reasoning-stage progression after comparability answers
|
||||
- graph-backed next questions via explicit unknown nodes
|
||||
- investigation-strategy-based question formulation
|
||||
- atomicity assessment for selected unknowns
|
||||
- composite-unknown decomposition into child unknowns
|
||||
- child-quality validation for decomposition outputs
|
||||
- upward propagation from resolved children to parents and ancestors
|
||||
- separation of evidence confidence, completeness, and conclusion confidence
|
||||
- deterministic cross-branch corroboration, conflict, and duplicate-evidence handling
|
||||
- developer-facing reasoning architecture documentation
|
||||
|
||||
## Reasoning pipeline summary
|
||||
|
||||
```text
|
||||
Scenario
|
||||
→ Reconstruction
|
||||
→ Initial graph
|
||||
→ Deterministic unknown selection
|
||||
→ Question
|
||||
→ Answer
|
||||
→ Proposal
|
||||
→ Proposal parsing / validation
|
||||
→ Graph update
|
||||
→ Reasoning-state rebuild
|
||||
→ Comparability assessment
|
||||
→ Relationship classification
|
||||
→ Emergent unknown creation / reuse
|
||||
→ Atomicity assessment
|
||||
→ Optional decomposition
|
||||
→ Propagation
|
||||
→ Confidence / completeness / corroboration update
|
||||
→ Next active unknown
|
||||
→ Next question
|
||||
```
|
||||
|
||||
## Core invariants
|
||||
|
||||
- every asked question must originate from an explicit unresolved unknown
|
||||
- unknown selection is deterministic
|
||||
- ambiguity is preserved explicitly when no justified distinction exists
|
||||
- relationship reasoning cannot precede comparability
|
||||
- parent nodes cannot resolve before completion rules are met
|
||||
- confidence cannot outrun completeness
|
||||
- duplicate evidence cannot increase confidence
|
||||
- conflicting evidence caps conclusion confidence
|
||||
- cross-branch corroboration only counts for distinct branches with distinct evidence keys
|
||||
- the LLM proposes updates but does not mutate the graph directly
|
||||
|
||||
## What v0.6 proved
|
||||
|
||||
- graph-backed questioning works better when every justified next question maps to an explicit unresolved node
|
||||
- broad unknowns can be decomposed deterministically before direct questioning
|
||||
- resolved child evidence can be propagated upward without prematurely resolving parent reasoning
|
||||
- confidence becomes easier to reason about when evidence quality, completeness, and conclusion strength are separated
|
||||
- deterministic cross-branch corroboration can improve support without double-counting repeated evidence
|
||||
|
||||
## Known limitations
|
||||
|
||||
- sibling selection still depends on the existing deterministic scorer and may choose a justified next branch that is not always the intuitively expected one
|
||||
- cross-branch corroboration is limited to direct child branches of the same parent
|
||||
- no multi-hop corroboration exists across unrelated subtrees
|
||||
- reasoning remains bounded to explicitly represented graph structure and user-provided answers
|
||||
|
||||
## Deliberate exclusions
|
||||
|
||||
- no persistence
|
||||
- no autonomous exploration
|
||||
- no probabilistic reasoning
|
||||
- no Bayesian reasoning
|
||||
- no semantic embeddings
|
||||
- no expert mode
|
||||
- no multi-hop corroboration across unrelated subtrees
|
||||
- no heavy graph visualisation
|
||||
|
||||
## Next experimental question
|
||||
|
||||
`Can the engine preserve and reuse successful reasoning structures across separate cases without turning prior experience into unquestioned assumptions?`
|
||||
@@ -0,0 +1,50 @@
|
||||
# v0.6 Selection Influence Experiment
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The initial unknown selected for the revenue-versus-cash scenario may be driven more by graph structure, more by semantic keyword matches, or by both together.
|
||||
|
||||
## Scenario
|
||||
|
||||
`Revenue increased by 18%, but cash in the bank fell over the same period.`
|
||||
|
||||
## Actual selected node
|
||||
|
||||
- Node ID: `nqdzobz`
|
||||
- Label: `Magnitude and nature of cash outflows (operating expenses, debt repayments, capex, or working capital shifts).`
|
||||
- Deterministic investigation strategy: `definition`
|
||||
- Deterministic question: `What evidence would resolve whether magnitude and nature of cash outflows (operating expenses, debt repayments, capex, or working capital shifts). is true?`
|
||||
|
||||
## Structural contribution
|
||||
|
||||
- Downstream dependency count: `0`
|
||||
- Prerequisite position: no unresolved prerequisites; count `0`
|
||||
- Dependency ordering / centrality: no candidate had downstream dependants or dependency depth advantage in the live graph
|
||||
|
||||
## Semantic contribution
|
||||
|
||||
- Objective: false
|
||||
- Actor: false
|
||||
- Criteria: false
|
||||
- Measurement: false
|
||||
- Terminology: false
|
||||
- Constraint: false
|
||||
- Pricing: false
|
||||
- Implementation: false
|
||||
- Optimisation: false
|
||||
- Speculative: false
|
||||
- Contribution list: only `downstream_dependencies` was present, with delta `0`
|
||||
|
||||
## Counterfactual results
|
||||
|
||||
- Live-shaped ordering: `nqdzobz` ranked above `niewza`, but both had score `0`, downstream `0`, and unresolved prerequisites `0`
|
||||
- Links removed: ordering stayed the same, because the live graph already provided no differentiating structure between the two unknowns
|
||||
- Wording neutralised: ordering flipped to the first unknown by neutral label order (`Unknown A` before `Unknown B`), showing the outcome remained tie-break-driven rather than structure-driven
|
||||
|
||||
## Conclusion
|
||||
|
||||
For this scenario, the actual winner was not selected because of graph structure and not selected because of semantic keyword weights. The live diagnostics show a complete tie on score, downstream influence, and prerequisite position, with every semantic match category false for both candidates. The winner was therefore chosen by the final tie-break rule, `label_asc`.
|
||||
|
||||
## Is a scoring change justified?
|
||||
|
||||
Not from this single experiment alone. The result shows a diagnostic gap for this scenario, but this task does not justify a scoring change by itself, and no scoring change is made.
|
||||
@@ -0,0 +1,288 @@
|
||||
## Post-update selection invariant
|
||||
|
||||
After every successful graph update, the full deterministic question-selection pipeline must run again whenever eligible unresolved unknowns remain.
|
||||
|
||||
The active reasoning pattern constrains which graph nodes may participate in reasoning.
|
||||
|
||||
That means the update path must not stop at graph mutation, child resolution, emergent unknown creation, decomposition, or upward propagation. It must continue through:
|
||||
|
||||
```text
|
||||
updated graph
|
||||
→ rebuild reasoning state
|
||||
→ identify unresolved candidates
|
||||
→ select active unknown
|
||||
→ atomicity assessment
|
||||
→ answerability assessment
|
||||
→ decompose if required
|
||||
→ reselect
|
||||
→ reasoning-pattern selection
|
||||
→ investigation-strategy selection
|
||||
→ question-family selection
|
||||
→ question formulation
|
||||
→ complexity validation
|
||||
→ selectedQuestion
|
||||
```
|
||||
|
||||
Returning no question is only valid when no eligible unresolved candidate remains, the case is complete, ambiguity cannot be safely resolved, or question formulation fails validation with an explicit deterministic reason.
|
||||
|
||||
## Graph validity vs reasoning-pattern validity
|
||||
|
||||
These are separate requirements.
|
||||
|
||||
- **Graph validity** means references, IDs, node shapes, and update semantics are structurally correct.
|
||||
- **Reasoning-pattern validity** means selectable investigation nodes are compatible with the current reasoning mode.
|
||||
|
||||
A graph can be structurally valid while still being reasoning-invalid.
|
||||
|
||||
Example: a decision investigation may still contain an unresolved comparison-style node such as `How the two observations were measured`. That node is structurally well-formed, but it is not allowed to participate as an active investigation target unless the reasoning pattern has actually shifted into comparison, contradiction, or explanation work.
|
||||
|
||||
The engine therefore needs both invariants:
|
||||
|
||||
1. the graph must be structurally valid
|
||||
2. every selectable unknown must be compatible with the active reasoning pattern
|
||||
|
||||
# v0.7 Question Simplicity Experiment
|
||||
|
||||
## Observed failure
|
||||
|
||||
The first v0.7 UI scenario exposed a reasoning failure where the selected unknown could still be directionally correct while the resulting question was too large to answer in one coherent response.
|
||||
|
||||
Example failure:
|
||||
|
||||
> Have you measured the current financial or operational cost to users who lack justified confidence, and what baseline budget do they currently allocate for comparable decision-support methods?
|
||||
|
||||
This question bundled multiple investigations:
|
||||
|
||||
- cost
|
||||
- user impact
|
||||
- existing alternatives
|
||||
- current budget
|
||||
|
||||
That violated the intended one-step reasoning discipline.
|
||||
|
||||
## Principle
|
||||
|
||||
**A correct unknown paired with an unanswerably broad question is still a reasoning failure.**
|
||||
|
||||
The engine should ask one question about one primary concept at a time.
|
||||
|
||||
**The reconstruction model may suggest a question, but only the graph-backed deterministic pipeline may select the user-facing question.**
|
||||
|
||||
**A node is only questionable if it is independently answerable.**
|
||||
|
||||
## One-question / one-concept rule
|
||||
|
||||
Every user-facing question should:
|
||||
|
||||
- contain one question mark
|
||||
- target one unresolved graph node
|
||||
- ask for one primary concept
|
||||
- request one coherent answer
|
||||
- avoid joined investigations
|
||||
- minimise cognitive effort while still reducing meaningful uncertainty
|
||||
|
||||
## Deterministic cognitive-load rules
|
||||
|
||||
The new deterministic question-complexity assessment marks a question as too broad when it shows signals such as:
|
||||
|
||||
- multiple requested answers joined by `and`
|
||||
- distinct measures combined in one prompt, such as cost plus budget
|
||||
- comma-list phrasing that expands the request into several sub-questions
|
||||
- more than one primary concept
|
||||
- abstract noun chains that make the question hard to parse on first reading
|
||||
- very long question length
|
||||
|
||||
The assessment returns:
|
||||
|
||||
- `acceptable`
|
||||
- `primaryConceptCount`
|
||||
- `compoundQuestionSignals`
|
||||
- `abstractTermCount`
|
||||
- `cognitiveLoad`
|
||||
- `reasons`
|
||||
|
||||
## Decomposition-before-rewording rule
|
||||
|
||||
The engine now treats broad commercial-validation unknowns as composite.
|
||||
|
||||
If the selected unknown still spans multiple validation dimensions, the system should not simply shorten the sentence. It should first decompose the unknown into smaller child unknowns and then select one foundational child.
|
||||
|
||||
For the current scenario, this meant creating child unknowns such as:
|
||||
|
||||
- who experiences the problem
|
||||
- what happens when it is not resolved
|
||||
- how often it happens
|
||||
- how people deal with it today
|
||||
- whether people actively look for help
|
||||
|
||||
The selector then reaches the first foundational child through prerequisite ordering encoded in the decomposition graph rather than through global scoring changes.
|
||||
|
||||
## Atomicity vs answerability
|
||||
|
||||
These are different reasoning properties.
|
||||
|
||||
- **Atomicity** asks: does this node describe one investigation or several bundled investigations?
|
||||
- **Answerability** asks: even if the wording looks singular, can this node be answered directly without first resolving multiple prerequisite dimensions?
|
||||
|
||||
A node can appear atomic in wording but still fail answerability.
|
||||
|
||||
Examples include broad evaluation containers such as product validation, customer value, business case, technical feasibility, or commercial justification. These often compress several prerequisite investigations into one conclusion-shaped unknown.
|
||||
|
||||
That means atomicity alone is not enough.
|
||||
|
||||
The engine now decomposes whenever either of these is true:
|
||||
|
||||
- the unknown is not atomic
|
||||
- the unknown is not independently answerable
|
||||
|
||||
This prevents a broad container node from becoming the selected question target even when its wording looks grammatically singular.
|
||||
|
||||
## Reasoning Pattern
|
||||
|
||||
The next failure exposed a deeper issue: even after atomicity and answerability were added, the engine could still choose a question template from the wrong reasoning family.
|
||||
|
||||
The live failure was an explanation-style prompt appearing in a commercial validation scenario:
|
||||
|
||||
> What changed during the period that could help explain why ...
|
||||
|
||||
That was wrong not because of wording, but because the engine had selected an **explanation family** when the actual task was a **decision investigation**.
|
||||
|
||||
To correct that, the deterministic pipeline now explicitly inserts a reasoning-pattern stage:
|
||||
|
||||
```text
|
||||
selected unknown
|
||||
→ atomicity
|
||||
→ answerability
|
||||
→ reasoning pattern
|
||||
→ investigation strategy
|
||||
→ question family
|
||||
→ question
|
||||
```
|
||||
|
||||
This matters because each stage must constrain the next.
|
||||
|
||||
- **Reasoning Pattern** decides what kind of reasoning is happening
|
||||
- **Investigation Strategy** decides how to reduce uncertainty within that pattern
|
||||
- **Question Family** decides what template space is allowed
|
||||
- **Question** is the final concrete wording
|
||||
|
||||
Without this stage separation, strategy and template selection can leak across domains and reuse relationship/explanation prompts too broadly.
|
||||
|
||||
## Deterministic reasoning-pattern vocabulary
|
||||
|
||||
The current deterministic pattern vocabulary is intentionally small:
|
||||
|
||||
- decision
|
||||
- explanation
|
||||
- contradiction
|
||||
- definition
|
||||
- diagnosis
|
||||
- comparison
|
||||
- prioritisation
|
||||
|
||||
Pattern selection uses graph structure rather than wording alone, including:
|
||||
|
||||
- node kind
|
||||
- relationship / observation topology
|
||||
- parent context
|
||||
- reasoning state
|
||||
- selected unknown role in the graph
|
||||
|
||||
## Question-family mapping
|
||||
|
||||
Patterns now constrain which question families are allowed.
|
||||
|
||||
- **decision**
|
||||
- decision_foundation
|
||||
- decision_evidence
|
||||
- decision_threshold
|
||||
- definition
|
||||
- **explanation**
|
||||
- explanation
|
||||
- comparison
|
||||
- **contradiction**
|
||||
- contradiction
|
||||
- comparison
|
||||
- explanation
|
||||
- **definition**
|
||||
- definition
|
||||
- **diagnosis**
|
||||
- diagnosis
|
||||
- comparison
|
||||
- **comparison**
|
||||
- comparison
|
||||
- **prioritisation**
|
||||
- prioritisation
|
||||
- decision_threshold
|
||||
|
||||
Most importantly:
|
||||
|
||||
- explanation templates are only allowed for `explanation` or `contradiction`
|
||||
- decision investigations cannot emit explanation-family questions
|
||||
|
||||
## Live correction
|
||||
|
||||
For the commercial-method scenario, the engine now classifies the reasoning as a **decision** pattern rather than an explanation pattern.
|
||||
|
||||
That means explanation-family templates are explicitly rejected, and the selected child unknown must be questioned using a decision-compatible family instead.
|
||||
|
||||
## UI result
|
||||
|
||||
The long compound question no longer survives as the first follow-up in the tested path.
|
||||
|
||||
The new first-step question is:
|
||||
|
||||
> Who experiences this problem?
|
||||
|
||||
This question:
|
||||
|
||||
- asks one thing
|
||||
- is understandable immediately
|
||||
- stays graph-backed
|
||||
- avoids pricing or budget before problem existence is established
|
||||
|
||||
## Start-case authority rule
|
||||
|
||||
There were previously two question paths during initial analysis:
|
||||
|
||||
- reconstruction model `nextQuestion`
|
||||
- graph-backed unknown selection and question formulation
|
||||
|
||||
The defect was that `startCase` copied the reconstruction `nextQuestion` directly into the normal UI.
|
||||
|
||||
That path is now closed.
|
||||
|
||||
Initial user-facing questioning now follows this pipeline:
|
||||
|
||||
```text
|
||||
reconstruction
|
||||
→ graph build
|
||||
→ unresolved unknown selection
|
||||
→ atomicity assessment
|
||||
→ decomposition if needed
|
||||
→ investigation strategy
|
||||
→ question formulation
|
||||
→ complexity validation
|
||||
→ selectedQuestion
|
||||
```
|
||||
|
||||
The reconstruction question is still retained in diagnostics as provenance, but it is not authoritative.
|
||||
|
||||
## Live result
|
||||
|
||||
Running the commercial-method scenario through the real environment now:
|
||||
|
||||
- succeeds without the enum compatibility failure
|
||||
- does not show the broad reconstruction question in the UI path
|
||||
- surfaces a graph-backed first question instead
|
||||
- keeps the reconstruction question only in diagnostics
|
||||
|
||||
For the tested scenario, the user-facing first question remained:
|
||||
|
||||
> Who experiences this problem?
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
- question-complexity assessment is still conservative and pattern-based rather than semantic in a richer linguistic sense
|
||||
- plain-language simplification currently uses a small deterministic replacement set
|
||||
- broader prerequisite ordering is strongest for decomposition structures that explicitly encode those dependencies
|
||||
@@ -0,0 +1,157 @@
|
||||
# v0.7 UX First Pass — User-Focused Reasoning Workspace
|
||||
|
||||
## UX Problem
|
||||
|
||||
The current interface exposes the reasoning engine's graph structure directly to users. It presents:
|
||||
|
||||
- Raw node-grouped tables with status/confidence badges
|
||||
- Diagnostic metadata (model name, prompt version, validation status)
|
||||
- Graph update change details (resolved nodes, affected nodes, proposal JSON)
|
||||
- A bare "Waiting for model response..." placeholder with no elapsed time or rotating status
|
||||
|
||||
This is useful as a developer/debug view but difficult to understand for non-technical users. The next question is visually buried under the graph tables, and there is no clear feedback during slow LLM analysis or update operations.
|
||||
|
||||
## Design Goals
|
||||
|
||||
- **Calmer default view**: Present scenario, understanding, focus, next question, and progress as a sequence of clean cards
|
||||
- **Preserve full debug access**: All existing graph, diagnostics, and update history components remain available behind a collapsed disclosure
|
||||
- **Clear slow-operation feedback**: Animated spinner, elapsed time, rotating plain-language status messages during analysis and update operations
|
||||
- **Professional visual tone**: Neutral colours, generous whitespace, restrained borders, no gradients or glassmorphism
|
||||
|
||||
## Main Workspace Structure
|
||||
|
||||
The `ReasoningWorkspace` component (`components/reasoning-workspace.jsx`) renders the result area. When a successful start analysis completes, it shows:
|
||||
|
||||
1. **Your situation** — Central statement from `situationGraph.centralStatement`, displayed in a white card
|
||||
2. **Current understanding** — The API's `currentSummary` text in a second white card
|
||||
3. **What we are working out** — The active unknown label, its description ("Why it matters"), and a plain-language status badge (e.g., "Under investigation")
|
||||
4. **Next question** — The largest visual element: green-bordered card with bold heading and prominent question text in `text-xl` font-weight-semibold
|
||||
5. **Progress** — A single inline bar showing resolved count + remaining unknown count (no percentage)
|
||||
6. **Answer form** — Visible only when a selected question exists; textarea + "Update situation" button, disabled during update loading
|
||||
7. **Developer details** — Collapsible `<details>` element with full SituationGraphView, GraphUpdateView, and DiagnosticsView inside; closed by default
|
||||
|
||||
When analysis completes without producing a graph:
|
||||
- A yellow warning card states the outcome plainly
|
||||
- Error messages remain in red cards above all content
|
||||
|
||||
When there is no next question:
|
||||
- A calm gray card says "There is no next question at the moment." with a contextual elaboration derived from `noQuestionReason` when available
|
||||
- No broken-looking empty areas appear
|
||||
|
||||
## Loading-State Behaviour
|
||||
|
||||
### Initial analysis (start request)
|
||||
|
||||
A blue-bordered card appears with:
|
||||
- **Spinner** — CSS-only spinning ring (`@keyframes spin`)
|
||||
- **Heading**: "Working through your situation"
|
||||
- **Rotating status text** (based on elapsed seconds):
|
||||
- 0–10s: "Reading your situation"
|
||||
- 10–25s: "Building a structured understanding"
|
||||
- 25–45s: "Identifying what is known and still unclear"
|
||||
- 45+s: "Selecting the next useful question"
|
||||
- **Elapsed time**: "This has been running for Xs."
|
||||
- **Reassuring copy** (shown after 30s): "This can take around a minute with the current local model."
|
||||
|
||||
### Answer update (update request)
|
||||
|
||||
Same card format, different status text pool:
|
||||
- 0–10s: "Considering your answer"
|
||||
- 10–25s: "Updating the situation"
|
||||
- 25–45s: "Checking what changed"
|
||||
- 45+s: "Choosing the next question"
|
||||
|
||||
### Duplicate submit prevention
|
||||
|
||||
Both "Analyse" and "Update situation" buttons are `disabled` while their respective `status` / `updateStatus` is `"loading"`. The answer textarea also disables during update loading.
|
||||
|
||||
## Debug View Preservation
|
||||
|
||||
All existing components are preserved inside the collapsed "Developer details" `<details>` element:
|
||||
|
||||
- **SituationGraphView** — Full node-grouped graph with badges, active unknown highlighting, newly surfaced markers, and raw JSON toggle
|
||||
- **GraphUpdateView** — Update history (resolved unknowns, newly surfaced unknowns, affected nodes, proposal details)
|
||||
- **DiagnosticsView** — Model name, provider, prompt version, duration, validation status, node/edge counts
|
||||
|
||||
These are only accessible by expanding the disclosure. Raw node IDs do not appear in any user-facing card text.
|
||||
|
||||
## Deliberate Exclusions (for this pass)
|
||||
|
||||
- Spider/dag graph rendering
|
||||
- Persistence or session handling
|
||||
- Navigation or routing changes
|
||||
- Accounts or authentication
|
||||
- Export functionality
|
||||
- Dark mode
|
||||
- Radical input page redesign
|
||||
- Backend code changes (APIs, routes, logic, prompts, schemas)
|
||||
- Reasoning test modifications
|
||||
- New component library additions
|
||||
|
||||
## Loading Feedback Refinement
|
||||
|
||||
The loading state was tightened for clarity:
|
||||
|
||||
- Reassurance message threshold moved from 30 s to 45 s to avoid premature reassurance.
|
||||
- Elapsed time displayed in seconds during both initial analysis and answer update.
|
||||
- Rotating status messages continue per the original pools, changing based on elapsed seconds only.
|
||||
|
||||
## Progress Card — Unexplained Counts Replaced
|
||||
|
||||
The standalone "X remaining" text was replaced with a `Reasoning progress` card:
|
||||
|
||||
- **Areas under investigation** — Plain-language statement of how many areas remain (e.g., "We have identified 1 area that still needs investigation.").
|
||||
- **Current focus** — The active unknown label, shown in plain language.
|
||||
- **Why this matters** — The active unknown's description, when available.
|
||||
- Fallback text ("There is no active area of investigation at the moment.") when there is no active unknown and no remaining areas.
|
||||
|
||||
Words such as "unknown nodes", "unresolved nodes", "remaining graph items", and "candidate count" are intentionally avoided in user-facing copy.
|
||||
|
||||
## Current Understanding Wording
|
||||
|
||||
The `Current understanding` card continues to display whatever text `currentSummary` provides from the API. When `currentSummary` is absent, a calm fallback message appears: "We have started to separate what is known from what still needs checking." Technical graph counts (node types, edge totals) are no longer constructed or displayed in user-facing sections — they are only available inside the collapsed Developer details disclosure.
|
||||
|
||||
## Developer-Detail Boundary
|
||||
|
||||
- **User-facing cards** show: situation summary, current understanding, reasoning progress with active focus, and next question — all without raw IDs, node kinds, or internal enum names.
|
||||
- **Developer details** (collapsed `<details>` element) preserves the full SituationGraphView (node groups, badges, edge info), GraphUpdateView (update history, proposal details), and DiagnosticsView (model name, prompt version, validation status, node/edge counts).
|
||||
- No user-facing card renders raw node IDs or technical graph metadata.
|
||||
|
||||
## Remaining UX Limitations
|
||||
|
||||
1. **Multi-turn not implemented** — The workspace currently reflects the one-update prototype limitation. A multi-turn version would need persistent state management between turns.
|
||||
2. **Timer is client-side only** — Elapsed time starts when loading begins but no backend stage telemetry is exposed yet, so rotating messages are honest approximations only.
|
||||
3. **No skeleton/loading shimmer** — The spinner card replaces content entirely during loading rather than showing a layout-aware skeleton. A skeleton approach would be a future enhancement.
|
||||
4. **Loading overlay does not persist across route changes** — No persistence layer means refresh loses state. This is intentional for the prototype scope.
|
||||
5. **No visual distinction between "idle" and "success" empty states** — Both render similarly when no answer is typed. A small hint like "Type an answer to continue" could be added later.
|
||||
6. **Progress count uses resolved/remaining labels only** — No percentage or bar despite having the data, per constraint. This is intentional; we avoid false precision in a prototype context.
|
||||
|
||||
## Files Changed
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `components/reasoning-workspace.jsx` | Loading feedback refinement (45 s threshold); ProgressSummary → ReasoningProgress card; CurrentUnderstanding simplified; DeveloperDetails boundary clarified |
|
||||
| `tests/ui/scenario-form.test.jsx` | Added 8 new focused UI tests covering progress card, reasoning focus, loading behavior, and technical-data isolation; removed outdated "remaining" count assertion |
|
||||
| `docs/v0.7-user-workspace-ux-first-pass.md` | Added sections for loading feedback refinement, progress-card replacement, current-understanding wording, developer-detail boundary |
|
||||
|
||||
## Test Results
|
||||
|
||||
- All 58 UI tests pass (50 existing + 8 new)
|
||||
- ESLint: no warnings or errors
|
||||
- Next.js build: clean, no new route entries or compilation issues
|
||||
|
||||
## Manual UI Notes
|
||||
|
||||
A single manual check was not performed in this pass. The next step for verification is:
|
||||
|
||||
1. Run `npm run dev`
|
||||
2. Submit a scenario to an available local LLM endpoint
|
||||
3. Confirm the initial loading card shows rotating status messages
|
||||
4. Confirm the result renders as a clean sequence of cards with "Next question" as the strongest visual element
|
||||
5. Expand "Developer details" and confirm graph/diagnostics/updates are preserved
|
||||
6. Submit an answer and confirm update loading feedback appears
|
||||
7. Confirm no raw node IDs appear outside the developer section
|
||||
|
||||
---
|
||||
|
||||
*This is a first-pass UX improvement only. Reasoning logic, API contracts, schemas, and tests remain unchanged.*
|
||||
Reference in New Issue
Block a user