robbond/confidence-engine

Fork 0

Files

T

robbond 787c8114ad docs(confidence-engine): record focused progression walkthrough findings

2026-08-26 12:40:37 +01:00

22 KiB

Raw Blame History

Confidence Engine — Current Handoff

Repository position

branch: feature/decision-closure-ownership-v0.47
checkpoint commit: 772ae49

Current green reasoning state

substantive-tie active ownership: In complete unresolved ties among top-scoring candidates, the currently active node is preserved as the selection winner rather than falling through to stable label/display-order ordering. This only applies when the active node is eligible and remains substantively tied at the structural level.
question-rejection active ownership: When a selected candidate's graph-backed question formulation is rejected as too complex (decomposition-required), the selected target node retains its ownership — it is not reseated to another candidate via reseatSelectionAfterQuestionRejection. Instead, the selection remains on the original node with selectedQuestion = null and an explicit rejection reason.
QUESTION_CONTINUATION: Existing question continuation logic remains intact and functional after the question-rejection fix.
60B.84: State B guard correctly does not fire for specific factor nodes merely because they are inside a decision context.
State B: Sufficiency question path reaches expected terminal state without unintended firings.
captured product-launch replay: The live product-launch start graph replays deterministically: ntpt9ki remains the active investigation target with no question selected after formulation rejection.

Latest resolved reasoning boundaries

1. Complete substantive selection tie

previous behaviour: When all leading structural candidates were tied, the system always fell through to stable label/display-order as the final deterministic tie-breaker, regardless of which node was currently active in the investigation.
corrected invariant: If the active node is among the tied structural candidates and remains eligible (unresolved, not contradicted), it is preserved as the winner. The stable label/display-order fallback is only used when there is no active candidate or when the active candidate does not remain among the top structural ties.
regression location: lib/graph/utils.js — classifyCandidateOrdering() now accepts an activeNodeId parameter and checks for active-tied candidates within the leading structural set before using display-order fallback. Callers in selectActiveUnknownCandidate() and explainUnknownSelection() pass graph.activeUnknownNodeId.
regression test location: tests/graph/utils.test.js — tests: "preserves the active candidate when it remains eligible and substantively tied", "transfers ownership when the active candidate substantively loses on score", "transfers ownership when the active candidate is resolved or ineligible"

2. Question-formulation rejection

previous behaviour: When a selected candidate's question formulation was rejected (decomposition required), determineGraphBackedQuestion called reseatSelectionAfterQuestionRejection with excludedNodeIds that excluded the current target, causing investigation ownership to transfer to another candidate — even though the original target remained the strongest unresolved unknown.
corrected invariant: The selected node keeps its status and selection identity. selectedQuestion is set to null and questionSuppressedReason records the rejection explanation. No reseating occurs.
regression/captured fixture location: tests/graph/orchestrator.test.js — "retains ownership when the strongest target's formulated question is rejected"; captured replay via tests/fixtures/live-product-launch-start-response.json

Current deterministic product-launch evidence

deterministicSelection.nodeId = ntpt9ki (active investigation target preserved)
selectedQuestion = null after rejected formulation
explicit noQuestionReason: "The selected investigation target remains active, but its current graph-backed question formulation was rejected as too complex."
nxmeiab is not substituted in place of ntpt9ki

Current product meaning

activeUnknownNodeId represents ongoing investigation ownership — it tracks which unknown candidate the system has committed to investigating.
Wording/formulation failure (question complexity / decomposition-required) does not itself invalidate the target. The target remains selected even when its formulated question cannot be answered in one step.
Stable label ordering remains only a final fallback after substantive scoring, structural comparison, semantic signature checks, and tie/ownership handling are all exhausted.

Not yet proved

live behavioural validation after these fixes (requires an actual product-launch run through the dev server)
whether same-target reformulation is better than no-question (system currently uses no-question approach)
broader/full-suite state beyond the six verified guards
correctness of the question-complexity heuristic itself (that is a separate design concern)

Canonical live apparatus for next validation

scripts/reproduce-multi-turn-investigation.mjs
existing healthy dev server
.env.local authoritative for:
- OLLAMA_BASE_URL
- OLLAMA_MODEL
no model discovery
no supplementary harnesses
no direct Ollama calls

Next recommended step

one observation-only live product-launch validation using the fixed scenario from the recorded journey
no production changes during the experiment
verify that the live LLM responds consistently to the null-question state and continues investigation on ntpt9ki

Apparatus correction: 60B.101 — null-question start capture

The canonical startOnly harness was corrected to accept successful Start responses with selectedQuestion = null. Previously, any successful Start returning no graph-backed question (legitimate outcome meaning "target remains active but no askable question available") caused the harness to block and fail.

Change: The harness now checks success === true + valid situationGraph as the sole gate for startOnly success. selectedQuestion is preserved exactly (including null) in the continuation state file without coercion.

Impact on 60B.100: The evidence from 60B.100 was captured via direct curl because the harness blocked on null-question Start. That evidence is now marked as apparatus-contaminated and provisional observation only.

Canonical harness gated apparatus (60B.99)

The canonical harness (scripts/reproduce-multi-turn-investigation.mjs) now supports a two-phase gated investigation pattern:

startOnly — FIXTURE_MODE=startOnly

Makes exactly one /api/cases/start request
Writes the captured Start state (graph + selectedQuestion) to .evidence-temp/continuation-start-only.json (or path set by CONTINUATION_FILE)
Issues zero Update requests
Exits successfully

continueOneUpdate — FIXTURE_MODE=continueOneUpdate CONTINUATION_ANSWER=<answer>

Loads the persisted Start continuation state
Requires explicit answer (blocks with exit code 1 if missing)
Makes exactly one /api/cases/update using preserved Start state + explicit answer
Issues zero Start requests
Exits

Normal mode (FIXTURE_MODE unset) — unchanged. Start → configured Update loop still works identically to pre-60B.99.

This apparatus corrects the apparatus defect proven in 60B.98: the canonical harness can now stop after Start, allow external semantic inspection of the returned question, and later continue from that exact captured state with an explicitly chosen answer.

Experiment 60B.95 result (2026-08-17)

Classification: E — LIVE PATH DIVERGED

The live model selected nk6eyn2 ("exact monetary value of potential enterprise contract relative to £300k launch cost") as the investigation target, not npzfx36 ("likelihood, negotiation stage, and targeted signing date for the large enterprise customer"). Both are unresolved unknowns in the same scenario. An acceptable question was produced ("What outcome would demonstrate enough value to justify launching a software product now?"), so the question-rejection boundary was not reached.

What this establishes: The live engine can produce an acceptable graph-backed question on a fresh product-launch start without requiring decomposition.

What this does NOT prove: Whether investigation ownership is preserved when a selected target's formulation is rejected (the core invariant from checkpoint 60B.93). The question-rejection boundary was not reached because the live model chose a different investigation target with an acceptable question path.

Experiment 60B.97 result (2026-08-18)

Classification: E — START PATH DIVERGED

The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison.

Valid evidence retained: Start = S2 — DIFFERENT START (the live model diverged from the expected financial-comparison path).

Update 1 evidence: DISCARDED. The canonical harness auto-continued with its preconfigured answers[0], so the Update occurred outside the experiment's semantic gate. This was an apparatus defect (60B.98) — the harness did not provide a post-Start stop gate at that time. The HTTP 500 is NOT established as a reasoning defect from 60B.97.

Apparatus correction: See section "Canonical harness gated apparatus (60B.99)" above for the fix.

Experiment 60B.100 result (2026-08-18)

Classification: B — DETERMINISTIC SELECTOR OVERRIDES MODEL QUESTION

On a fresh product-launch Start, the LLM reconstruction question targeted one uncertainty ("What is the estimated probability that the large enterprise customer will sign?") while the deterministic graph-backed selector chose another ("What evidence would clarify the exact percentage of total projected revenue attributable to the enterprise customer?"). These are materially different: one asks about deal timing/commitment probability, the other asks about financial proportion/magnitude.

The override was produced by fixed actor_match keyword scoring: node n65sgyd's label contained "enterprise customer" which matched the actor dictionary (+10 delta), giving it a decisive score of 10 vs 4 for both competitors. No tie/fallback was involved — the winner was determined entirely by keyword rule weighting.

What this establishes: On fresh Start calls, deterministic keyword signals can override model-inferred investigation priority when node labels differ in dictionary-match patterns. The final investigation target is not the model's contextual judgment but the highest-scoring candidate under fixed scoring rules.

What this does NOT prove: Whether the deterministic selection is better or worse than the model's suggestion; consistency across scenario types; or downstream investigation quality impact.

RETURN-TO-ORIGIN CHECKPOINT

selector-led compulsory next-question optimisation is PAUSED

semantic-selector replacement is also PAUSED

Recent work is preserved as valuable experimental learning. The graph/reconstruction/decomposition/invariant work remains potentially reusable. No replacement architecture has been selected.

The next phase starts from the workspace/methodology question, not from a preselected technical solution.

Required methodology-continuity context (read first)

Before continuing any Confidence Engine reasoning development, load these in order:

docs/current-working-principles.md §0 (Axiomatic Principles A1–A12)

docs/Confidence_Engine_Return_to_Origin_Methodology_Context_2026-08-18.md

The axiomatic principles in current-working-principles.md are now the durable methodological anchor. They codify delivery-platform independence, meaning-over-dictionary reasoning, user-directed non-linear investigation, provisional closure, and progressive disclosure. All future work must be evaluated against these 12 axioms.

The RTO methodology context document covers:

What Confidence Engine is (reasoning method, not software or LLM)
What it is not
The granular answer-fragment hypothesis (working — not architecture)
RTO.14–17 learning and why they did not define the next architecture
Memory/graph separation principle
Optional capability: "Help me answer" / "Answer for me"
Methodology test for future development
Recent experimental evidence (RTO.18, RTO.20–RTO.22) supporting durable principles

Do not begin reasoning development from the latest experiment apparatus. Always load this methodology continuity context first. Without it, a new session will reconstruct the project from recent technical experiments and lose sight of the method the application is meant to embody.

Current Git checkpoint

Branch: feature/decision-closure-ownership-v0.47
HEAD: e1b54e407389c7f1c25ba20645540d404b9e9df9
Latest commit: e1b54e4 test(experiment): checkpoint granular answer fragment apparatus

Status of RTO.14–17

RTO.14–17 experiments proved that an LLM can retain focused knowledge and carry coherent reasoning across turns inside a single investigation. This was valuable evidence.

However, they also revealed that the cumulative reconstruction pattern was reproduced at a lower scope. Do not treat RTO.14–17 as the automatic architecture direction to continue. The next reasoning direction is the granular answer-fragment hypothesis: independent user-chosen question/answer deconstruction without accumulated focused state being passed between LLM calls.

Prohibition: Do not automatically continue into cumulative-state compression or fourth-turn experiments without first revisiting the methodology continuity document above.

Focused Semantic Deconstruction — Reasoning Status (2026-08-24)

Current state

Branch: feature/decision-closure-ownership-v0.47
HEAD: 2c108df5a92ec4cb0c4fa7878002cb7eb0aea4fd

Working tree:

 M lib/graph/focused-investigation.js
?? tests/graph/live-focused-deconstruct-experiment-helper.mjs

No unexpected temporary files or evidence directories.

Canonical focused helper

tests/graph/live-focused-deconstruct-experiment-helper.mjs

Exports runLiveFocusedDeconstructExperiment. Future focused semantic experiments should: import it directly; use the real production buildFocusedDeconstructPrompt; use the production validator/provider through the helper; configure .env.local for OLLAMA_BASE_URL and OLLAMA_MODEL; make the explicitly budgeted number of live calls; never inline prompt/provider/schema logic; never replace the helper ad hoc.

Product flow context

initial scenario → initial/global SituationGraph → user selects original open question/thread → focused formulate → user answer → focused deconstruct → contribution retained against original targetNodeId → subsequent focused formulate/deconstruct rounds

Focused deconstruction accumulates focused contributions rather than rewriting the initial global SituationGraph. Reuse/integration of accumulated focused learning is a later question, not part of this checkpoint.

Semantic rules currently under review (validated in the cumulative diff)

The production buildFocusedDeconstructPrompt encodes six semantic principles:

Observation fidelity — only meaning directly supported by what the user's answer states becomes an observation; do not strengthen implications into observations.
Uncertainty fidelity — preserve uncertainty at the narrowest scope justified by evidence; when one factor is established but nothing else is, keep remaining uncertainty broad rather than inventing specific additional factors/deficits/causes/interventions.
Assumption attribution — attribute only propositions that the user's answer would cease to make sense if false; do not import plausible interpretations from investigation context or model analysis into the assumptions field; return assumptions: [] when no genuine assumption exists.
Relationship strength — only connections the user's answer directly establishes become relationships; co-mentioned facts do not create causal/constraint/dependency links.
Proposition ownership — track who owns each proposition (user's comfort, willingness, threshold, belief, judgement, etc.); do not elevate subjective stance into objective requirement or situational constraint.
Evidence-first / action-appropriate follow-ups — before formulating follow-ups, check whether the question tests a proposition against current epistemic state; if an explanation/deficit/cause/intervention has not been established by evidence, phrase the question to test whether that proposition is true rather than assuming it; do not jump to implementation details until intervention is the relevant next issue.

A supporting negative principle also holds: Do not solve semantic interpretation with an ever-growing deterministic dictionary/regex system. Structure may be deterministic; meaning remains semantic/LLM-based.

Validated learning (bounded evidence)

Bounded focused experiments on this branch showed:

The original real UI failure was reproduced on the actual focused-deconstruct path.
Observations could remain faithful while assumptions/relationships were weakened beyond what the user stated.
Tightening the assumptions contract materially reduced model-generated assumptions being attributed to the user, supported by bounded contrast.
Genuine implicit dependencies can still be preserved through the tightened boundary.
Proposition ownership preserves subjective criteria without weakening genuine factual reports.
Tentative factual reports can retain their uncertainty.
Evidence-first follow-ups avoid premature remedy; practical follow-ups remain possible once a blocker is established.
Uncertainty specificity follows user-supplied evidence rather than defaulting to generic broadness.

Do not treat these as generalisation: each finding is supported by the bounded contrast of tested cases only.

EXP13 status: CONTAMINATED

EXP13 - CONTAMINATED — excluded from evidence

Reasons: iterative production prompt refinement; best-supported live-call count = 4; not valid bounded experiment evidence; tuned EXP13 uncertainty wording was removed during the diff. EXP13 must not be cited as proof of uncertainty behaviour.

EXP13R / EXP14 (valid but conservative)

EXP13R: materially reduced invented uncertainty specificity but retained some residual operational specialisation.
EXP14: showed the same rule could preserve explicit user-supplied BACS/failed-payment specificity while leaving unidentified additional gaps broad.

Do not conclude that the uncertainty boundary is fully solved. The contrast between these two cases does not establish generalisation.

Current restart point

The immediate semantic pass has established a coherent working principle:

The Engine should preserve what was said, how certain it was, who owns the proposition, and how strongly evidence supports relationships; then investigate missing evidence before inventing explanations or remedies.

Deterministic tests (22/22) verify the structural boundaries. Live-call evidence is bounded. No new experiment is run in this checkpoint.

LIVE MANUAL WALKTHROUGH FINDING — Focused Progression UI (2026-08-26)

Verified during repaired Run A multi-turn focused-investigation flow

The following was established by live manual walkthrough:

Run A presentation entry routing: FIXED
Run A focused lifecycle continuity: FIXED
focused question/form/response rendering inside Run A: FIXED
same-node completed result reopen: FIXED
multi-turn focused continuation: WORKING

KNOWN UI PROGRESSION / OWNERSHIP DEFECT — Follow-up Selected-Question Duplication

During the repaired Run A multi-turn focused-investigation flow:

focused answer
→ deconstruction
→ "Questions this raises"
→ user clicks the single proposed follow-up
→ that follow-up becomes the active QUESTION above the textarea

The UI then continues to show the same previous proposal below under:

QUESTIONS THIS RAISES
[same question] → pick this question

So the selected follow-up is simultaneously displayed as:

ACTIVE QUESTION

and:

STILL-AVAILABLE ACTIONABLE FOLLOW-UP

This is visibly duplicated and demonstrates that the proposal state and active-question state are disconnected.

Product interpretation: This is a KNOWN UI PROGRESSION / OWNERSHIP DEFECT, not merely a styling issue. The useful semantic progression is:

answer → what this tells us → still unclear → proposed next question → user selects it → selected question becomes active

Once selected, its old representation must not continue to invite:

→ pick this question

The old proposal should either become non-actionable provenance/history or cease being displayed as a candidate. Do not decide the final UI treatment in this checkpoint.

Record also that possibleFollowUpQuestions currently contains exactly one proposed question, while the user-facing heading says "Questions this raises" and presents it as a selectable candidate. This raises a later product/ownership question:

Is this genuinely a menu of candidate questions, or is it the engine's proposed continuation of the current investigation?

Do not resolve that question now.

Status of remaining items after this walkthrough

follow-up selected-question duplication:
OPEN UI ISSUE

focused → global ownership/integration:
STILL OPEN

Current Understanding incorporating focused learning:
STILL OPEN

Important separation note

The latest second-turn semantic output exposed a separate reasoning issue around assumption attribution/frontier prioritisation. This must not be conflated with the UI duplication defect described above. They are orthogonal concerns — one is presentation state management, the other is semantic reasoning fidelity.

22 KiB Raw Blame History Unescape Escape