179 lines
14 KiB
Markdown
179 lines
14 KiB
Markdown
# Confidence Engine — Current Handoff
|
||
|
||
## Repository position
|
||
- branch: `feature/decision-closure-ownership-v0.47`
|
||
- checkpoint commit: `772ae49`
|
||
|
||
## Current green reasoning state
|
||
- **substantive-tie active ownership**: In complete unresolved ties among top-scoring candidates, the currently active node is preserved as the selection winner rather than falling through to stable label/display-order ordering. This only applies when the active node is eligible and remains substantively tied at the structural level.
|
||
- **question-rejection active ownership**: When a selected candidate's graph-backed question formulation is rejected as too complex (decomposition-required), the selected target node retains its ownership — it is not reseated to another candidate via `reseatSelectionAfterQuestionRejection`. Instead, the selection remains on the original node with `selectedQuestion = null` and an explicit rejection reason.
|
||
- **QUESTION_CONTINUATION**: Existing question continuation logic remains intact and functional after the question-rejection fix.
|
||
- **60B.84**: State B guard correctly does not fire for specific factor nodes merely because they are inside a decision context.
|
||
- **State B**: Sufficiency question path reaches expected terminal state without unintended firings.
|
||
- **captured product-launch replay**: The live product-launch start graph replays deterministically: ntpt9ki remains the active investigation target with no question selected after formulation rejection.
|
||
|
||
## Latest resolved reasoning boundaries
|
||
|
||
### 1. Complete substantive selection tie
|
||
- previous behaviour: When all leading structural candidates were tied, the system always fell through to stable label/display-order as the final deterministic tie-breaker, regardless of which node was currently active in the investigation.
|
||
- corrected invariant: If the active node is among the tied structural candidates and remains eligible (unresolved, not contradicted), it is preserved as the winner. The stable label/display-order fallback is only used when there is no active candidate or when the active candidate does not remain among the top structural ties.
|
||
- regression location: `lib/graph/utils.js` — `classifyCandidateOrdering()` now accepts an `activeNodeId` parameter and checks for active-tied candidates within the leading structural set before using display-order fallback. Callers in `selectActiveUnknownCandidate()` and `explainUnknownSelection()` pass `graph.activeUnknownNodeId`.
|
||
- regression test location: `tests/graph/utils.test.js` — tests: "preserves the active candidate when it remains eligible and substantively tied", "transfers ownership when the active candidate substantively loses on score", "transfers ownership when the active candidate is resolved or ineligible"
|
||
|
||
### 2. Question-formulation rejection
|
||
- previous behaviour: When a selected candidate's question formulation was rejected (decomposition required), `determineGraphBackedQuestion` called `reseatSelectionAfterQuestionRejection` with `excludedNodeIds` that excluded the current target, causing investigation ownership to transfer to another candidate — even though the original target remained the strongest unresolved unknown.
|
||
- corrected invariant: The selected node keeps its status and selection identity. `selectedQuestion` is set to `null` and `questionSuppressedReason` records the rejection explanation. No reseating occurs.
|
||
- regression/captured fixture location: `tests/graph/orchestrator.test.js` — "retains ownership when the strongest target's formulated question is rejected"; captured replay via `tests/fixtures/live-product-launch-start-response.json`
|
||
|
||
## Current deterministic product-launch evidence
|
||
- `deterministicSelection.nodeId = ntpt9ki` (active investigation target preserved)
|
||
- `selectedQuestion = null` after rejected formulation
|
||
- explicit `noQuestionReason`: "The selected investigation target remains active, but its current graph-backed question formulation was rejected as too complex."
|
||
- `nxmeiab` is not substituted in place of ntpt9ki
|
||
|
||
## Current product meaning
|
||
- `activeUnknownNodeId` represents ongoing investigation ownership — it tracks which unknown candidate the system has committed to investigating.
|
||
- Wording/formulation failure (question complexity / decomposition-required) does not itself invalidate the target. The target remains selected even when its formulated question cannot be answered in one step.
|
||
- Stable label ordering remains only a final fallback after substantive scoring, structural comparison, semantic signature checks, and tie/ownership handling are all exhausted.
|
||
|
||
## Not yet proved
|
||
- live behavioural validation after these fixes (requires an actual product-launch run through the dev server)
|
||
- whether same-target reformulation is better than no-question (system currently uses no-question approach)
|
||
- broader/full-suite state beyond the six verified guards
|
||
- correctness of the question-complexity heuristic itself (that is a separate design concern)
|
||
|
||
## Canonical live apparatus for next validation
|
||
- `scripts/reproduce-multi-turn-investigation.mjs`
|
||
- existing healthy dev server
|
||
- `.env.local` authoritative for:
|
||
- `OLLAMA_BASE_URL`
|
||
- `OLLAMA_MODEL`
|
||
- no model discovery
|
||
- no supplementary harnesses
|
||
- no direct Ollama calls
|
||
|
||
## Next recommended step
|
||
- one observation-only live product-launch validation using the fixed scenario from the recorded journey
|
||
- no production changes during the experiment
|
||
- verify that the live LLM responds consistently to the null-question state and continues investigation on ntpt9ki
|
||
|
||
## Apparatus correction: 60B.101 — null-question start capture
|
||
|
||
The canonical `startOnly` harness was corrected to accept successful Start responses with `selectedQuestion = null`. Previously, any successful Start returning no graph-backed question (legitimate outcome meaning "target remains active but no askable question available") caused the harness to block and fail.
|
||
|
||
**Change:** The harness now checks `success === true` + valid `situationGraph` as the sole gate for startOnly success. `selectedQuestion` is preserved exactly (including null) in the continuation state file without coercion.
|
||
|
||
**Impact on 60B.100:** The evidence from 60B.100 was captured via direct curl because the harness blocked on null-question Start. That evidence is now marked as apparatus-contaminated and provisional observation only.
|
||
|
||
---
|
||
|
||
## Canonical harness gated apparatus (60B.99)
|
||
|
||
The canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`) now supports a two-phase gated investigation pattern:
|
||
|
||
**startOnly** — `FIXTURE_MODE=startOnly`
|
||
- Makes exactly one `/api/cases/start` request
|
||
- Writes the captured Start state (graph + selectedQuestion) to `.evidence-temp/continuation-start-only.json` (or path set by `CONTINUATION_FILE`)
|
||
- Issues zero Update requests
|
||
- Exits successfully
|
||
|
||
**continueOneUpdate** — `FIXTURE_MODE=continueOneUpdate CONTINUATION_ANSWER=<answer>`
|
||
- Loads the persisted Start continuation state
|
||
- Requires explicit answer (blocks with exit code 1 if missing)
|
||
- Makes exactly one `/api/cases/update` using preserved Start state + explicit answer
|
||
- Issues zero Start requests
|
||
- Exits
|
||
|
||
**Normal mode** (`FIXTURE_MODE` unset) — unchanged. Start → configured Update loop still works identically to pre-60B.99.
|
||
|
||
This apparatus corrects the apparatus defect proven in 60B.98: the canonical harness can now stop after Start, allow external semantic inspection of the returned question, and later continue from that exact captured state with an explicitly chosen answer.
|
||
|
||
---
|
||
|
||
## Experiment 60B.95 result (2026-08-17)
|
||
|
||
**Classification: E — LIVE PATH DIVERGED**
|
||
|
||
The live model selected nk6eyn2 ("exact monetary value of potential enterprise contract relative to £300k launch cost") as the investigation target, not npzfx36 ("likelihood, negotiation stage, and targeted signing date for the large enterprise customer"). Both are unresolved unknowns in the same scenario. An acceptable question was produced ("What outcome would demonstrate enough value to justify launching a software product now?"), so the question-rejection boundary was not reached.
|
||
|
||
**What this establishes:** The live engine can produce an acceptable graph-backed question on a fresh product-launch start without requiring decomposition.
|
||
|
||
**What this does NOT prove:** Whether investigation ownership is preserved when a selected target's formulation is rejected (the core invariant from checkpoint 60B.93). The question-rejection boundary was not reached because the live model chose a different investigation target with an acceptable question path.
|
||
|
||
|
||
## Experiment 60B.97 result (2026-08-18)
|
||
|
||
**Classification: E — START PATH DIVERGED**
|
||
|
||
The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison.
|
||
|
||
**Valid evidence retained:** Start = S2 — DIFFERENT START (the live model diverged from the expected financial-comparison path).
|
||
|
||
**Update 1 evidence: DISCARDED.** The canonical harness auto-continued with its preconfigured `answers[0]`, so the Update occurred outside the experiment's semantic gate. This was an apparatus defect (60B.98) — the harness did not provide a post-Start stop gate at that time. The HTTP 500 is NOT established as a reasoning defect from 60B.97.
|
||
|
||
**Apparatus correction:** See section "Canonical harness gated apparatus (60B.99)" above for the fix.
|
||
|
||
|
||
## Experiment 60B.100 result (2026-08-18)
|
||
|
||
**Classification: B — DETERMINISTIC SELECTOR OVERRIDES MODEL QUESTION**
|
||
|
||
On a fresh product-launch Start, the LLM reconstruction question targeted one uncertainty ("What is the estimated probability that the large enterprise customer will sign?") while the deterministic graph-backed selector chose another ("What evidence would clarify the exact percentage of total projected revenue attributable to the enterprise customer?"). These are materially different: one asks about deal timing/commitment probability, the other asks about financial proportion/magnitude.
|
||
|
||
The override was produced by fixed `actor_match` keyword scoring: node n65sgyd's label contained "enterprise customer" which matched the actor dictionary (+10 delta), giving it a decisive score of 10 vs 4 for both competitors. No tie/fallback was involved — the winner was determined entirely by keyword rule weighting.
|
||
|
||
**What this establishes:** On fresh Start calls, deterministic keyword signals can override model-inferred investigation priority when node labels differ in dictionary-match patterns. The final investigation target is not the model's contextual judgment but the highest-scoring candidate under fixed scoring rules.
|
||
|
||
**What this does NOT prove:** Whether the deterministic selection is better or worse than the model's suggestion; consistency across scenario types; or downstream investigation quality impact.
|
||
|
||
|
||
---
|
||
|
||
## RETURN-TO-ORIGIN CHECKPOINT
|
||
|
||
**selector-led compulsory next-question optimisation is PAUSED**
|
||
|
||
**semantic-selector replacement is also PAUSED**
|
||
|
||
Recent work is preserved as valuable experimental learning. The graph/reconstruction/decomposition/invariant work remains potentially reusable. No replacement architecture has been selected.
|
||
|
||
The next phase starts from the workspace/methodology question, not from a preselected technical solution.
|
||
|
||
### Required methodology-continuity context (read first)
|
||
|
||
> **Before continuing any Confidence Engine reasoning development, load these in order:**
|
||
>
|
||
> 1. `docs/current-working-principles.md` §0 (Axiomatic Principles A1–A12)
|
||
> 2. `docs/Confidence_Engine_Return_to_Origin_Methodology_Context_2026-08-18.md`
|
||
|
||
**The axiomatic principles in current-working-principles.md are now the durable methodological anchor.** They codify delivery-platform independence, meaning-over-dictionary reasoning, user-directed non-linear investigation, provisional closure, and progressive disclosure. All future work must be evaluated against these 12 axioms.
|
||
|
||
The RTO methodology context document covers:
|
||
- What Confidence Engine is (reasoning method, not software or LLM)
|
||
- What it is not
|
||
- The granular answer-fragment hypothesis (working — not architecture)
|
||
- RTO.14–17 learning and why they did not define the next architecture
|
||
- Memory/graph separation principle
|
||
- Optional capability: "Help me answer" / "Answer for me"
|
||
- Methodology test for future development
|
||
- Recent experimental evidence (RTO.18, RTO.20–RTO.22) supporting durable principles
|
||
|
||
**Do not begin reasoning development from the latest experiment apparatus.** Always load this methodology continuity context first. Without it, a new session will reconstruct the project from recent technical experiments and lose sight of the method the application is meant to embody.
|
||
|
||
### Current Git checkpoint
|
||
|
||
- **Branch:** `feature/decision-closure-ownership-v0.47`
|
||
- **HEAD:** `e1b54e407389c7f1c25ba20645540d404b9e9df9`
|
||
- **Latest commit:** `e1b54e4 test(experiment): checkpoint granular answer fragment apparatus`
|
||
|
||
### Status of RTO.14–17
|
||
|
||
RTO.14–17 experiments proved that an LLM can retain focused knowledge and carry coherent reasoning across turns inside a single investigation. This was valuable evidence.
|
||
|
||
**However, they also revealed that the cumulative reconstruction pattern was reproduced at a lower scope.** Do not treat RTO.14–17 as the automatic architecture direction to continue. The next reasoning direction is the granular answer-fragment hypothesis: independent user-chosen question/answer deconstruction without accumulated focused state being passed between LLM calls.
|
||
|
||
**Prohibition:** Do not automatically continue into cumulative-state compression or fourth-turn experiments without first revisiting the methodology continuity document above.
|
||
|
||
See also:
|
||
- `docs/methodology-checkpoint-return-to-origin.md` — repository-facing checkpoint summary
|