From 9d670822a35d550a2708d6fb54fbd52628a57ef9 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 24 Aug 2026 10:11:06 +0100 Subject: [PATCH] feat(confidence-engine): preserve focused deconstruction semantic fidelity --- docs/current-handoff.md | 82 +++++++++++++++++++ lib/graph/focused-investigation.js | 13 +-- ...-focused-deconstruct-experiment-helper.mjs | 68 +++++++++++++++ 3 files changed, 157 insertions(+), 6 deletions(-) create mode 100644 tests/graph/live-focused-deconstruct-experiment-helper.mjs diff --git a/docs/current-handoff.md b/docs/current-handoff.md index e499494..e0af8f6 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -176,3 +176,85 @@ RTO.14–17 experiments proved that an LLM can retain focused knowledge and carr See also: - `docs/methodology-checkpoint-return-to-origin.md` — repository-facing checkpoint summary + +--- + +## Focused Semantic Deconstruction — Reasoning Status (2026-08-24) + +### Current state + +- **Branch:** `feature/decision-closure-ownership-v0.47` +- **HEAD:** `2c108df5a92ec4cb0c4fa7878002cb7eb0aea4fd` +- **Working tree:** + ``` + M lib/graph/focused-investigation.js + ?? tests/graph/live-focused-deconstruct-experiment-helper.mjs + ``` +- No unexpected temporary files or evidence directories. + +### Canonical focused helper + +``` +tests/graph/live-focused-deconstruct-experiment-helper.mjs +``` + +Exports `runLiveFocusedDeconstructExperiment`. Future focused semantic experiments should: import it directly; use the real production `buildFocusedDeconstructPrompt`; use the production validator/provider through the helper; configure `.env.local` for `OLLAMA_BASE_URL` and `OLLAMA_MODEL`; make the explicitly budgeted number of live calls; never inline prompt/provider/schema logic; never replace the helper ad hoc. + +### Product flow context + +``` +initial scenario → initial/global SituationGraph → user selects original open question/thread → focused formulate → user answer → focused deconstruct → contribution retained against original targetNodeId → subsequent focused formulate/deconstruct rounds +``` + +Focused deconstruction accumulates focused contributions rather than rewriting the initial global SituationGraph. Reuse/integration of accumulated focused learning is a later question, not part of this checkpoint. + +### Semantic rules currently under review (validated in the cumulative diff) + +The production `buildFocusedDeconstructPrompt` encodes six semantic principles: + +1. **Observation fidelity** — only meaning directly supported by what the user's answer states becomes an observation; do not strengthen implications into observations. +2. **Uncertainty fidelity** — preserve uncertainty at the narrowest scope justified by evidence; when one factor is established but nothing else is, keep remaining uncertainty broad rather than inventing specific additional factors/deficits/causes/interventions. +3. **Assumption attribution** — attribute only propositions that the user's answer would cease to make sense if false; do not import plausible interpretations from investigation context or model analysis into the assumptions field; return `assumptions: []` when no genuine assumption exists. +4. **Relationship strength** — only connections the user's answer directly establishes become relationships; co-mentioned facts do not create causal/constraint/dependency links. +5. **Proposition ownership** — track who owns each proposition (user's comfort, willingness, threshold, belief, judgement, etc.); do not elevate subjective stance into objective requirement or situational constraint. +6. **Evidence-first / action-appropriate follow-ups** — before formulating follow-ups, check whether the question tests a proposition against current epistemic state; if an explanation/deficit/cause/intervention has not been established by evidence, phrase the question to test whether that proposition is true rather than assuming it; do not jump to implementation details until intervention is the relevant next issue. + +A supporting negative principle also holds: *Do not solve semantic interpretation with an ever-growing deterministic dictionary/regex system. Structure may be deterministic; meaning remains semantic/LLM-based.* + +### Validated learning (bounded evidence) + +Bounded focused experiments on this branch showed: + +- The original real UI failure was reproduced on the actual focused-deconstruct path. +- Observations could remain faithful while assumptions/relationships were weakened beyond what the user stated. +- Tightening the assumptions contract materially reduced model-generated assumptions being attributed to the user, supported by bounded contrast. +- Genuine implicit dependencies can still be preserved through the tightened boundary. +- Proposition ownership preserves subjective criteria without weakening genuine factual reports. +- Tentative factual reports can retain their uncertainty. +- Evidence-first follow-ups avoid premature remedy; practical follow-ups remain possible once a blocker is established. +- Uncertainty specificity follows user-supplied evidence rather than defaulting to generic broadness. + +**Do not treat these as generalisation:** each finding is supported by the bounded contrast of tested cases only. + +### EXP13 status: CONTAMINATED + +``` +EXP13 - CONTAMINATED — excluded from evidence +``` + +Reasons: iterative production prompt refinement; best-supported live-call count = 4; not valid bounded experiment evidence; tuned EXP13 uncertainty wording was removed during the diff. EXP13 must not be cited as proof of uncertainty behaviour. + +### EXP13R / EXP14 (valid but conservative) + +- **EXP13R:** materially reduced invented uncertainty specificity but retained some residual operational specialisation. +- **EXP14:** showed the same rule could preserve explicit user-supplied BACS/failed-payment specificity while leaving unidentified additional gaps broad. + +Do not conclude that the uncertainty boundary is fully solved. The contrast between these two cases does not establish generalisation. + +### Current restart point + +The immediate semantic pass has established a coherent working principle: + +> **The Engine should preserve what was said, how certain it was, who owns the proposition, and how strongly evidence supports relationships; then investigate missing evidence before inventing explanations or remedies.** + +Deterministic tests (22/22) verify the structural boundaries. Live-call evidence is bounded. No new experiment is run in this checkpoint. diff --git a/lib/graph/focused-investigation.js b/lib/graph/focused-investigation.js index e349092..81c8499 100644 --- a/lib/graph/focused-investigation.js +++ b/lib/graph/focused-investigation.js @@ -94,13 +94,14 @@ Required top-level fields: - relationships - possibleFollowUpQuestions -Field rules: +Field rules (semantic contract): - targetNodeId must be included as a string identifying this investigation node -- observations: only statements directly supported by the answer -- uncertainties: only things the answer explicitly leaves unknown or unclear -- assumptions: include only if the answer itself relies on an assumption -- relationships: only direct supported relationships among extracted items, each with { from, to, type, rationale } -- possibleFollowUpQuestions: unresolved questions genuinely exposed by this answer, unranked +- observations: only meaning directly supported by what the user's answer states. Do not strengthen implications into observations. +- uncertainties: only things the answer explicitly leaves unknown or unclear. Preserve uncertainty at the narrowest scope justified by the answer: when the answer establishes one factor but provides no evidence about what else may matter, keep the remaining uncertainty broad rather than inventing specific additional factors, deficits, causes, requirements, or interventions. +- assumptions: what unstated proposition does the user's answer itself rely upon for it to make sense? Include only when such a proposition is genuinely attributable to the user's reasoning. The boundary is narrow: attribute only propositions that the user's answer would cease to make sense if they were false. Do NOT import plausible interpretations from the wider investigation context, scenario framing, domain relevance, strategic implications, or model-generated analysis into this field — those belong in uncertainties, relationships (where permitted), or possibleFollowUpQuestions. Do NOT connect a factual statement the user makes to a broader capability or constraint concept unless the user explicitly links them. Example: answering "I only have bank account access" to a question about delegation constraints does NOT assume that "delegation feasibility is contingent upon banking access" — it only states a fact about access, and connecting that fact to delegation feasibility is your own scenario-level inference, not a user-held assumption. If the user's answer does not contain or rely upon an identifiable assumption, return assumptions: []. Do NOT require verbatim copying from the user's answer; paraphrasing is allowed only when the reasoning genuinely relies on it. +- relationships: only connections that the user's answer directly establishes between items. Co-mentioned facts do not by themselves create causal, constraint, or dependency relationships. If a relationship is only plausible, omit it rather than assert it. +- possibleFollowUpQuestions: questions that investigate genuinely unresolved areas exposed by this answer. Before formulating each follow-up, check whether the question tests a proposition (e.g., "there is a deficit", "X is required", "intervention Y should happen") against the current epistemic state or assumes it as already established. If an explanation, deficit, dependency, cause, intervention, recommendation, or solution has not been established by prior evidence, phrase the question so it tests whether that proposition is true rather than assuming it — verify the unresolved fact before seeking remedy. Prefer questions that identify what remains unknown, distinguish competing explanations, test whether a suspected factor actually matters, clarify scope, or identify what evidence would change the investigation. Do not jump to implementation details unless the answer has already established that intervention as the relevant next issue. +- cross-field ownership: preserve who or what owns each proposition. When a statement expresses the user's comfort, willingness, threshold, belief, uncertainty, preference, or judgement, keep it attached to that stance — do not elevate it into an objective requirement, capability fact, or situational constraint. Focused case context: - target label: ${targetLabel} diff --git a/tests/graph/live-focused-deconstruct-experiment-helper.mjs b/tests/graph/live-focused-deconstruct-experiment-helper.mjs new file mode 100644 index 0000000..8421051 --- /dev/null +++ b/tests/graph/live-focused-deconstruct-experiment-helper.mjs @@ -0,0 +1,68 @@ +/** + * Reusable live-focused-deconstruction experiment helper. + * + * Makes one real focused-deconstruction call using the same production + * semantic operation as /api/focused-investigation/deconstruct: + * + * buildFocusedDeconstructPrompt -> provider.generateReconstruction -> validateFocusedDeconstructSchema + * + * Inputs are the minimum fields required by buildFocusedDeconstructPrompt. + * Relies on environment variables OLLAMA_BASE_URL and OLLAMA_MODEL being set. + */ + +import { buildFocusedDeconstructPrompt, validateFocusedDeconstructSchema } from "@/lib/graph/focused-investigation.js"; +import { getProvider } from "@/lib/llm/provider.js"; + +/** + * Run one live focused-deconstruction experiment. + * + * @param {object} params + * @param {string} params.targetNodeId - the target node ID under investigation + * @param {string} params.targetLabel - label of the target node + * @param {string} params.targetDescription - description of the target node + * @param {string} params.centralStatement - the case's central statement + * @param {string} params.question - the exact real production question + * @param {string} params.answer - the exact real production answer + * @param {object} [params.provider] - optional injected provider (for test isolation) + * @returns {object} validated result + timing details + */ +export async function runLiveFocusedDeconstructExperiment(params) { + const { targetNodeId, targetLabel, targetDescription, centralStatement, question, answer, provider } = params; + + // Production path: real prompt builder (never copies prompt logic) + const prompt = buildFocusedDeconstructPrompt({ + targetLabel, targetDescription, centralStatement, question, answer, + }); + + // Provider: use injected (test) or real (production) + const actualProvider = provider ?? getProvider(); + + const ollamaModel = process.env.OLLAMA_MODEL; + if (!ollamaModel) throw new Error("OLLAMA_MODEL is not set in environment"); + + const startedAt = Date.now(); + const raw = await actualProvider.generateReconstruction(prompt, ollamaModel); + const elapsedMs = Date.now() - startedAt; + + // Production path: real schema validator (never copies schema logic) + const validationErrors = validateFocusedDeconstructSchema(raw); + if (validationErrors.length > 0) { + throw new Error( + "Focused deconstruction result did not match expected schema:\n" + + validationErrors.map((e) => " - " + e).join("\n") + ); + } + + return { + success: true, + targetNodeId, + observations: raw.observations, + uncertainties: raw.uncertainties, + assumptions: raw.assumptions, + relationships: raw.relationships, + possibleFollowUpQuestions: raw.possibleFollowUpQuestions, + elapsedMs, + }; +} + +export { buildFocusedDeconstructPrompt, validateFocusedDeconstructSchema };