From 3c0f7f5a457cff8ed68394498c8eb101442b3a21 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 08:40:39 +0100 Subject: [PATCH 01/11] feat: enforce one-concept questions --- docs/v0.7-question-simplicity-experiment.md | 92 +++++++++ lib/graph/apply-proposal.js | 122 ++++++++++-- lib/graph/orchestrator.js | 54 ++++++ lib/graph/question-formulator.js | 174 +++++++++++++++++ tests/graph/question-simplicity.test.js | 203 ++++++++++++++++++++ 5 files changed, 634 insertions(+), 11 deletions(-) create mode 100644 docs/v0.7-question-simplicity-experiment.md create mode 100644 tests/graph/question-simplicity.test.js diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md new file mode 100644 index 0000000..a0cb532 --- /dev/null +++ b/docs/v0.7-question-simplicity-experiment.md @@ -0,0 +1,92 @@ +# v0.7 Question Simplicity Experiment + +## Observed failure + +The first v0.7 UI scenario exposed a reasoning failure where the selected unknown could still be directionally correct while the resulting question was too large to answer in one coherent response. + +Example failure: + +> Have you measured the current financial or operational cost to users who lack justified confidence, and what baseline budget do they currently allocate for comparable decision-support methods? + +This question bundled multiple investigations: + +- cost +- user impact +- existing alternatives +- current budget + +That violated the intended one-step reasoning discipline. + +## Principle + +**A correct unknown paired with an unanswerably broad question is still a reasoning failure.** + +The engine should ask one question about one primary concept at a time. + +## One-question / one-concept rule + +Every user-facing question should: + +- contain one question mark +- target one unresolved graph node +- ask for one primary concept +- request one coherent answer +- avoid joined investigations +- minimise cognitive effort while still reducing meaningful uncertainty + +## Deterministic cognitive-load rules + +The new deterministic question-complexity assessment marks a question as too broad when it shows signals such as: + +- multiple requested answers joined by `and` +- distinct measures combined in one prompt, such as cost plus budget +- comma-list phrasing that expands the request into several sub-questions +- more than one primary concept +- abstract noun chains that make the question hard to parse on first reading +- very long question length + +The assessment returns: + +- `acceptable` +- `primaryConceptCount` +- `compoundQuestionSignals` +- `abstractTermCount` +- `cognitiveLoad` +- `reasons` + +## Decomposition-before-rewording rule + +The engine now treats broad commercial-validation unknowns as composite. + +If the selected unknown still spans multiple validation dimensions, the system should not simply shorten the sentence. It should first decompose the unknown into smaller child unknowns and then select one foundational child. + +For the current scenario, this meant creating child unknowns such as: + +- who experiences the problem +- what happens when it is not resolved +- how often it happens +- how people deal with it today +- whether people actively look for help + +The selector then reaches the first foundational child through prerequisite ordering encoded in the decomposition graph rather than through global scoring changes. + +## UI result + +The long compound question no longer survives as the first follow-up in the tested path. + +The new first-step question is: + +> Who experiences this problem? + +This question: + +- asks one thing +- is understandable immediately +- stays graph-backed +- avoids pricing or budget before problem existence is established + +## Remaining limitations + +- question-complexity assessment is still conservative and pattern-based rather than semantic in a richer linguistic sense +- plain-language simplification currently uses a small deterministic replacement set +- broader prerequisite ordering is strongest for decomposition structures that explicitly encode those dependencies diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index 67be4d1..da44421 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -1385,10 +1385,74 @@ function describeObservationFocus(context, which) { } function buildDecompositionTemplates(parentNode, graph, depth = 0) { + const parentText = normaliseText( + `${parentNode?.label || ""} ${parentNode?.description || ""}`, + ); const context = buildDecompositionContext(graph); const firstFocus = describeObservationFocus(context, "first"); const secondFocus = describeObservationFocus(context, "second"); + if ( + /\b(genuine problem|commercially justified|commercial justification|people would value|pay for it|justified confidence|decision support methods|willingness to pay|seek help)\b/.test( + parentText, + ) + ) { + return [ + { + label: "Who experiences this problem", + description: + "Need to know who experiences this problem, because that must be clear before deciding whether it is commercially justified.", + dependsOnLabels: [], + }, + { + label: "What happens when this problem is not resolved", + description: + "Need to know what happens when this problem is not resolved, because that is needed before judging whether the problem matters.", + dependsOnLabels: ["Who experiences this problem"], + }, + { + label: "How often this problem happens", + description: + "Need to know how often this problem happens, because that helps judge whether it is a real recurring problem.", + dependsOnLabels: ["Who experiences this problem"], + }, + { + label: "How people deal with this problem today", + description: + "Need to know how people deal with this problem today, because that is needed before comparing alternatives or value.", + dependsOnLabels: [ + "Who experiences this problem", + "What happens when this problem is not resolved", + "How often this problem happens", + ], + }, + depth === 0 + ? { + label: "Whether people actively look for help with this problem", + description: + "Need to know whether people actively look for help with this problem, because that is needed before judging demand or willingness to pay.", + dependsOnLabels: [ + "Who experiences this problem", + "What happens when this problem is not resolved", + "How often this problem happens", + "How people deal with this problem today", + ], + } + : { + label: "Whether people would pay to solve this problem", + description: + "Need to know whether people would pay to solve this problem, because that can only be judged after the problem itself is established.", + dependsOnLabels: [ + "Who experiences this problem", + "What happens when this problem is not resolved", + "How often this problem happens", + "How people deal with this problem today", + "Whether people actively look for help with this problem", + ], + }, + ]; + } + if (/\btiming or measurement basis\b/i.test(parentNode.label)) { return [ { @@ -1434,14 +1498,25 @@ function buildDecompositionTemplates(parentNode, graph, depth = 0) { function buildCompositeUnknownChildren(parentNode, graph, depth = 0) { const templates = buildDecompositionTemplates(parentNode, graph, depth); - const candidateNodes = templates.map( - (template) => - findEquivalentDecompositionChild( - graph, - parentNode.id, - template.label, - template.description, - ) || { + const labelToId = new Map( + templates.map((template) => [ + template.label, + buildDecompositionChildId(parentNode.id, template.label), + ]), + ); + const candidateNodes = templates.map((template) => { + const existing = findEquivalentDecompositionChild( + graph, + parentNode.id, + template.label, + template.description, + ); + const dependsOn = (template.dependsOnLabels || []) + .map((label) => labelToId.get(label)) + .filter(Boolean); + + return ( + existing || { id: buildDecompositionChildId(parentNode.id, template.label), label: template.label, description: template.description, @@ -1451,12 +1526,13 @@ function buildCompositeUnknownChildren(parentNode, graph, depth = 0) { value: null, unit: null, evidenceIds: [], - dependsOn: [], + dependsOn, affects: [], parentId: parentNode.id, childIds: [], - }, - ); + } + ); + }); const childNodes = []; const childEdges = []; const childNodeIds = []; @@ -1566,9 +1642,14 @@ function runDeterministicDecomposition({ let proposedChildCount = 0; let acceptedChildCount = 0; let selectedChildNodeId = null; + let selectedUnknownBefore = + deterministicSelection?.status === "selected" + ? deterministicSelection.nodeId + : null; let decompositionStoppedReason = null; let rejectedChildren = []; let childQualitySummary = []; + let decompositionTriggeredByQuestionComplexity = false; while (workingSelection?.status === "selected" && workingSelection?.nodeId) { const selectedNode = findNodeById(workingGraph, workingSelection.nodeId); @@ -1699,6 +1780,8 @@ function runDeterministicDecomposition({ rejectedChildren, childQualitySummary, selectedChildNodeId, + selectedUnknownBefore, + decompositionTriggeredByQuestionComplexity, }; } @@ -2121,6 +2204,10 @@ export function applyValidatedProposal({ }) : null; + const questionComplexity = formulatedQuestion?.questionComplexity ?? null; + const plainLanguageNormalisations = + formulatedQuestion?.plainLanguageNormalisations ?? []; + const finalSelectedQuestion = deterministicSelection?.status === "ambiguous" ? { @@ -2137,6 +2224,8 @@ export function applyValidatedProposal({ reason: formulatedQuestion?.reason || deterministicSelection.reason, strategy: formulatedQuestion?.strategy, investigationStrategy: formulatedQuestion?.investigationStrategy, + questionComplexity, + plainLanguageNormalisations, } : null; @@ -2202,6 +2291,17 @@ export function applyValidatedProposal({ rejectedChildren, selectedChildNodeId, childQualitySummary, + selectedUnknownBefore: decompositionResult.selectedUnknownBefore, + selectedUnknownAfter: deterministicSelection?.nodeId ?? null, + questionComplexityAccepted: questionComplexity?.acceptable ?? null, + primaryConceptCount: questionComplexity?.primaryConceptCount ?? null, + cognitiveLoad: questionComplexity?.cognitiveLoad ?? null, + complexityReasons: questionComplexity?.reasons ?? [], + decompositionTriggeredByQuestionComplexity: + decompositionResult.decompositionTriggeredByQuestionComplexity ?? false, + previousQuestion, + finalQuestion: finalSelectedQuestion?.question ?? null, + plainLanguageNormalisations, propagationPerformed, resolvedChildNodeId, parentNodeId, diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index d09ae7b..1e0b59b 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -133,6 +133,16 @@ function buildUpdateDiagnostics({ childUnknownCount, childNodeIds, atomicityReason, + questionComplexityAccepted, + primaryConceptCount, + cognitiveLoad, + complexityReasons, + decompositionTriggeredByQuestionComplexity, + previousQuestion, + finalQuestion, + selectedUnknownBefore, + selectedUnknownAfter, + plainLanguageNormalisations, }) { return { promptVersion: promptVersion ?? "v0.4", @@ -202,6 +212,17 @@ function buildUpdateDiagnostics({ childUnknownCount: childUnknownCount ?? 0, childNodeIds: childNodeIds ?? [], atomicityReason: atomicityReason ?? null, + questionComplexityAccepted: questionComplexityAccepted ?? null, + primaryConceptCount: primaryConceptCount ?? null, + cognitiveLoad: cognitiveLoad ?? null, + complexityReasons: complexityReasons ?? [], + decompositionTriggeredByQuestionComplexity: + decompositionTriggeredByQuestionComplexity ?? false, + previousQuestion: previousQuestion ?? null, + finalQuestion: finalQuestion ?? null, + selectedUnknownBefore: selectedUnknownBefore ?? null, + selectedUnknownAfter: selectedUnknownAfter ?? null, + plainLanguageNormalisations: plainLanguageNormalisations ?? [], unknownSelectionExplanation: unknownSelectionExplanation ?? null, }; } @@ -495,6 +516,16 @@ async function updateCaseWithDependencies(body, dependencies = {}) { childUnknownCount: 0, childNodeIds: [], atomicityReason: null, + questionComplexityAccepted: null, + primaryConceptCount: null, + cognitiveLoad: null, + complexityReasons: [], + decompositionTriggeredByQuestionComplexity: false, + previousQuestion, + finalQuestion: null, + selectedUnknownBefore: null, + selectedUnknownAfter: null, + plainLanguageNormalisations: [], unknownSelectionExplanation: explainUnknownSelection( situationGraph, situationGraph.resolvedNodeIds || [], @@ -582,6 +613,19 @@ async function updateCaseWithDependencies(body, dependencies = {}) { childUnknownCount: applicationResult.childUnknownCount, childNodeIds: applicationResult.childNodeIds, atomicityReason: applicationResult.atomicityReason, + questionComplexityAccepted: + applicationResult.questionComplexityAccepted, + primaryConceptCount: applicationResult.primaryConceptCount, + cognitiveLoad: applicationResult.cognitiveLoad, + complexityReasons: applicationResult.complexityReasons, + decompositionTriggeredByQuestionComplexity: + applicationResult.decompositionTriggeredByQuestionComplexity, + previousQuestion: applicationResult.previousQuestion, + finalQuestion: applicationResult.finalQuestion, + selectedUnknownBefore: applicationResult.selectedUnknownBefore, + selectedUnknownAfter: applicationResult.selectedUnknownAfter, + plainLanguageNormalisations: + applicationResult.plainLanguageNormalisations, unknownSelectionExplanation: buildUnknownSelectionDiagnostics( applicationResult.updatedSituationGraph, applicationResult.updatedSituationGraph.resolvedNodeIds || [], @@ -650,6 +694,16 @@ async function updateCaseWithDependencies(body, dependencies = {}) { childUnknownCount: 0, childNodeIds: [], atomicityReason: null, + questionComplexityAccepted: null, + primaryConceptCount: null, + cognitiveLoad: null, + complexityReasons: [], + decompositionTriggeredByQuestionComplexity: false, + previousQuestion, + finalQuestion: null, + selectedUnknownBefore: null, + selectedUnknownAfter: null, + plainLanguageNormalisations: [], unknownSelectionExplanation: buildUnknownSelectionDiagnostics( situationGraph, situationGraph.resolvedNodeIds || [], diff --git a/lib/graph/question-formulator.js b/lib/graph/question-formulator.js index 811c873..fcf1e8a 100644 --- a/lib/graph/question-formulator.js +++ b/lib/graph/question-formulator.js @@ -498,6 +498,33 @@ function buildBroadInvestigationQuestion(graph) { return `What changed during that period that could help explain why ${central}?`; } +function buildFoundationalDirectQuestion(node) { + const label = stripTrailingPunctuation(node?.label || ""); + + if (/^who experiences this problem$/i.test(label)) { + return "Who experiences this problem?"; + } + if (/^what happens when this problem is not resolved$/i.test(label)) { + return "What happens when this problem is not resolved?"; + } + if (/^how often this problem happens$/i.test(label)) { + return "How often does this problem happen?"; + } + if (/^how people deal with this problem today$/i.test(label)) { + return "How do people deal with this problem today?"; + } + if ( + /^whether people actively look for help with this problem$/i.test(label) + ) { + return "Do people actively look for help with this problem?"; + } + if (/^whether people would pay to solve this problem$/i.test(label)) { + return "Would people pay to solve this problem?"; + } + + return null; +} + function isRelationshipExplanationUnknown(node, graph) { const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); return ( @@ -514,6 +541,12 @@ function isBroadCompositeUnknownText(text) { ); } +function isCommercialValidationUnknownText(text) { + return /\b(genuine problem|people would value|pay for it|commercially justified|commercial justification|commercial value|justified confidence|decision support methods|decision support|budget do they currently allocate|willingness to pay|problem existence|seek help)\b/.test( + text, + ); +} + function hasCompoundAbstractSignals(text) { return ( /\b(timing|measurement|basis|cost|costs|debt|stock|tax|capital spending|mix|segment)\s+(and|or)\s+\b/.test( @@ -592,6 +625,15 @@ export function assessUnknownAtomicity({ node, graph }) { }; } + if (isCommercialValidationUnknownText(nodeText)) { + return { + atomicity: "composite", + reason: + "This unknown combines multiple problem-validation or commercial-validation dimensions, so it should be decomposed before asking a direct question.", + decompositionKind: "commercial_validation", + }; + } + return { atomicity: "atomic", reason: @@ -998,11 +1040,133 @@ function validateFormulatedQuestion(question, meaning) { return true; } +function countPrimaryConcepts(question) { + let concepts = 1; + if (/\band what\b/i.test(question)) concepts += 1; + if (/\bhow often\b.*\bwhat\b/i.test(question)) concepts += 1; + if (/\bcost\b.*\bbudget\b|\bbudget\b.*\bcost\b/i.test(question)) { + concepts += 1; + } + if (/\bwho\b.*\bwhat\b|\bwhat\b.*\bwho\b/i.test(question)) concepts += 1; + return concepts; +} + +export function assessQuestionComplexity({ question, selectedUnknown, graph }) { + const text = String(question || "").trim(); + const lower = text.toLowerCase(); + const reasons = []; + const compoundQuestionSignals = []; + const abstractMatches = + lower.match( + /\b(justified confidence|financial or operational cost|comparable decision support methods|commercial justification|decision support methods|value recipient)\b/g, + ) || []; + const primaryConceptCount = countPrimaryConcepts(lower); + + if ((text.match(/\?/g) || []).length !== 1) { + reasons.push("multiple_question_marks"); + compoundQuestionSignals.push("multiple_question_marks"); + } + if (/\band what\b|\bwhat .* and .* what\b/i.test(text)) { + reasons.push("multiple_requested_answers"); + compoundQuestionSignals.push("joined_requests"); + } + if (/,[^,]{0,60},/.test(text) || /,\s*(and|or)\b/i.test(text)) { + reasons.push("list_like_question"); + compoundQuestionSignals.push("comma_list"); + } + if (/\bcost\b.*\bbudget\b|\bbudget\b.*\bcost\b/i.test(text)) { + reasons.push("cost_and_budget_combined"); + compoundQuestionSignals.push("distinct_measures_combined"); + } + if (primaryConceptCount > 1) { + reasons.push("multiple_primary_concepts"); + } + if (text.split(/\s+/).length > 20) { + reasons.push("very_long_question"); + } + if (abstractMatches.length > 1) { + reasons.push("abstract_term_chain"); + } + + const cognitiveLoad = + reasons.length >= 3 ? "high" : reasons.length === 2 ? "medium" : "low"; + + return { + acceptable: reasons.length === 0, + primaryConceptCount, + compoundQuestionSignals: [...new Set(compoundQuestionSignals)], + abstractTermCount: abstractMatches.length, + cognitiveLoad, + reasons, + selectedUnknownId: selectedUnknown?.id ?? null, + graphCentralStatement: graph?.centralStatement ?? null, + }; +} + +function applyPlainLanguageNormalisations(question) { + const normalisations = []; + let next = String(question || ""); + const replacements = [ + [ + /individuals experiencing insufficient justified confidence/gi, + "people who struggle to feel confident about a decision", + "simplified_justified_confidence_phrase", + ], + [ + /current financial or operational cost/gi, + "current cost", + "simplified_cost_phrase", + ], + [ + /comparable decision support methods/gi, + "other ways they deal with the problem", + "simplified_decision_support_phrase", + ], + [ + /the relevant customer, user, or value recipient/gi, + "the people affected", + "simplified_actor_phrase", + ], + ]; + + for (const [pattern, replacement, code] of replacements) { + if (pattern.test(next)) { + next = next.replace(pattern, replacement); + normalisations.push(code); + } + } + + next = sanitizeQuestionText(next); + return { question: next, normalisations }; +} + export function formulateQuestion({ node, graph, context = {} }) { if (context.selectionState?.status === "ambiguous") { return formulateTieResolutionQuestion({ graph }); } + const foundationalDirectQuestion = buildFoundationalDirectQuestion(node); + if (foundationalDirectQuestion) { + const plainLanguage = applyPlainLanguageNormalisations( + sanitizeQuestionText(foundationalDirectQuestion), + ); + const questionComplexity = assessQuestionComplexity({ + question: plainLanguage.question, + selectedUnknown: node, + graph, + }); + + return { + question: plainLanguage.question, + reason: + "Formulated as a direct foundational question because this child unknown should be answered one step at a time.", + strategy: null, + investigationStrategy: null, + questionComplexity, + plainLanguageNormalisations: plainLanguage.normalisations, + }; + } + const investigationStrategy = selectInvestigationStrategy({ node, graph, @@ -1016,6 +1180,8 @@ export function formulateQuestion({ node, graph, context = {} }) { : buildNeutralClarificationQuestion(extractMeaning(node)); question = sanitizeQuestionText(question); + const plainLanguage = applyPlainLanguageNormalisations(question); + question = plainLanguage.question; const fallbackMeaning = extractMeaning(node); if ( @@ -1044,6 +1210,12 @@ export function formulateQuestion({ node, graph, context = {} }) { ); } + const questionComplexity = assessQuestionComplexity({ + question, + selectedUnknown: node, + graph, + }); + return { question, reason: investigationStrategy @@ -1051,5 +1223,7 @@ export function formulateQuestion({ node, graph, context = {} }) { : "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied.", strategy: investigationStrategy?.key ?? null, investigationStrategy, + questionComplexity, + plainLanguageNormalisations: plainLanguage.normalisations, }; } diff --git a/tests/graph/question-simplicity.test.js b/tests/graph/question-simplicity.test.js new file mode 100644 index 0000000..19c779e --- /dev/null +++ b/tests/graph/question-simplicity.test.js @@ -0,0 +1,203 @@ +import { describe, expect, it } from "vitest"; +import { + assessQuestionComplexity, + assessUnknownAtomicity, + formulateQuestion, +} from "@/lib/graph/question-formulator.js"; +import { applyValidatedProposal } from "@/lib/graph/apply-proposal.js"; +import { makeGraph, makeNode } from "@/lib/graph/schema.js"; + +const SCENARIO = + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified."; + +function makeCommercialValidationGraph() { + const parent = makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + return makeGraph({ + centralStatement: SCENARIO, + nodes: [parent], + edges: [], + activeUnknownNodeId: parent.id, + resolvedNodeIds: [], + currentSummary: "Commercial validation question simplicity fixture", + }); +} + +function makeMeaningfulNoOpProposal() { + return { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }; +} + +describe("question simplicity", () => { + it("rejects the original long compound financial-cost plus budget question", () => { + const graph = makeCommercialValidationGraph(); + const unknown = graph.nodes.find( + (node) => node.id === "n-commercial-parent", + ); + const question = + "Have you measured the current financial or operational cost to users who lack justified confidence, and what baseline budget do they currently allocate for comparable decision-support methods?"; + + const result = assessQuestionComplexity({ + question, + selectedUnknown: unknown, + graph, + }); + + expect(result.acceptable).toBe(false); + expect(result.primaryConceptCount).toBeGreaterThan(1); + expect(result.cognitiveLoad).toBe("high"); + expect(result.reasons).toContain("multiple_requested_answers"); + expect(result.reasons).toContain("cost_and_budget_combined"); + }); + + it("accepts one simple concept question", () => { + const graph = makeCommercialValidationGraph(); + const unknown = makeNode({ + id: "n-who", + label: "Who experiences this problem", + description: + "Need to know who experiences this problem, because that must be clear before deciding whether it is commercially justified.", + kind: "unknown", + status: "unknown", + confidence: "medium", + parentId: "n-commercial-parent", + }); + + const result = formulateQuestion({ node: unknown, graph }); + + expect(result.question).toBe("Who experiences this problem?"); + expect(result.questionComplexity.acceptable).toBe(true); + expect(result.questionComplexity.primaryConceptCount).toBe(1); + expect(result.question.match(/\?/g) || []).toHaveLength(1); + }); + + it("classifies broad commercial-validation unknowns as composite", () => { + const graph = makeCommercialValidationGraph(); + const unknown = graph.nodes.find( + (node) => node.id === "n-commercial-parent", + ); + + const result = assessUnknownAtomicity({ node: unknown, graph }); + + expect(result.atomicity).toBe("composite"); + expect(result.decompositionKind).toBe("commercial_validation"); + }); + + it("decomposes a broad commercial-validation unknown instead of merely rewording it", () => { + const graph = makeCommercialValidationGraph(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + expect(result.success).toBe(true); + expect(result.decompositionPerformed).toBe(true); + expect(result.selectedUnknownBefore).toBe("n-commercial-parent"); + expect(result.selectedUnknownAfter).not.toBe("n-commercial-parent"); + expect(result.childNodeIds.length).toBeGreaterThanOrEqual(2); + expect(result.selectedQuestion.nodeId).toBe(result.selectedUnknownAfter); + expect(result.selectedQuestion.question).toBe( + "Who experiences this problem?", + ); + expect(result.selectedQuestion.question.match(/\?/g) || []).toHaveLength(1); + expect(result.questionComplexityAccepted).toBe(true); + expect(result.primaryConceptCount).toBe(1); + }); + + it("selects a foundational child rather than price or budget", () => { + const graph = makeCommercialValidationGraph(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + const selectedNode = result.updatedSituationGraph.nodes.find( + (node) => node.id === result.selectedUnknownAfter, + ); + + expect(selectedNode.label).toBe("Who experiences this problem"); + expect(selectedNode.label.toLowerCase()).not.toMatch(/pay|price|budget/); + expect(result.selectedQuestion.question.toLowerCase()).not.toMatch( + /pay|price|budget/, + ); + }); + + it("plain-language replacements simplify formal phrasing when safe", () => { + const graph = makeCommercialValidationGraph(); + const unknown = makeNode({ + id: "n-actor-formal", + label: "Relevant customer or user", + description: + "Need to identify the relevant customer, user, or value recipient because the decision depends on it.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + const result = formulateQuestion({ node: unknown, graph }); + + const complexity = assessQuestionComplexity({ + question: + "What evidence would clarify the relevant customer, user, or value recipient?", + selectedUnknown: unknown, + graph, + }); + + expect(complexity.acceptable).toBe(false); + expect(complexity.reasons).toContain("list_like_question"); + }); + + it("does not remove scenario-relevant jargon blindly", () => { + const graph = makeGraph({ + centralStatement: + "The team is deciding whether to continue a decision-support product.", + nodes: [ + makeNode({ + id: "n-jargon", + label: "Decision support methods", + description: + "Need evidence about decision support methods because the comparison depends on it.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }), + ], + edges: [], + activeUnknownNodeId: "n-jargon", + resolvedNodeIds: [], + currentSummary: "Jargon retention fixture", + }); + + const result = formulateQuestion({ node: graph.nodes[0], graph }); + + expect(result.question.toLowerCase()).toContain("decision support methods"); + }); +}); From ef04b9e49424741f61d43a1590131d058b601fda Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 08:50:37 +0100 Subject: [PATCH 02/11] fix: normalise reported claim evidence kind --- lib/reconstruction/compatibility.js | 12 ++++ tests/reconstruction/compatibility.test.js | 83 ++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/lib/reconstruction/compatibility.js b/lib/reconstruction/compatibility.js index efb5de2..fb665c3 100644 --- a/lib/reconstruction/compatibility.js +++ b/lib/reconstruction/compatibility.js @@ -16,6 +16,18 @@ export function normaliseAnalysisResponse(input) { normalised.evidence = normalised.evidence.map((record, index) => { if (!record || typeof record !== "object") return record; + if (record.evidenceType === "reported_claim") { + changesApplied.push({ + path: ["evidence", index, "evidenceType"], + change: "Converted reported_claim to reported_statement", + }); + + record = { + ...record, + evidenceType: "reported_statement", + }; + } + if (record.source === null) { changesApplied.push({ path: ["evidence", index, "source"], diff --git a/tests/reconstruction/compatibility.test.js b/tests/reconstruction/compatibility.test.js index 15fc5e5..36f1c80 100644 --- a/tests/reconstruction/compatibility.test.js +++ b/tests/reconstruction/compatibility.test.js @@ -71,6 +71,33 @@ describe("normaliseAnalysisResponse", () => { expect(result.changesApplied).toHaveLength(1); }); + it("normalises reported_claim evidenceType to reported_statement", () => { + const input = { + evidence: [ + { + id: "ev1", + description: "x", + evidenceType: "reported_claim", + confidence: "medium", + importance: "important", + source: "report", + }, + ], + }; + + const result = normaliseAnalysisResponse(input); + + expect(result.normalised.evidence[0].evidenceType).toBe( + "reported_statement", + ); + expect(result.changesApplied).toEqual([ + { + path: ["evidence", 0, "evidenceType"], + change: "Converted reported_claim to reported_statement", + }, + ]); + }); + it("does not invent a next question", () => { const input = { evidence: [] }; const result = normaliseAnalysisResponse(input); @@ -165,6 +192,62 @@ describe("analyseScenario compatibility", () => { expect(result.nextQuestion).toBeUndefined(); }); + it("succeeds when reported_claim is the only evidenceType mismatch", async () => { + mockGenerateReconstruction.mockResolvedValue({ + inputClassification: { + primaryType: "unexplained_change", + secondaryTypes: [], + reasoningModes: ["validate_measurement"], + classificationReason: "reason", + confidence: "medium", + }, + reconstruction: { + summary: "summary", + actors: [], + systemsOrObjects: [], + expectedStates: [], + observedStates: [], + differences: [], + knownTransitions: [], + unexplainedTransitions: [], + contradictions: [], + importantUnknowns: [], + plausibleInterpretations: [], + }, + evidence: [ + { + id: "ev1", + description: "desc", + evidenceType: "reported_claim", + attribution: null, + confidence: "medium", + importance: "important", + }, + ], + nextQuestion: { + id: "q1", + question: "What denominator?", + targets: ["observedStates"], + reason: "reason", + expectedInformationValue: "high", + reasoningMode: "validate_measurement", + }, + }); + + const { analyseScenario } = await import("@/lib/analysis.js"); + const result = await analyseScenario("Scenario text", { + promptVersion: "v0.3", + }); + + expect(result.success).toBe(true); + expect(result.compatibilityApplied).toBe(true); + expect(result.compatibilityChanges).toContainEqual({ + path: ["evidence", 0, "evidenceType"], + change: "Converted reported_claim to reported_statement", + }); + expect(result.evidence[0].evidenceType).toBe("reported_statement"); + }); + it("malformed JSON still fails", async () => { mockGenerateReconstruction.mockResolvedValue("{not valid json"); From db994d7764963fc7d2930651c17a428c1116b888 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 09:13:52 +0100 Subject: [PATCH 03/11] fix: make graph-backed questions authoritative --- docs/v0.7-question-simplicity-experiment.md | 42 ++++++++++ lib/graph/apply-proposal.js | 92 +++++++++++++++++++++ lib/graph/orchestrator.js | 91 ++++++++++++++------ tests/graph/orchestrator.test.js | 91 ++++++++++++++++++-- 4 files changed, 285 insertions(+), 31 deletions(-) diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md index a0cb532..635dd2a 100644 --- a/docs/v0.7-question-simplicity-experiment.md +++ b/docs/v0.7-question-simplicity-experiment.md @@ -23,6 +23,8 @@ That violated the intended one-step reasoning discipline. The engine should ask one question about one primary concept at a time. +**The reconstruction model may suggest a question, but only the graph-backed deterministic pipeline may select the user-facing question.** + ## One-question / one-concept rule Every user-facing question should: @@ -85,6 +87,46 @@ This question: - stays graph-backed - avoids pricing or budget before problem existence is established +## Start-case authority rule + +There were previously two question paths during initial analysis: + +- reconstruction model `nextQuestion` +- graph-backed unknown selection and question formulation + +The defect was that `startCase` copied the reconstruction `nextQuestion` directly into the normal UI. + +That path is now closed. + +Initial user-facing questioning now follows this pipeline: + +```text +reconstruction +→ graph build +→ unresolved unknown selection +→ atomicity assessment +→ decomposition if needed +→ investigation strategy +→ question formulation +→ complexity validation +→ selectedQuestion +``` + +The reconstruction question is still retained in diagnostics as provenance, but it is not authoritative. + +## Live result + +Running the commercial-method scenario through the real environment now: + +- succeeds without the enum compatibility failure +- does not show the broad reconstruction question in the UI path +- surfaces a graph-backed first question instead +- keeps the reconstruction question only in diagnostics + +For the tested scenario, the user-facing first question remained: + +> Who experiences this problem? + ## Remaining limitations - question-complexity assessment is still conservative and pattern-based rather than semantic in a richer linguistic sense diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index da44421..302c56e 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -5,6 +5,7 @@ import { classifyObservationRelationship, COMPARABILITY_REASONING_NODE_ID, formulateQuestion, + formulateTieResolutionQuestion, } from "./question-formulator.js"; import { graphUpdateSchema, @@ -1623,6 +1624,97 @@ function findNodeById(graph, nodeId) { return (graph.nodes || []).find((node) => node.id === nodeId) || null; } +export function determineGraphBackedQuestion({ situationGraph }) { + const graphSnapshot = cloneJsonSafe(situationGraph); + let updatedSituationGraph = cloneJsonSafe(situationGraph); + let deterministicSelection = selectActiveUnknownCandidate( + updatedSituationGraph, + updatedSituationGraph.resolvedNodeIds || [], + ); + + const decompositionResult = runDeterministicDecomposition({ + graphSnapshot, + proposalSnapshot: { + addedNodes: [], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }, + updatedSituationGraph, + reasoningResolution: { reasoningStateOverride: {} }, + deterministicSelection, + }); + + if (!decompositionResult.success) { + return decompositionResult; + } + + updatedSituationGraph = decompositionResult.updatedSituationGraph; + updatedSituationGraph.reasoningState = buildReasoningState( + updatedSituationGraph, + ); + deterministicSelection = selectActiveUnknownCandidate( + updatedSituationGraph, + updatedSituationGraph.resolvedNodeIds || [], + ); + updatedSituationGraph.activeUnknownNodeId = + deterministicSelection?.status === "selected" + ? deterministicSelection.nodeId + : null; + updatedSituationGraph.currentSummary = describeGraph(updatedSituationGraph); + + const selectedNode = + deterministicSelection?.status === "selected" + ? findNodeById(updatedSituationGraph, deterministicSelection.nodeId) + : null; + const formulatedQuestion = selectedNode + ? formulateQuestion({ + node: selectedNode, + graph: updatedSituationGraph, + context: { selectionState: deterministicSelection }, + }) + : null; + + return { + success: true, + updatedSituationGraph, + deterministicSelection, + selectedQuestion: + deterministicSelection?.status === "ambiguous" + ? { + id: "q_tie_resolution", + ...formulateTieResolutionQuestion({ graph: updatedSituationGraph }), + nodeId: null, + tiedCandidateIds: deterministicSelection.tiedCandidateIds, + } + : deterministicSelection?.status === "selected" && formulatedQuestion + ? { + nodeId: deterministicSelection.nodeId, + question: + formulatedQuestion.question || deterministicSelection.question, + reason: formulatedQuestion.reason, + strategy: formulatedQuestion.strategy, + investigationStrategy: formulatedQuestion.investigationStrategy, + questionComplexity: formulatedQuestion.questionComplexity, + plainLanguageNormalisations: + formulatedQuestion.plainLanguageNormalisations, + } + : null, + atomicityAssessment: decompositionResult.atomicityAssessment, + decompositionPerformed: + decompositionResult.decompositionAttempted && + decompositionResult.decompositionAccepted, + decompositionAttempted: decompositionResult.decompositionAttempted, + selectedUnknownBefore: decompositionResult.selectedUnknownBefore, + selectedUnknownAfter: deterministicSelection?.nodeId ?? null, + questionComplexityAssessment: + formulatedQuestion?.questionComplexity ?? null, + }; +} + function runDeterministicDecomposition({ graphSnapshot, proposalSnapshot, diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 1e0b59b..103ec16 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -13,10 +13,14 @@ import { updateCaseRequestSchema, } from "./schema.js"; import { buildInitialGraph, describeGraph } from "./builder.js"; -import { applyValidatedProposal } from "./apply-proposal.js"; +import { + applyValidatedProposal, + determineGraphBackedQuestion, +} from "./apply-proposal.js"; import { buildGraphUpdatePrompt } from "./prompt-builder.js"; import { buildReasoningState, + formulateQuestion, formulateTieResolutionQuestion, } from "./question-formulator.js"; import { parseGraphUpdateProposal } from "./update-proposal.js"; @@ -41,6 +45,13 @@ function buildDiagnostics({ graph, graphReferenceValidation, unknownSelectionExplanation, + reconstructionQuestion, + reconstructionQuestionAccepted, + reconstructionQuestionRejectionReasons, + finalGraphBackedQuestion, + selectedUnknownNodeId, + decompositionApplied, + questionComplexityAssessment, }) { return { promptVersion: analysis?.promptVersion ?? null, @@ -54,6 +65,14 @@ function buildDiagnostics({ compatibilityChanges: analysis?.compatibilityChanges ?? [], compatibilityWarnings: analysis?.compatibilityWarnings ?? [], unknownSelectionExplanation: unknownSelectionExplanation ?? null, + reconstructionQuestion: reconstructionQuestion ?? null, + reconstructionQuestionAccepted: reconstructionQuestionAccepted ?? null, + reconstructionQuestionRejectionReasons: + reconstructionQuestionRejectionReasons ?? [], + finalGraphBackedQuestion: finalGraphBackedQuestion ?? null, + selectedUnknownNodeId: selectedUnknownNodeId ?? null, + decompositionApplied: decompositionApplied ?? false, + questionComplexityAssessment: questionComplexityAssessment ?? null, }; } @@ -263,23 +282,11 @@ export async function startCase(body) { }); const currentSummary = describeGraph(initialGraph); - const deterministicSelection = selectActiveUnknownCandidate( - { - ...initialGraph, - resolvedNodeIds: [], - }, - [], - ); - const activeUnknownNodeId = - deterministicSelection?.status === "selected" - ? deterministicSelection.nodeId - : null; - - const situationGraph = makeGraph({ + const initialSituationGraph = makeGraph({ centralStatement: scenario, nodes: initialGraph.nodes, edges: initialGraph.edges, - activeUnknownNodeId, + activeUnknownNodeId: null, resolvedNodeIds: [], currentSummary, reasoningState: buildReasoningState({ @@ -290,17 +297,20 @@ export async function startCase(body) { }), }); - situationGraphSchema.parse(situationGraph); + situationGraphSchema.parse(initialSituationGraph); - const graphReferenceValidation = validateGraphReferences(situationGraph); - const selectedQuestion = - deterministicSelection?.status === "ambiguous" - ? { - id: "q_tie_resolution", - ...formulateTieResolutionQuestion({ graph: situationGraph }), - tiedCandidateIds: deterministicSelection.tiedCandidateIds, - } - : (analysis.nextQuestion ?? null); + const graphReferenceValidation = validateGraphReferences( + initialSituationGraph, + ); + const initialQuestionResult = determineGraphBackedQuestion({ + situationGraph: initialSituationGraph, + }); + const situationGraph = initialQuestionResult.success + ? initialQuestionResult.updatedSituationGraph + : initialSituationGraph; + const selectedQuestion = initialQuestionResult.success + ? initialQuestionResult.selectedQuestion + : null; const unknownSelectionExplanation = buildUnknownSelectionDiagnostics( situationGraph, [], @@ -315,6 +325,22 @@ export async function startCase(body) { graph: situationGraph, graphReferenceValidation, unknownSelectionExplanation, + reconstructionQuestion: analysis.nextQuestion?.question ?? null, + reconstructionQuestionAccepted: false, + reconstructionQuestionRejectionReasons: + analysis.nextQuestion?.question != null + ? [ + "reconstruction_question_not_authoritative", + "graph_backed_pipeline_required", + ] + : [], + finalGraphBackedQuestion: selectedQuestion?.question ?? null, + selectedUnknownNodeId: + initialQuestionResult.selectedUnknownAfter ?? null, + decompositionApplied: + initialQuestionResult.decompositionPerformed ?? false, + questionComplexityAssessment: + initialQuestionResult.questionComplexityAssessment ?? null, }), validationErrors: graphReferenceValidation.errors, statusCode: 500, @@ -330,6 +356,21 @@ export async function startCase(body) { graph: situationGraph, graphReferenceValidation, unknownSelectionExplanation, + reconstructionQuestion: analysis.nextQuestion?.question ?? null, + reconstructionQuestionAccepted: false, + reconstructionQuestionRejectionReasons: + analysis.nextQuestion?.question != null + ? [ + "reconstruction_question_not_authoritative", + "graph_backed_pipeline_required", + ] + : [], + finalGraphBackedQuestion: selectedQuestion?.question ?? null, + selectedUnknownNodeId: initialQuestionResult.selectedUnknownAfter ?? null, + decompositionApplied: + initialQuestionResult.decompositionPerformed ?? false, + questionComplexityAssessment: + initialQuestionResult.questionComplexityAssessment ?? null, }), }; } diff --git a/tests/graph/orchestrator.test.js b/tests/graph/orchestrator.test.js index 0286d9b..74574d2 100644 --- a/tests/graph/orchestrator.test.js +++ b/tests/graph/orchestrator.test.js @@ -56,6 +56,41 @@ function makeAnalysisResult(overrides = {}) { }; } +function makeCommercialAnalysisResult(overrides = {}) { + return makeAnalysisResult({ + reconstruction: { + summary: + "A new reasoning method may become a commercial product, but problem existence and value remain unresolved.", + actors: [], + systemsOrObjects: [], + expectedStates: [], + observedStates: [], + differences: [], + knownTransitions: [], + unexplainedTransitions: [], + contradictions: [], + importantUnknowns: [ + { + id: "unk-commercial", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + confidence: "high", + }, + ], + plausibleInterpretations: [], + }, + nextQuestion: { + id: "q-commercial", + question: + "What specific validation metrics, pilot feedback, or competitive benchmarking results have you collected to measure whether the method solves a recognized problem and how target users evaluate its practical utility compared to existing tools?", + reason: "Model-proposed broad validation question", + }, + ...overrides, + }); +} + function makeUpdateGraph() { const unknown = makeNode({ id: "n-unknown", @@ -401,9 +436,15 @@ describe("lib/graph/orchestrator startCase", () => { }); }); - it("returns null selectedQuestion when analysis has no nextQuestion", async () => { + it("returns null selectedQuestion when neither analysis nor graph path yields a question", async () => { mockAnalyseScenario.mockResolvedValue( - makeAnalysisResult({ nextQuestion: undefined }), + makeAnalysisResult({ + nextQuestion: undefined, + reconstruction: { + ...makeAnalysisResult().reconstruction, + importantUnknowns: [], + }, + }), ); const { startCase } = await import("@/lib/graph/orchestrator.js"); @@ -413,6 +454,40 @@ describe("lib/graph/orchestrator startCase", () => { expect(result.selectedQuestion).toBeNull(); }); + it("uses the graph-backed question path instead of the reconstruction nextQuestion on startCase", async () => { + mockAnalyseScenario.mockResolvedValue(makeCommercialAnalysisResult()); + const { startCase } = await import("@/lib/graph/orchestrator.js"); + + const result = await startCase({ + scenario: + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision.", + }); + + expect(result.success).toBe(true); + expect(result.selectedQuestion?.question).toBe( + "Who experiences this problem?", + ); + expect(result.selectedQuestion?.nodeId).toBe( + result.situationGraph.activeUnknownNodeId, + ); + expect(result.selectedQuestion?.question).not.toContain( + "validation metrics, pilot feedback, or competitive benchmarking", + ); + expect(result.diagnostics.reconstructionQuestion).toContain( + "validation metrics, pilot feedback, or competitive benchmarking", + ); + expect(result.diagnostics.reconstructionQuestionAccepted).toBe(false); + expect(result.diagnostics.reconstructionQuestionRejectionReasons).toContain( + "graph_backed_pipeline_required", + ); + expect(result.diagnostics.finalGraphBackedQuestion).toBe( + "Who experiences this problem?", + ); + expect(result.diagnostics.selectedUnknownNodeId).toBe( + result.situationGraph.activeUnknownNodeId, + ); + }); + it("includes compatibility diagnostics when provided by analysis", async () => { mockAnalyseScenario.mockResolvedValue( makeAnalysisResult({ @@ -1093,16 +1168,20 @@ describe("lib/graph/orchestrator startCase", () => { ); }); - it("startCase behaviour remains unchanged", async () => { + it("startCase no longer copies analysis nextQuestion directly when a graph-backed question exists", async () => { mockAnalyseScenario.mockResolvedValue(makeAnalysisResult()); const { startCase } = await import("@/lib/graph/orchestrator.js"); const result = await startCase({ scenario: "Scenario text" }); expect(result.success).toBe(true); - expect(result.selectedQuestion).toEqual({ - id: "q-1", - question: "What denominator is being used for the complaint rate?", + expect(result.selectedQuestion).toMatchObject({ + nodeId: result.situationGraph.activeUnknownNodeId, + question: + "What would clarify need the denominator for complaint rate in this situation?", }); + expect(result.diagnostics.reconstructionQuestion).toBe( + "What denominator is being used for the complaint rate?", + ); }); }); From 42d4da349697eb0f73ebca1b5cb84c560021e555 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 09:53:28 +0100 Subject: [PATCH 04/11] feat: decompose non-answerable unknowns --- docs/v0.7-question-simplicity-experiment.md | 22 +++ lib/graph/apply-proposal.js | 200 ++++++++++++++++++-- lib/graph/orchestrator.js | 42 ++++ lib/graph/question-formulator.js | 87 +++++++++ tests/graph/unknown-answerability.test.js | 138 ++++++++++++++ 5 files changed, 477 insertions(+), 12 deletions(-) create mode 100644 tests/graph/unknown-answerability.test.js diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md index 635dd2a..dcf2e40 100644 --- a/docs/v0.7-question-simplicity-experiment.md +++ b/docs/v0.7-question-simplicity-experiment.md @@ -25,6 +25,8 @@ The engine should ask one question about one primary concept at a time. **The reconstruction model may suggest a question, but only the graph-backed deterministic pipeline may select the user-facing question.** +**A node is only questionable if it is independently answerable.** + ## One-question / one-concept rule Every user-facing question should: @@ -72,6 +74,26 @@ For the current scenario, this meant creating child unknowns such as: The selector then reaches the first foundational child through prerequisite ordering encoded in the decomposition graph rather than through global scoring changes. +## Atomicity vs answerability + +These are different reasoning properties. + +- **Atomicity** asks: does this node describe one investigation or several bundled investigations? +- **Answerability** asks: even if the wording looks singular, can this node be answered directly without first resolving multiple prerequisite dimensions? + +A node can appear atomic in wording but still fail answerability. + +Examples include broad evaluation containers such as product validation, customer value, business case, technical feasibility, or commercial justification. These often compress several prerequisite investigations into one conclusion-shaped unknown. + +That means atomicity alone is not enough. + +The engine now decomposes whenever either of these is true: + +- the unknown is not atomic +- the unknown is not independently answerable + +This prevents a broad container node from becoming the selected question target even when its wording looks grammatically singular. + ## UI result The long compound question no longer survives as the first follow-up in the tested path. diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index 302c56e..d644aad 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -1,5 +1,6 @@ import { describeGraph } from "./builder.js"; import { + assessUnknownAnswerability, assessUnknownAtomicity, buildReasoningState, classifyObservationRelationship, @@ -1624,6 +1625,63 @@ function findNodeById(graph, nodeId) { return (graph.nodes || []).find((node) => node.id === nodeId) || null; } +function isSelectableUnresolvedUnknown(graph, nodeId) { + const node = findNodeById(graph, nodeId); + return Boolean( + node && + node.kind === "unknown" && + !["resolved", "contradicted"].includes(node.status) && + !(graph.resolvedNodeIds || []).includes(node.id), + ); +} + +function selectedQuestionBelongsToChild(graph, selectedQuestion) { + if (!selectedQuestion?.nodeId) return false; + return Boolean(findNodeById(graph, selectedQuestion.nodeId)?.parentId); +} + +function selectDecompositionChildCandidate(graph, parentNodeId) { + const childCandidates = findDirectChildUnknowns(graph, parentNodeId).filter( + (node) => + node.kind === "unknown" && + !["resolved", "contradicted"].includes(node.status), + ); + + if (childCandidates.length === 0) { + return { status: "none", nodeId: null, tiedCandidateIds: [] }; + } + + const scored = childCandidates.map((node) => ({ + node, + score: + scoreUnknownCandidate(graph, node, graph.resolvedNodeIds || []).score ?? + Number.NEGATIVE_INFINITY, + })); + const topScore = Math.max(...scored.map((item) => item.score)); + const top = scored.filter((item) => item.score === topScore); + + if (top.length === 0) { + return { status: "none", nodeId: null, tiedCandidateIds: [] }; + } + + if (top.length > 1) { + return { + status: "ambiguous", + nodeId: null, + tiedCandidateIds: top.map((item) => item.node.id), + reason: + "Multiple decomposition children remain equally good next investigations.", + }; + } + + return { + status: "selected", + nodeId: top[0].node.id, + reason: + "Selected the strongest direct child investigation for a non-answerable parent unknown.", + }; +} + export function determineGraphBackedQuestion({ situationGraph }) { const graphSnapshot = cloneJsonSafe(situationGraph); let updatedSituationGraph = cloneJsonSafe(situationGraph); @@ -1656,10 +1714,20 @@ export function determineGraphBackedQuestion({ situationGraph }) { updatedSituationGraph.reasoningState = buildReasoningState( updatedSituationGraph, ); - deterministicSelection = selectActiveUnknownCandidate( + deterministicSelection = isSelectableUnresolvedUnknown( updatedSituationGraph, - updatedSituationGraph.resolvedNodeIds || [], - ); + decompositionResult.selectedChildNodeId, + ) + ? { + status: "selected", + nodeId: decompositionResult.selectedChildNodeId, + reason: + "Selected the preserved decomposition child because it remains the strongest independently answerable investigation.", + } + : selectActiveUnknownCandidate( + updatedSituationGraph, + updatedSituationGraph.resolvedNodeIds || [], + ); updatedSituationGraph.activeUnknownNodeId = deterministicSelection?.status === "selected" ? deterministicSelection.nodeId @@ -1704,10 +1772,26 @@ export function determineGraphBackedQuestion({ situationGraph }) { } : null, atomicityAssessment: decompositionResult.atomicityAssessment, + answerabilityAssessment: decompositionResult.answerabilityAssessment, + independentlyAnswerable: + decompositionResult.answerabilityAssessment?.independentlyAnswerable ?? + null, + prerequisiteConceptCount: + decompositionResult.answerabilityAssessment?.prerequisiteConceptCount ?? + null, decompositionPerformed: decompositionResult.decompositionAttempted && decompositionResult.decompositionAccepted, decompositionAttempted: decompositionResult.decompositionAttempted, + decompositionTriggeredByAnswerability: + decompositionResult.decompositionTriggeredByAnswerability ?? false, + selectedContainerUnknown: + decompositionResult.selectedContainerUnknown ?? null, + selectedChildUnknown: + decompositionResult.selectedChildNodeId ?? + (deterministicSelection?.status === "selected" + ? deterministicSelection.nodeId + : null), selectedUnknownBefore: decompositionResult.selectedUnknownBefore, selectedUnknownAfter: deterministicSelection?.nodeId ?? null, questionComplexityAssessment: @@ -1727,7 +1811,9 @@ function runDeterministicDecomposition({ let workingProposal = proposalSnapshot; let nextReasoningState = workingGraph.reasoningState; let lastAtomicityAssessment = null; + let lastAnswerabilityAssessment = null; let rootAtomicityAssessment = null; + let rootAnswerabilityAssessment = null; let decompositionDepth = 0; let decompositionAttempted = false; let decompositionAccepted = false; @@ -1742,6 +1828,8 @@ function runDeterministicDecomposition({ let rejectedChildren = []; let childQualitySummary = []; let decompositionTriggeredByQuestionComplexity = false; + let decompositionTriggeredByAnswerability = false; + let selectedContainerUnknown = null; while (workingSelection?.status === "selected" && workingSelection?.nodeId) { const selectedNode = findNodeById(workingGraph, workingSelection.nodeId); @@ -1755,13 +1843,28 @@ function runDeterministicDecomposition({ node: selectedNode, graph: workingGraph, }); + const answerabilityAssessment = assessUnknownAnswerability({ + node: selectedNode, + graph: workingGraph, + }); lastAtomicityAssessment = atomicityAssessment; + lastAnswerabilityAssessment = answerabilityAssessment; if (!rootAtomicityAssessment) { rootAtomicityAssessment = atomicityAssessment; } + if (!rootAnswerabilityAssessment) { + rootAnswerabilityAssessment = answerabilityAssessment; + } - if (atomicityAssessment.atomicity === "atomic") { - selectedChildNodeId = decompositionDepth > 0 ? selectedNode.id : null; + const decompositionRequired = + atomicityAssessment.atomicity !== "atomic" || + !answerabilityAssessment.independentlyAnswerable; + + if (!decompositionRequired) { + selectedChildNodeId = + decompositionDepth > 0 || selectedNode.parentId + ? selectedNode.id + : null; decompositionStoppedReason = decompositionDepth > 0 ? "Selected child is atomic and directly answerable." @@ -1769,9 +1872,37 @@ function runDeterministicDecomposition({ break; } + if (!selectedContainerUnknown) { + selectedContainerUnknown = selectedNode.id; + } + if (!answerabilityAssessment.independentlyAnswerable) { + decompositionTriggeredByAnswerability = true; + } + if (hasExistingDecompositionChildren(workingGraph, selectedNode.id)) { + const childSelection = selectDecompositionChildCandidate( + workingGraph, + selectedNode.id, + ); + if (childSelection.status === "selected") { + workingSelection = childSelection; + decompositionStoppedReason = + "Selected child is atomic and directly answerable."; + selectedChildNodeId = + findNodeById(workingGraph, childSelection.nodeId)?.parentId === + selectedNode.id + ? childSelection.nodeId + : null; + break; + } + if (childSelection.status === "ambiguous") { + workingSelection = childSelection; + decompositionStoppedReason = + "Selected parent is not independently answerable and its existing child investigations are tied."; + break; + } decompositionStoppedReason = - "Selected composite parent already has decomposition children, so they should be reused instead of regenerated."; + "Selected parent is not independently answerable, but no unresolved child investigation remained available."; break; } @@ -1831,9 +1962,9 @@ function runDeterministicDecomposition({ reasoningResolution.reasoningStateOverride, ); workingGraph.reasoningState = nextReasoningState; - workingSelection = selectActiveUnknownCandidate( + workingSelection = selectDecompositionChildCandidate( workingGraph, - workingGraph.resolvedNodeIds, + selectedNode.id, ); if (workingSelection?.status !== "selected") { @@ -1851,6 +1982,13 @@ function runDeterministicDecomposition({ break; } + if ( + findNodeById(workingGraph, workingSelection.nodeId)?.parentId === + selectedNode.id + ) { + selectedChildNodeId = workingSelection.nodeId; + } + decompositionAccepted = true; decompositionDepth += 1; } @@ -1863,6 +2001,8 @@ function runDeterministicDecomposition({ deterministicSelection: workingSelection, atomicityAssessment: rootAtomicityAssessment ?? lastAtomicityAssessment ?? null, + answerabilityAssessment: + rootAnswerabilityAssessment ?? lastAnswerabilityAssessment ?? null, decompositionDepth, decompositionAttempted, decompositionAccepted, @@ -1874,6 +2014,8 @@ function runDeterministicDecomposition({ selectedChildNodeId, selectedUnknownBefore, decompositionTriggeredByQuestionComplexity, + decompositionTriggeredByAnswerability, + selectedContainerUnknown, }; } @@ -2205,12 +2347,23 @@ export function applyValidatedProposal({ reasoningResolution.reasoningStateOverride, ); updatedSituationGraph.reasoningState = nextReasoningState; - deterministicSelection = selectActiveUnknownCandidate( + deterministicSelection = isSelectableUnresolvedUnknown( updatedSituationGraph, - updatedSituationGraph.resolvedNodeIds, - ); + decompositionResult.selectedChildNodeId, + ) + ? { + status: "selected", + nodeId: decompositionResult.selectedChildNodeId, + reason: + "Preserved the selected decomposition child because it remains unresolved after propagation.", + } + : selectActiveUnknownCandidate( + updatedSituationGraph, + updatedSituationGraph.resolvedNodeIds, + ); const atomicityAssessment = decompositionResult.atomicityAssessment; + const answerabilityAssessment = decompositionResult.answerabilityAssessment; const decompositionDepth = decompositionResult.decompositionDepth; const decompositionAttempted = decompositionResult.decompositionAttempted; const decompositionAccepted = decompositionResult.decompositionAccepted; @@ -2321,6 +2474,15 @@ export function applyValidatedProposal({ } : null; + const finalSelectedChildNodeId = + selectedChildNodeId ?? + (selectedQuestionBelongsToChild( + updatedSituationGraph, + finalSelectedQuestion, + ) + ? finalSelectedQuestion?.nodeId + : null); + const resultGraphValidation = situationGraphSchema.safeParse( updatedSituationGraph, ); @@ -2374,6 +2536,11 @@ export function applyValidatedProposal({ emergentReasoningNodeReason: emergentReasoningUnknown?.reason ?? null, atomicityAssessment: atomicityAssessment?.atomicity ?? null, atomicityDecisionReason: atomicityAssessment?.reason ?? null, + answerabilityAssessment, + independentlyAnswerable: + answerabilityAssessment?.independentlyAnswerable ?? null, + prerequisiteConceptCount: + answerabilityAssessment?.prerequisiteConceptCount ?? null, decompositionDepth, decompositionAttempted, decompositionAccepted, @@ -2381,7 +2548,7 @@ export function applyValidatedProposal({ proposedChildCount, acceptedChildCount, rejectedChildren, - selectedChildNodeId, + selectedChildNodeId: finalSelectedChildNodeId, childQualitySummary, selectedUnknownBefore: decompositionResult.selectedUnknownBefore, selectedUnknownAfter: deterministicSelection?.nodeId ?? null, @@ -2391,6 +2558,8 @@ export function applyValidatedProposal({ complexityReasons: questionComplexity?.reasons ?? [], decompositionTriggeredByQuestionComplexity: decompositionResult.decompositionTriggeredByQuestionComplexity ?? false, + decompositionTriggeredByAnswerability: + decompositionResult.decompositionTriggeredByAnswerability ?? false, previousQuestion, finalQuestion: finalSelectedQuestion?.question ?? null, plainLanguageNormalisations, @@ -2423,6 +2592,13 @@ export function applyValidatedProposal({ decompositionPerformed, childUnknownCount: decompositionChildNodeIds.length, childNodeIds: decompositionChildNodeIds, + selectedContainerUnknown: + decompositionResult.selectedContainerUnknown ?? null, + selectedChildUnknown: + finalSelectedChildNodeId ?? + (deterministicSelection?.status === "selected" + ? deterministicSelection.nodeId + : null), atomicityReason: propagationReason || decompositionReason || diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 103ec16..a622a62 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -52,6 +52,13 @@ function buildDiagnostics({ selectedUnknownNodeId, decompositionApplied, questionComplexityAssessment, + answerabilityAssessment, + independentlyAnswerable, + prerequisiteConceptCount, + decompositionTriggeredByAnswerability, + decompositionReason, + selectedContainerUnknown, + selectedChildUnknown, }) { return { promptVersion: analysis?.promptVersion ?? null, @@ -73,6 +80,14 @@ function buildDiagnostics({ selectedUnknownNodeId: selectedUnknownNodeId ?? null, decompositionApplied: decompositionApplied ?? false, questionComplexityAssessment: questionComplexityAssessment ?? null, + answerabilityAssessment: answerabilityAssessment ?? null, + independentlyAnswerable: independentlyAnswerable ?? null, + prerequisiteConceptCount: prerequisiteConceptCount ?? null, + decompositionTriggeredByAnswerability: + decompositionTriggeredByAnswerability ?? false, + decompositionReason: decompositionReason ?? null, + selectedContainerUnknown: selectedContainerUnknown ?? null, + selectedChildUnknown: selectedChildUnknown ?? null, }; } @@ -341,6 +356,20 @@ export async function startCase(body) { initialQuestionResult.decompositionPerformed ?? false, questionComplexityAssessment: initialQuestionResult.questionComplexityAssessment ?? null, + answerabilityAssessment: + initialQuestionResult.answerabilityAssessment ?? null, + independentlyAnswerable: + initialQuestionResult.independentlyAnswerable ?? null, + prerequisiteConceptCount: + initialQuestionResult.prerequisiteConceptCount ?? null, + decompositionTriggeredByAnswerability: + initialQuestionResult.decompositionTriggeredByAnswerability ?? false, + decompositionReason: + initialQuestionResult.selectedQuestion?.reason ?? null, + selectedContainerUnknown: + initialQuestionResult.selectedContainerUnknown ?? null, + selectedChildUnknown: + initialQuestionResult.selectedChildUnknown ?? null, }), validationErrors: graphReferenceValidation.errors, statusCode: 500, @@ -371,6 +400,19 @@ export async function startCase(body) { initialQuestionResult.decompositionPerformed ?? false, questionComplexityAssessment: initialQuestionResult.questionComplexityAssessment ?? null, + answerabilityAssessment: + initialQuestionResult.answerabilityAssessment ?? null, + independentlyAnswerable: + initialQuestionResult.independentlyAnswerable ?? null, + prerequisiteConceptCount: + initialQuestionResult.prerequisiteConceptCount ?? null, + decompositionTriggeredByAnswerability: + initialQuestionResult.decompositionTriggeredByAnswerability ?? false, + decompositionReason: + initialQuestionResult.selectedQuestion?.reason ?? null, + selectedContainerUnknown: + initialQuestionResult.selectedContainerUnknown ?? null, + selectedChildUnknown: initialQuestionResult.selectedChildUnknown ?? null, }), }; } diff --git a/lib/graph/question-formulator.js b/lib/graph/question-formulator.js index fcf1e8a..cb556f2 100644 --- a/lib/graph/question-formulator.js +++ b/lib/graph/question-formulator.js @@ -569,6 +569,93 @@ function isDirectlyAnswerableObservationChildText(text) { ); } +function countPrerequisiteConceptSignals(text) { + let count = 0; + + if (/\bproblem\b/.test(text)) count += 1; + if (/\b(audience|customer|user|buyer|stakeholder|recipient)\b/.test(text)) + count += 1; + if (/\b(demand|seek help|actively look for help)\b/.test(text)) count += 1; + if (/\b(pay|willingness to pay|price|pricing)\b/.test(text)) count += 1; + if ( + /\b(compare|comparison|different from|alternatives|alternative|existing alternatives|existing tools|better than)\b/.test( + text, + ) + ) + count += 1; + if (/\b(value|viability|justified|business case|commercial)\b/.test(text)) + count += 1; + if (/\b(feasibility|technical)\b/.test(text)) count += 1; + + return count; +} + +function countIndependentAnswerDimensions(node, graph) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + const relatedNodes = collectRelatedNodes(node, graph); + const unresolvedDependencies = relatedNodes.filter( + (relatedNode) => + relatedNode.kind === "unknown" && relatedNode.status !== "resolved", + ).length; + const conjunctionCount = (text.match(/\b(and|or)\b/g) || []).length; + const prerequisiteConceptCount = countPrerequisiteConceptSignals(text); + const implicitConclusion = + /\b(commercially justified|commercial justification|commercial viability|business case|customer value|market demand|product validation|technical feasibility)\b/.test( + text, + ) || + (/\bevidence that\b/.test(text) && + /\b(problem|need|demand|audience|customer|user|alternatives|better than)\b/.test( + text, + )) || + (/\bwhether\b/.test(text) && + /\b(addresses|solve|solves|justifies|supports|demonstrates)\b/.test( + text, + ) && + /\b(problem|value|need|demand|audience|customer|user)\b/.test(text)); + + return { + prerequisiteConceptCount, + unresolvedDependencies, + conjunctionCount, + implicitConclusion, + multipleEvidenceDimensions: + prerequisiteConceptCount >= 2 || conjunctionCount >= 2, + }; +} + +export function assessUnknownAnswerability({ node, graph }) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + const dimensionSummary = countIndependentAnswerDimensions(node, graph); + const independentlyAnswerable = + !dimensionSummary.implicitConclusion && + !dimensionSummary.multipleEvidenceDimensions && + dimensionSummary.unresolvedDependencies === 0 && + dimensionSummary.prerequisiteConceptCount <= 1; + + if (independentlyAnswerable) { + return { + independentlyAnswerable: true, + reason: + "This unknown can be answered directly without first resolving several prerequisite investigations.", + prerequisiteConceptCount: dimensionSummary.prerequisiteConceptCount, + decompositionRequired: false, + }; + } + + return { + independentlyAnswerable: false, + reason: dimensionSummary.implicitConclusion + ? "This unknown asks for a higher-level conclusion that depends on several smaller investigations." + : "This unknown still bundles multiple prerequisite evidence dimensions, so it should be decomposed before it becomes the selected question.", + prerequisiteConceptCount: Math.max( + dimensionSummary.prerequisiteConceptCount, + dimensionSummary.unresolvedDependencies, + dimensionSummary.conjunctionCount + 1, + ), + decompositionRequired: true, + }; +} + export function assessUnknownAtomicity({ node, graph }) { const nodeText = normaliseText( `${node?.label || ""} ${node?.description || ""}`, diff --git a/tests/graph/unknown-answerability.test.js b/tests/graph/unknown-answerability.test.js new file mode 100644 index 0000000..4bf21bc --- /dev/null +++ b/tests/graph/unknown-answerability.test.js @@ -0,0 +1,138 @@ +import { describe, expect, it } from "vitest"; +import { + assessUnknownAnswerability, + assessUnknownAtomicity, +} from "@/lib/graph/question-formulator.js"; +import { applyValidatedProposal } from "@/lib/graph/apply-proposal.js"; +import { makeGraph, makeNode } from "@/lib/graph/schema.js"; + +const COMMERCIAL_SCENARIO = + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified."; + +function makeMeaningfulNoOpProposal() { + return { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }; +} + +function makeCommercialContainerGraph() { + const parent = makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + return makeGraph({ + centralStatement: COMMERCIAL_SCENARIO, + nodes: [parent], + edges: [], + activeUnknownNodeId: parent.id, + resolvedNodeIds: [], + currentSummary: "Commercial answerability fixture", + }); +} + +describe("assessUnknownAnswerability", () => { + it("flags commercial-validation container unknowns as non-answerable", () => { + const graph = makeCommercialContainerGraph(); + const unknown = graph.nodes[0]; + + const atomicity = assessUnknownAtomicity({ node: unknown, graph }); + const answerability = assessUnknownAnswerability({ node: unknown, graph }); + + expect(atomicity.atomicity).toBe("composite"); + expect(answerability.independentlyAnswerable).toBe(false); + expect(answerability.decompositionRequired).toBe(true); + expect(answerability.prerequisiteConceptCount).toBeGreaterThan(1); + }); + + it("keeps one-concept denominator unknowns independently answerable", () => { + const unknown = makeNode({ + id: "n-denominator", + label: "Complaint rate denominator", + description: + "Need the denominator because it directly determines the complaint rate.", + kind: "unknown", + status: "unknown", + confidence: "high", + }); + const graph = makeGraph({ + centralStatement: "Production increased while complaints increased.", + nodes: [unknown], + edges: [], + activeUnknownNodeId: unknown.id, + resolvedNodeIds: [], + currentSummary: "Denominator answerability fixture", + }); + + const result = assessUnknownAnswerability({ node: unknown, graph }); + + expect(result.independentlyAnswerable).toBe(true); + expect(result.decompositionRequired).toBe(false); + expect(result.prerequisiteConceptCount).toBeLessThanOrEqual(1); + }); +}); + +describe("answerability-triggered decomposition", () => { + it("decomposes a non-answerable parent into independently answerable child investigations", () => { + const graph = makeCommercialContainerGraph(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + expect(result.success).toBe(true); + expect(result.decompositionPerformed).toBe(true); + expect(result.decompositionTriggeredByAnswerability).toBe(true); + expect(result.selectedContainerUnknown).toBe("n-commercial-parent"); + expect(result.selectedChildUnknown).toBe(result.selectedUnknownAfter); + expect(result.independentlyAnswerable).toBe(false); + expect(result.prerequisiteConceptCount).toBeGreaterThan(1); + expect(result.selectedUnknownAfter).not.toBe("n-commercial-parent"); + expect(result.selectedQuestion.question).toBe( + "Who experiences this problem?", + ); + }); + + it("keeps the parent unresolved while selecting a child unknown", () => { + const graph = makeCommercialContainerGraph(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + const parentNode = result.updatedSituationGraph.nodes.find( + (node) => node.id === "n-commercial-parent", + ); + const selectedChild = result.updatedSituationGraph.nodes.find( + (node) => node.id === result.selectedUnknownAfter, + ); + + expect(parentNode.status).toBe("unknown"); + expect(selectedChild.parentId).toBe(parentNode.id); + expect(selectedChild.status).toBe("unknown"); + }); +}); From b00928d6fbefa7581186df685d76495850f854e1 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 10:25:26 +0100 Subject: [PATCH 05/11] feat: introduce reasoning pattern selection --- docs/v0.7-question-simplicity-experiment.md | 89 +++ lib/graph/apply-proposal.js | 18 + lib/graph/orchestrator.js | 70 +++ lib/graph/question-formulator.js | 557 +++++++++++++++++- tests/graph/question-formulator.test.js | 65 ++ tests/graph/question-simplicity.test.js | 5 + .../graph/reasoning-pattern-selection.test.js | 201 +++++++ 7 files changed, 986 insertions(+), 19 deletions(-) create mode 100644 tests/graph/reasoning-pattern-selection.test.js diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md index dcf2e40..bdeca1b 100644 --- a/docs/v0.7-question-simplicity-experiment.md +++ b/docs/v0.7-question-simplicity-experiment.md @@ -94,6 +94,95 @@ The engine now decomposes whenever either of these is true: This prevents a broad container node from becoming the selected question target even when its wording looks grammatically singular. +## Reasoning Pattern + +The next failure exposed a deeper issue: even after atomicity and answerability were added, the engine could still choose a question template from the wrong reasoning family. + +The live failure was an explanation-style prompt appearing in a commercial validation scenario: + +> What changed during the period that could help explain why ... + +That was wrong not because of wording, but because the engine had selected an **explanation family** when the actual task was a **decision investigation**. + +To correct that, the deterministic pipeline now explicitly inserts a reasoning-pattern stage: + +```text +selected unknown +→ atomicity +→ answerability +→ reasoning pattern +→ investigation strategy +→ question family +→ question +``` + +This matters because each stage must constrain the next. + +- **Reasoning Pattern** decides what kind of reasoning is happening +- **Investigation Strategy** decides how to reduce uncertainty within that pattern +- **Question Family** decides what template space is allowed +- **Question** is the final concrete wording + +Without this stage separation, strategy and template selection can leak across domains and reuse relationship/explanation prompts too broadly. + +## Deterministic reasoning-pattern vocabulary + +The current deterministic pattern vocabulary is intentionally small: + +- decision +- explanation +- contradiction +- definition +- diagnosis +- comparison +- prioritisation + +Pattern selection uses graph structure rather than wording alone, including: + +- node kind +- relationship / observation topology +- parent context +- reasoning state +- selected unknown role in the graph + +## Question-family mapping + +Patterns now constrain which question families are allowed. + +- **decision** + - decision_foundation + - decision_evidence + - decision_threshold + - definition +- **explanation** + - explanation + - comparison +- **contradiction** + - contradiction + - comparison + - explanation +- **definition** + - definition +- **diagnosis** + - diagnosis + - comparison +- **comparison** + - comparison +- **prioritisation** + - prioritisation + - decision_threshold + +Most importantly: + +- explanation templates are only allowed for `explanation` or `contradiction` +- decision investigations cannot emit explanation-family questions + +## Live correction + +For the commercial-method scenario, the engine now classifies the reasoning as a **decision** pattern rather than an explanation pattern. + +That means explanation-family templates are explicitly rejected, and the selected child unknown must be questioned using a decision-compatible family instead. + ## UI result The long compound question no longer survives as the first follow-up in the tested path. diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index d644aad..88a995e 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -1766,6 +1766,15 @@ export function determineGraphBackedQuestion({ situationGraph }) { reason: formulatedQuestion.reason, strategy: formulatedQuestion.strategy, investigationStrategy: formulatedQuestion.investigationStrategy, + reasoningPattern: formulatedQuestion.reasoningPattern, + reasoningPatternReason: formulatedQuestion.reasoningPatternReason, + questionFamily: formulatedQuestion.questionFamily, + allowedQuestionFamilies: + formulatedQuestion.allowedQuestionFamilies, + rejectedQuestionFamilies: + formulatedQuestion.rejectedQuestionFamilies, + selectedQuestionTemplate: + formulatedQuestion.selectedQuestionTemplate, questionComplexity: formulatedQuestion.questionComplexity, plainLanguageNormalisations: formulatedQuestion.plainLanguageNormalisations, @@ -2469,6 +2478,15 @@ export function applyValidatedProposal({ reason: formulatedQuestion?.reason || deterministicSelection.reason, strategy: formulatedQuestion?.strategy, investigationStrategy: formulatedQuestion?.investigationStrategy, + reasoningPattern: formulatedQuestion?.reasoningPattern, + reasoningPatternReason: formulatedQuestion?.reasoningPatternReason, + questionFamily: formulatedQuestion?.questionFamily, + allowedQuestionFamilies: + formulatedQuestion?.allowedQuestionFamilies, + rejectedQuestionFamilies: + formulatedQuestion?.rejectedQuestionFamilies, + selectedQuestionTemplate: + formulatedQuestion?.selectedQuestionTemplate, questionComplexity, plainLanguageNormalisations, } diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index a622a62..7a34ff9 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -59,6 +59,12 @@ function buildDiagnostics({ decompositionReason, selectedContainerUnknown, selectedChildUnknown, + reasoningPattern, + questionFamily, + allowedQuestionFamilies, + rejectedQuestionFamilies, + selectedQuestionTemplate, + reasoningPatternReason, }) { return { promptVersion: analysis?.promptVersion ?? null, @@ -88,6 +94,12 @@ function buildDiagnostics({ decompositionReason: decompositionReason ?? null, selectedContainerUnknown: selectedContainerUnknown ?? null, selectedChildUnknown: selectedChildUnknown ?? null, + reasoningPattern: reasoningPattern ?? null, + questionFamily: questionFamily ?? null, + allowedQuestionFamilies: allowedQuestionFamilies ?? [], + rejectedQuestionFamilies: rejectedQuestionFamilies ?? [], + selectedQuestionTemplate: selectedQuestionTemplate ?? null, + reasoningPatternReason: reasoningPatternReason ?? null, }; } @@ -177,6 +189,12 @@ function buildUpdateDiagnostics({ selectedUnknownBefore, selectedUnknownAfter, plainLanguageNormalisations, + reasoningPattern, + questionFamily, + allowedQuestionFamilies, + rejectedQuestionFamilies, + selectedQuestionTemplate, + reasoningPatternReason, }) { return { promptVersion: promptVersion ?? "v0.4", @@ -257,6 +275,12 @@ function buildUpdateDiagnostics({ selectedUnknownBefore: selectedUnknownBefore ?? null, selectedUnknownAfter: selectedUnknownAfter ?? null, plainLanguageNormalisations: plainLanguageNormalisations ?? [], + reasoningPattern: reasoningPattern ?? null, + questionFamily: questionFamily ?? null, + allowedQuestionFamilies: allowedQuestionFamilies ?? [], + rejectedQuestionFamilies: rejectedQuestionFamilies ?? [], + selectedQuestionTemplate: selectedQuestionTemplate ?? null, + reasoningPatternReason: reasoningPatternReason ?? null, unknownSelectionExplanation: unknownSelectionExplanation ?? null, }; } @@ -370,6 +394,21 @@ export async function startCase(body) { initialQuestionResult.selectedContainerUnknown ?? null, selectedChildUnknown: initialQuestionResult.selectedChildUnknown ?? null, + reasoningPattern: + initialQuestionResult.selectedQuestion?.reasoningPattern ?? null, + questionFamily: + initialQuestionResult.selectedQuestion?.questionFamily ?? null, + allowedQuestionFamilies: + initialQuestionResult.selectedQuestion?.allowedQuestionFamilies ?? [], + rejectedQuestionFamilies: + initialQuestionResult.selectedQuestion?.rejectedQuestionFamilies ?? + [], + selectedQuestionTemplate: + initialQuestionResult.selectedQuestion?.selectedQuestionTemplate ?? + null, + reasoningPatternReason: + initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? + null, }), validationErrors: graphReferenceValidation.errors, statusCode: 500, @@ -413,6 +452,19 @@ export async function startCase(body) { selectedContainerUnknown: initialQuestionResult.selectedContainerUnknown ?? null, selectedChildUnknown: initialQuestionResult.selectedChildUnknown ?? null, + reasoningPattern: + initialQuestionResult.selectedQuestion?.reasoningPattern ?? null, + questionFamily: + initialQuestionResult.selectedQuestion?.questionFamily ?? null, + allowedQuestionFamilies: + initialQuestionResult.selectedQuestion?.allowedQuestionFamilies ?? [], + rejectedQuestionFamilies: + initialQuestionResult.selectedQuestion?.rejectedQuestionFamilies ?? [], + selectedQuestionTemplate: + initialQuestionResult.selectedQuestion?.selectedQuestionTemplate ?? + null, + reasoningPatternReason: + initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? null, }), }; } @@ -709,6 +761,18 @@ async function updateCaseWithDependencies(body, dependencies = {}) { selectedUnknownAfter: applicationResult.selectedUnknownAfter, plainLanguageNormalisations: applicationResult.plainLanguageNormalisations, + reasoningPattern: + applicationResult.selectedQuestion?.reasoningPattern ?? null, + questionFamily: + applicationResult.selectedQuestion?.questionFamily ?? null, + allowedQuestionFamilies: + applicationResult.selectedQuestion?.allowedQuestionFamilies ?? [], + rejectedQuestionFamilies: + applicationResult.selectedQuestion?.rejectedQuestionFamilies ?? [], + selectedQuestionTemplate: + applicationResult.selectedQuestion?.selectedQuestionTemplate ?? null, + reasoningPatternReason: + applicationResult.selectedQuestion?.reasoningPatternReason ?? null, unknownSelectionExplanation: buildUnknownSelectionDiagnostics( applicationResult.updatedSituationGraph, applicationResult.updatedSituationGraph.resolvedNodeIds || [], @@ -787,6 +851,12 @@ async function updateCaseWithDependencies(body, dependencies = {}) { selectedUnknownBefore: null, selectedUnknownAfter: null, plainLanguageNormalisations: [], + reasoningPattern: null, + questionFamily: null, + allowedQuestionFamilies: [], + rejectedQuestionFamilies: [], + selectedQuestionTemplate: null, + reasoningPatternReason: null, unknownSelectionExplanation: buildUnknownSelectionDiagnostics( situationGraph, situationGraph.resolvedNodeIds || [], diff --git a/lib/graph/question-formulator.js b/lib/graph/question-formulator.js index cb556f2..111f7f8 100644 --- a/lib/graph/question-formulator.js +++ b/lib/graph/question-formulator.js @@ -527,6 +527,10 @@ function buildFoundationalDirectQuestion(node) { function isRelationshipExplanationUnknown(node, graph) { const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + if (isDirectlyAnswerableObservationChildText(text)) { + return false; + } + return ( collectObservationNodes(graph).length >= 2 && /\b(explain|explanation|divergence|moved differently|difference between|change or event|what changed|why the observations)/.test( @@ -729,6 +733,433 @@ export function assessUnknownAtomicity({ node, graph }) { }; } +const ALL_REASONING_PATTERNS = [ + "decision", + "explanation", + "contradiction", + "definition", + "diagnosis", + "comparison", + "prioritisation", +]; + +const QUESTION_FAMILIES_BY_PATTERN = { + decision: [ + "decision_foundation", + "decision_evidence", + "decision_threshold", + "definition", + ], + explanation: ["explanation", "comparison"], + contradiction: ["contradiction", "comparison", "explanation"], + definition: ["definition"], + diagnosis: ["diagnosis", "comparison"], + comparison: ["comparison"], + prioritisation: ["prioritisation", "decision_threshold"], +}; + +const STRATEGIES_BY_PATTERN = { + decision: [ + "decision_threshold", + "evidence_gathering", + "definition", + "baseline_reconstruction", + ], + explanation: ["evidence_gathering", "baseline_reconstruction"], + contradiction: [ + "contradiction_resolution", + "baseline_reconstruction", + "evidence_gathering", + ], + definition: ["definition"], + diagnosis: ["evidence_gathering", "baseline_reconstruction"], + comparison: ["baseline_reconstruction", "evidence_gathering"], + prioritisation: ["decision_threshold", "evidence_gathering"], +}; + +function buildParentChain(node, graph) { + const nodesById = buildNodeMap(graph); + const chain = []; + let current = node?.parentId ? nodesById.get(node.parentId) : null; + + while (current) { + chain.push(current); + current = current.parentId ? nodesById.get(current.parentId) : null; + } + + return chain; +} + +function hasDecisionContext(node, graph, relatedNodes = []) { + const ancestry = buildParentChain(node, graph); + const contextText = normaliseText( + [ + graph?.centralStatement, + node?.label, + node?.description, + ...relatedNodes.map((relatedNode) => relatedNode.label), + ...relatedNodes.map((relatedNode) => relatedNode.description), + ...ancestry.map((ancestor) => ancestor.label), + ...ancestry.map((ancestor) => ancestor.description), + ...collectResolvedContextValues(graph), + ] + .filter(Boolean) + .join(" "), + ); + + return /\b(whether to|build|launch|continue|proceed|invest|commercially justified|commercial justification|commercial value|business case|viability)\b/.test( + contextText, + ); +} + +function hasObservationRelationshipTopology(node, graph, relatedNodes = []) { + const ancestry = buildParentChain(node, graph); + const topologyNodes = [node, ...relatedNodes, ...ancestry].filter(Boolean); + + return ( + collectObservationNodes(graph).length >= 2 && + topologyNodes.some( + (candidate) => + candidate.kind === "observation" || candidate.kind === "relationship", + ) + ); +} + +function isDefinitionPatternCandidate(node, graph) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + return isDefinitionLikeUnknown( + text, + `${text} ${graph?.centralStatement || ""}`, + ); +} + +function isContradictionPatternCandidate(node, graph, relatedNodes = []) { + const relationshipStatus = graph?.reasoningState?.relationshipStatus ?? null; + const text = normaliseText( + [ + node?.label, + node?.description, + ...relatedNodes.map((relatedNode) => relatedNode.label), + ...relatedNodes.map((relatedNode) => relatedNode.description), + ] + .filter(Boolean) + .join(" "), + ); + + return ( + relationshipStatus === "contradictory" || + /\b(contradiction|contradict|conflict|inconsistent|mismatch|opposing)\b/.test( + text, + ) + ); +} + +function isComparisonPatternCandidate(node, graph, relatedNodes = []) { + if (isRelationshipExplanationUnknown(node, graph)) { + return false; + } + + const text = normaliseText( + [ + node?.label, + node?.description, + ...relatedNodes.map((relatedNode) => relatedNode.label), + ...relatedNodes.map((relatedNode) => relatedNode.description), + ] + .filter(Boolean) + .join(" "), + ); + + return ( + (hasObservationRelationshipTopology(node, graph, relatedNodes) || + collectObservationNodes(graph).length >= 2 || + graph?.reasoningState?.comparabilityStatus === "uncertain") && + /\b(compare|comparison|different timing|measured|measurement|basis|scale|period|alternative|alternatives|better than)\b/.test( + text, + ) + ); +} + +function isExplanationPatternCandidate(node, graph, relatedNodes = []) { + if (isRelationshipExplanationUnknown(node, graph)) return true; + + const text = normaliseText( + [ + node?.label, + node?.description, + ...relatedNodes.map((relatedNode) => relatedNode.label), + ...relatedNodes.map((relatedNode) => relatedNode.description), + ] + .filter(Boolean) + .join(" "), + ); + + return ( + hasObservationRelationshipTopology(node, graph, relatedNodes) && + /\b(explain|explanation|why .* but|difference between|divergence|what changed|moved differently)\b/.test( + text, + ) + ); +} + +function isPrioritisationPatternCandidate(node, graph, relatedNodes = []) { + const text = normaliseText( + [ + node?.label, + node?.description, + graph?.centralStatement, + ...relatedNodes.map((relatedNode) => relatedNode.label), + ] + .filter(Boolean) + .join(" "), + ); + + return /\b(prioritise|prioritize|priority|rank|ranking|trade off|tradeoff|which first)\b/.test( + text, + ); +} + +export function selectReasoningPattern({ node, graph, context = {} }) { + const relatedNodes = collectRelatedNodes(node, graph); + const patternContext = { + hasDecisionContext: hasDecisionContext(node, graph, relatedNodes), + hasObservationRelationshipTopology: hasObservationRelationshipTopology( + node, + graph, + relatedNodes, + ), + }; + + if (isDefinitionPatternCandidate(node, graph)) { + return { + pattern: "definition", + reason: + "Selected definition because the active unknown is about meaning, scope, or term boundaries.", + context: patternContext, + }; + } + + if (isContradictionPatternCandidate(node, graph, relatedNodes)) { + return { + pattern: "contradiction", + reason: + "Selected contradiction because the graph indicates opposing claims or incompatible observations.", + context: patternContext, + }; + } + + if (isComparisonPatternCandidate(node, graph, relatedNodes)) { + return { + pattern: "comparison", + reason: + "Selected comparison because the active unknown is about distinguishing measurements, timing, basis, or alternatives.", + context: patternContext, + }; + } + + if (isExplanationPatternCandidate(node, graph, relatedNodes)) { + return { + pattern: "explanation", + reason: + "Selected explanation because the active unknown is about accounting for a relationship between observations.", + context: patternContext, + }; + } + + if (patternContext.hasDecisionContext) { + return { + pattern: "decision", + reason: + "Selected decision because the active unknown sits inside a build, continue, invest, or commercial-justification decision context.", + context: patternContext, + }; + } + + if (isPrioritisationPatternCandidate(node, graph, relatedNodes)) { + return { + pattern: "prioritisation", + reason: + "Selected prioritisation because the active unknown is about ordering options or trade-offs.", + context: patternContext, + }; + } + + return { + pattern: "diagnosis", + reason: + "Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation.", + context: patternContext, + }; +} + +function allowedQuestionFamiliesForPattern(pattern) { + return QUESTION_FAMILIES_BY_PATTERN[pattern] || [pattern]; +} + +function rejectedQuestionFamiliesForPattern(pattern) { + const allowed = new Set(allowedQuestionFamiliesForPattern(pattern)); + return ALL_REASONING_PATTERNS.flatMap((candidatePattern) => + ( + QUESTION_FAMILIES_BY_PATTERN[candidatePattern] || [candidatePattern] + ).filter((family) => !allowed.has(family)), + ).filter((family, index, list) => list.indexOf(family) === index); +} + +function constrainStrategyToReasoningPattern(strategy, reasoningPattern) { + if (!strategy) return null; + const allowedStrategies = STRATEGIES_BY_PATTERN[reasoningPattern] || []; + return allowedStrategies.includes(strategy.key) ? strategy : null; +} + +function selectQuestionFamily({ + node, + graph, + reasoningPattern, + investigationStrategy, +}) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + + if (reasoningPattern === "definition") { + return { family: "definition", template: "definition_meaning" }; + } + + if (reasoningPattern === "contradiction") { + if (/\b(period|timing|basis|scale|measure|measured)\b/.test(text)) { + return { + family: "comparison", + template: "comparison_reconcile_measurement", + }; + } + return { + family: "contradiction", + template: "contradiction_resolve_opposition", + }; + } + + if (reasoningPattern === "explanation") { + return { + family: "explanation", + template: "explanation_broad_investigation", + }; + } + + if (reasoningPattern === "comparison") { + return /\b(period|timing)\b/.test(text) && + !/\bhow the two observations were measured|measurement\b/.test(text) + ? { family: "comparison", template: "comparison_timing_basis" } + : { family: "comparison", template: "comparison_measurement_basis" }; + } + + if (reasoningPattern === "decision") { + if ( + /\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/.test( + text, + ) + ) { + return { family: "decision_foundation", template: "decision_audience" }; + } + if ( + /\b(alternative|alternatives|better than|different from|deal with)\b/.test( + text, + ) + ) { + return { + family: "decision_foundation", + template: "decision_current_alternatives", + }; + } + if (/\b(problem|need|demand)\b/.test(text)) { + return { + family: "decision_foundation", + template: "decision_problem_existence", + }; + } + if (investigationStrategy?.key === "decision_threshold") { + return { + family: "decision_threshold", + template: "decision_threshold_outcome", + }; + } + return { + family: "decision_evidence", + template: "decision_evidence_clarification", + }; + } + + if (reasoningPattern === "prioritisation") { + return { family: "prioritisation", template: "prioritisation_tradeoff" }; + } + + return investigationStrategy?.key === "baseline_reconstruction" + ? { family: "comparison", template: "diagnosis_baseline_comparison" } + : { family: "diagnosis", template: "diagnosis_evidence" }; +} + +function buildQuestionFromFamily({ + node, + graph, + reasoningPattern, + questionFamily, + selectedQuestionTemplate, + investigationStrategy, +}) { + const meaning = extractMeaning(node); + + if (reasoningPattern === "decision") { + if (selectedQuestionTemplate === "decision_audience") { + return "Who experiences this problem?"; + } + if (selectedQuestionTemplate === "decision_current_alternatives") { + return "How do people deal with this today?"; + } + if (selectedQuestionTemplate === "decision_problem_existence") { + return "What makes you think this is a real problem?"; + } + if (selectedQuestionTemplate === "decision_threshold_outcome") { + return buildQuestionFromStrategy( + investigationStrategy || { + key: "decision_threshold", + meaning, + actionPhrase: null, + }, + ); + } + return `What evidence would clarify ${stripTrailingPunctuation(meaning)}?`; + } + + if (questionFamily === "definition") { + return `What does ${meaning} mean in this situation?`; + } + + if (reasoningPattern === "comparison") { + if (selectedQuestionTemplate === "comparison_timing_basis") { + return `What evidence would clarify whether ${stripTrailingPunctuation(meaning)}?`; + } + if (selectedQuestionTemplate === "comparison_measurement_basis") { + return `What evidence would clarify ${stripTrailingPunctuation(meaning)}?`; + } + return `What evidence would clarify ${stripTrailingPunctuation(meaning)}?`; + } + + if (reasoningPattern === "contradiction") { + return investigationStrategy?.key === "contradiction_resolution" + ? buildQuestionFromStrategy(investigationStrategy) + : `What fact would resolve the contradiction about ${stripTrailingPunctuation(meaning)}?`; + } + + if (reasoningPattern === "explanation") { + return buildBroadInvestigationQuestion(graph); + } + + if (reasoningPattern === "prioritisation") { + return `Which option should be investigated first, and why?`; + } + + return investigationStrategy + ? buildQuestionFromStrategy(investigationStrategy) + : buildNeutralClarificationQuestion(meaning); +} + export function formulateTieResolutionQuestion({ graph }) { const comparability = assessComparability(graph); if (comparability.comparabilityStatus === "uncertain") { @@ -747,6 +1178,14 @@ export function formulateTieResolutionQuestion({ graph }) { relationshipStatus: deferredRelationship.relationshipStatus, relationshipReason: deferredRelationship.reason, relationshipAssessed: deferredRelationship.relationshipAssessed, + reasoningPattern: "comparison", + reasoningPatternReason: + "Tie resolution is using the comparison family because comparability is still unresolved.", + allowedQuestionFamilies: allowedQuestionFamiliesForPattern("comparison"), + rejectedQuestionFamilies: + rejectedQuestionFamiliesForPattern("comparison"), + questionFamily: "comparison", + selectedQuestionTemplate: "comparison_tie_resolution", questionRequired: true, reasoningStages: deferredRelationship.reasoningStages, }; @@ -766,6 +1205,14 @@ export function formulateTieResolutionQuestion({ graph }) { relationshipReason: relationship.reason, relationshipAssessed: relationship.relationshipAssessed, contradictionReasoningAllowed: relationship.contradictionReasoningAllowed, + reasoningPattern: "comparison", + reasoningPatternReason: + "Tie resolution remains in the comparison family because no distinct winning unknown exists.", + allowedQuestionFamilies: allowedQuestionFamiliesForPattern("comparison"), + rejectedQuestionFamilies: + rejectedQuestionFamiliesForPattern("comparison"), + questionFamily: "comparison", + selectedQuestionTemplate: "comparison_no_question_required", questionRequired: relationship.questionRequired, questionSuppressedReason: relationship.questionSuppressedReason, reasoningStages: relationship.reasoningStages, @@ -786,6 +1233,14 @@ export function formulateTieResolutionQuestion({ graph }) { relationshipReason: relationship.reason, relationshipAssessed: relationship.relationshipAssessed, contradictionReasoningAllowed: relationship.contradictionReasoningAllowed, + reasoningPattern: "explanation", + reasoningPatternReason: + "Tie resolution is using the explanation family because the observations appear related and need a neutral explanation question.", + allowedQuestionFamilies: allowedQuestionFamiliesForPattern("explanation"), + rejectedQuestionFamilies: + rejectedQuestionFamiliesForPattern("explanation"), + questionFamily: "explanation", + selectedQuestionTemplate: "explanation_broad_investigation", questionRequired: relationship.questionRequired, reasoningStages: relationship.reasoningStages, }; @@ -812,6 +1267,14 @@ export function formulateTieResolutionQuestion({ graph }) { relationshipReason: relationship.reason, relationshipAssessed: relationship.relationshipAssessed, contradictionReasoningAllowed: relationship.contradictionReasoningAllowed, + reasoningPattern: "contradiction", + reasoningPatternReason: + "Tie resolution is using the contradiction family because the graph is trying to distinguish incompatible explanations.", + allowedQuestionFamilies: allowedQuestionFamiliesForPattern("contradiction"), + rejectedQuestionFamilies: + rejectedQuestionFamiliesForPattern("contradiction"), + questionFamily: "contradiction", + selectedQuestionTemplate: "contradiction_distinguishing_change", questionRequired: relationship.questionRequired, reasoningStages: relationship.reasoningStages, }; @@ -899,6 +1362,9 @@ function buildInvestigationStrategy({ } export function selectInvestigationStrategy({ node, graph, context = {} }) { + const effectiveReasoningPattern = + context.reasoningPattern || + selectReasoningPattern({ node, graph, context }).pattern; const relatedNodes = collectRelatedNodes(node, graph); const meaning = extractMeaning(node); const combinedText = [ @@ -949,8 +1415,10 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { const hasPrimaryBaselineLanguage = /\b(before|previous|baseline|prior|comparable state)\b/.test(nodeText); + let selectedStrategy = null; + if (hasBaselineLanguage && hasPrimaryBaselineLanguage) { - return buildInvestigationStrategy({ + selectedStrategy = buildInvestigationStrategy({ key: "baseline_reconstruction", reason: "Selected because the unknown explicitly references a missing previous or baseline state.", @@ -991,8 +1459,11 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { relatedNode.kind === "conclusion", ); - if (hasPrimaryDefinitionLanguage || hasDefinitionLanguage) { - return buildInvestigationStrategy({ + if ( + !selectedStrategy && + (hasPrimaryDefinitionLanguage || hasDefinitionLanguage) + ) { + selectedStrategy = buildInvestigationStrategy({ key: "definition", reason: "Selected because the unknown is primarily about clarifying what a term means in this case.", @@ -1004,8 +1475,8 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { }); } - if (hasDecisionValueLanguage || hasCriteriaLanguage) { - return buildInvestigationStrategy({ + if (!selectedStrategy && (hasDecisionValueLanguage || hasCriteriaLanguage)) { + selectedStrategy = buildInvestigationStrategy({ key: "decision_threshold", reason: "Selected because the unknown determines the threshold for making or justifying a decision.", @@ -1017,8 +1488,11 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { }); } - if (hasPrimaryBaselineLanguage || hasBaselineLanguage) { - return buildInvestigationStrategy({ + if ( + !selectedStrategy && + (hasPrimaryBaselineLanguage || hasBaselineLanguage) + ) { + selectedStrategy = buildInvestigationStrategy({ key: "baseline_reconstruction", reason: "Selected because reconstructing the prior state is the most direct way to resolve the unknown.", @@ -1030,8 +1504,8 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { }); } - if (hasContradictionLanguage) { - return buildInvestigationStrategy({ + if (!selectedStrategy && hasContradictionLanguage) { + selectedStrategy = buildInvestigationStrategy({ key: "contradiction_resolution", reason: "Selected because the graph context indicates conflicting claims or inconsistent states that must be reconciled.", @@ -1043,8 +1517,11 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { }); } - if (hasEvidenceLanguage || hasMeasurementLanguage || hasConstraintLanguage) { - return buildInvestigationStrategy({ + if ( + !selectedStrategy && + (hasEvidenceLanguage || hasMeasurementLanguage || hasConstraintLanguage) + ) { + selectedStrategy = buildInvestigationStrategy({ key: "evidence_gathering", reason: hasConstraintLanguage && hasPrimaryConstraintLanguage @@ -1058,7 +1535,10 @@ export function selectInvestigationStrategy({ node, graph, context = {} }) { }); } - return null; + return constrainStrategyToReasoningPattern( + selectedStrategy, + effectiveReasoningPattern, + ); } function buildQuestionFromStrategy(strategy) { @@ -1232,8 +1712,22 @@ export function formulateQuestion({ node, graph, context = {} }) { return formulateTieResolutionQuestion({ graph }); } + const reasoningPatternSelection = selectReasoningPattern({ + node, + graph, + context, + }); + const allowedQuestionFamilies = allowedQuestionFamiliesForPattern( + reasoningPatternSelection.pattern, + ); + const rejectedQuestionFamilies = rejectedQuestionFamiliesForPattern( + reasoningPatternSelection.pattern, + ); const foundationalDirectQuestion = buildFoundationalDirectQuestion(node); - if (foundationalDirectQuestion) { + if ( + foundationalDirectQuestion && + reasoningPatternSelection.pattern === "decision" + ) { const plainLanguage = applyPlainLanguageNormalisations( sanitizeQuestionText(foundationalDirectQuestion), ); @@ -1249,6 +1743,12 @@ export function formulateQuestion({ node, graph, context = {} }) { "Formulated as a direct foundational question because this child unknown should be answered one step at a time.", strategy: null, investigationStrategy: null, + reasoningPattern: reasoningPatternSelection.pattern, + reasoningPatternReason: reasoningPatternSelection.reason, + questionFamily: "decision_foundation", + allowedQuestionFamilies, + rejectedQuestionFamilies, + selectedQuestionTemplate: "decision_foundation_direct_child", questionComplexity, plainLanguageNormalisations: plainLanguage.normalisations, }; @@ -1257,14 +1757,27 @@ export function formulateQuestion({ node, graph, context = {} }) { const investigationStrategy = selectInvestigationStrategy({ node, graph, - context, + context: { + ...context, + reasoningPattern: reasoningPatternSelection.pattern, + }, }); - let question = investigationStrategy - ? buildQuestionFromStrategy(investigationStrategy) - : isRelationshipExplanationUnknown(node, graph) - ? buildBroadInvestigationQuestion(graph) - : buildNeutralClarificationQuestion(extractMeaning(node)); + const questionFamilySelection = selectQuestionFamily({ + node, + graph, + reasoningPattern: reasoningPatternSelection.pattern, + investigationStrategy, + }); + + let question = buildQuestionFromFamily({ + node, + graph, + reasoningPattern: reasoningPatternSelection.pattern, + questionFamily: questionFamilySelection.family, + selectedQuestionTemplate: questionFamilySelection.template, + investigationStrategy, + }); question = sanitizeQuestionText(question); const plainLanguage = applyPlainLanguageNormalisations(question); @@ -1310,6 +1823,12 @@ export function formulateQuestion({ node, graph, context = {} }) { : "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied.", strategy: investigationStrategy?.key ?? null, investigationStrategy, + reasoningPattern: reasoningPatternSelection.pattern, + reasoningPatternReason: reasoningPatternSelection.reason, + questionFamily: questionFamilySelection.family, + allowedQuestionFamilies, + rejectedQuestionFamilies, + selectedQuestionTemplate: questionFamilySelection.template, questionComplexity, plainLanguageNormalisations: plainLanguage.normalisations, }; diff --git a/tests/graph/question-formulator.test.js b/tests/graph/question-formulator.test.js index b34ccde..97a8512 100644 --- a/tests/graph/question-formulator.test.js +++ b/tests/graph/question-formulator.test.js @@ -3,6 +3,7 @@ import { assessUnknownAtomicity, formulateQuestion, formulateTieResolutionQuestion, + selectReasoningPattern, selectInvestigationStrategy, } from "@/lib/graph/question-formulator.js"; import { makeEdge, makeGraph, makeNode } from "@/lib/graph/schema.js"; @@ -105,6 +106,8 @@ describe("formulateQuestion", () => { const result = formulateQuestion({ node: unknown, graph }); expect(result.strategy).toBe("decision_threshold"); + expect(result.reasoningPattern).toBe("decision"); + expect(result.questionFamily).toBe("decision_threshold"); expect(result.question).toContain("What outcome"); expect(result.question.toLowerCase()).toContain("justify"); }); @@ -144,6 +147,8 @@ describe("formulateQuestion", () => { }); expect(result.strategy).toBe("definition"); + expect(result.reasoningPattern).toBe("definition"); + expect(result.questionFamily).toBe("definition"); expect(result.question).toMatch(/^What does /); }); @@ -213,9 +218,68 @@ describe("formulateQuestion", () => { }); expect(result.strategy).toBe("contradiction_resolution"); + expect(result.reasoningPattern).toBe("contradiction"); expect(result.question).toContain("resolve the contradiction"); }); + it("reasoning pattern selection marks commercial validation as decision rather than explanation", () => { + const unknown = makeNode({ + id: "n-commercial-pattern", + label: + "Whether the method addresses a genuine, high-priority problem for a specific audience", + description: + "Need to know whether this solves a real problem for a clear audience before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "high", + }); + const graph = makeGraphFor(unknown, { + centralStatement: + "Before investing more, we need to know whether continuing development is commercially justified.", + }); + + const result = selectReasoningPattern({ node: unknown, graph }); + + expect(result.pattern).toBe("decision"); + }); + + it("comparison scenario selects the comparison pattern", () => { + const unknown = makeNode({ + id: "n-comparison-pattern", + label: "How the two observations were measured", + description: + "Need evidence about the measure used for each observation, because that could help explain the difference.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraphFor(unknown, { + centralStatement: "Traffic increased, but sales stayed flat.", + nodes: [ + makeNode({ + id: "n-traffic-observation", + label: "Traffic increased.", + description: "Traffic increased.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-sales-observation", + label: "Sales stayed flat.", + description: "Sales stayed flat.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + }); + + const result = selectReasoningPattern({ node: unknown, graph }); + + expect(result.pattern).toBe("comparison"); + }); + it("constraint unknown uses evidence-gathering within the fixed strategy set", () => { const unknown = makeNode({ id: "n-constraint", @@ -445,6 +509,7 @@ describe("formulateQuestion", () => { node: unknown, graph: makeGraphFor(unknown), }); + expect(result.reasoningPattern).toBe("diagnosis"); expect(result.strategy).toBeNull(); expect(result.question).toBe( "What would clarify possible causes of the divergence in this situation?", diff --git a/tests/graph/question-simplicity.test.js b/tests/graph/question-simplicity.test.js index 19c779e..2765346 100644 --- a/tests/graph/question-simplicity.test.js +++ b/tests/graph/question-simplicity.test.js @@ -126,6 +126,11 @@ describe("question simplicity", () => { expect(result.selectedQuestion.question).toBe( "Who experiences this problem?", ); + expect(result.selectedQuestion.reasoningPattern).toBe("decision"); + expect(result.selectedQuestion.questionFamily).toBe("decision_foundation"); + expect(result.selectedQuestion.selectedQuestionTemplate).toBe( + "decision_foundation_direct_child", + ); expect(result.selectedQuestion.question.match(/\?/g) || []).toHaveLength(1); expect(result.questionComplexityAccepted).toBe(true); expect(result.primaryConceptCount).toBe(1); diff --git a/tests/graph/reasoning-pattern-selection.test.js b/tests/graph/reasoning-pattern-selection.test.js new file mode 100644 index 0000000..20eac33 --- /dev/null +++ b/tests/graph/reasoning-pattern-selection.test.js @@ -0,0 +1,201 @@ +import { describe, expect, it } from "vitest"; +import { + formulateQuestion, + formulateTieResolutionQuestion, + selectReasoningPattern, +} from "@/lib/graph/question-formulator.js"; +import { makeGraph, makeNode } from "@/lib/graph/schema.js"; + +function makeGraphFor(node, extra = {}) { + return makeGraph({ + centralStatement: extra.centralStatement || "Decision context", + nodes: [node, ...(extra.nodes || [])], + edges: extra.edges || [], + activeUnknownNodeId: node.id, + resolvedNodeIds: extra.resolvedNodeIds || [], + currentSummary: "Test summary", + reasoningState: extra.reasoningState, + }); +} + +describe("reasoning pattern selection", () => { + it("commercial-method scenario selects the decision pattern and rejects explanation family", () => { + const unknown = makeNode({ + id: "n-commercial-method", + label: + "Whether the method addresses a genuine, high-priority problem for a specific audience", + description: + "Need to know whether this solves a real problem for a clear audience before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "high", + }); + const graph = makeGraphFor(unknown, { + centralStatement: + "Before investing significant time and money, we need to know whether continuing development is commercially justified.", + }); + + const result = formulateQuestion({ node: unknown, graph }); + + expect(result.reasoningPattern).toBe("decision"); + expect(result.questionFamily).not.toBe("explanation"); + expect(result.allowedQuestionFamilies).toContain("decision_foundation"); + expect(result.rejectedQuestionFamilies).toContain("explanation"); + }); + + it("revenue and cash relationship scenario allows the explanation family", () => { + const unknown = makeNode({ + id: "n-revenue-cash-explanation", + label: + "Explanation for why revenue increased by 18%, but cash in the bank fell over the same period", + description: + "Need to understand what change or event could explain why these observations differ.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraphFor(unknown, { + centralStatement: + "Revenue increased by 18%, but cash in the bank fell over the same period.", + nodes: [ + makeNode({ + id: "n-revenue-observation", + label: "Revenue increased by 18%.", + description: "Revenue increased by 18%.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-cash-observation", + label: "Cash in the bank fell over the same period.", + description: "Cash in the bank fell over the same period.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + }); + + const result = formulateQuestion({ node: unknown, graph }); + + expect(result.reasoningPattern).toBe("explanation"); + expect(result.allowedQuestionFamilies).toContain("explanation"); + }); + + it("duplicate observations reject explanation family during tie resolution", () => { + const graph = makeGraph({ + centralStatement: "The same figure was repeated twice.", + nodes: [ + makeNode({ + id: "n-obs-1", + label: "Revenue increased by 10%.", + description: "Revenue increased by 10%.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-obs-2", + label: "Revenue increased by 10%.", + description: "Revenue increased by 10%.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + edges: [], + activeUnknownNodeId: null, + resolvedNodeIds: [], + currentSummary: "Duplicate observation fixture", + }); + + const result = formulateTieResolutionQuestion({ graph }); + + expect(result.questionFamily).not.toBe("explanation"); + expect(result.rejectedQuestionFamilies).toContain("explanation"); + }); + + it("definition scenario selects the definition pattern", () => { + const unknown = makeNode({ + id: "n-definition-pattern", + label: "Definition of justified confidence", + description: "The term needs clearer boundaries.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + const result = selectReasoningPattern({ + node: unknown, + graph: makeGraphFor(unknown), + }); + + expect(result.pattern).toBe("definition"); + }); + + it("comparison scenario selects the comparison pattern", () => { + const unknown = makeNode({ + id: "n-comparison-pattern-2", + label: "How the two observations were measured", + description: + "Need evidence about the measure used for each observation before comparing them.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraphFor(unknown, { + centralStatement: "Traffic increased, but sales stayed flat.", + nodes: [ + makeNode({ + id: "n-traffic-2", + label: "Traffic increased.", + description: "Traffic increased.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-sales-2", + label: "Sales stayed flat.", + description: "Sales stayed flat.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + reasoningState: { + comparabilityStatus: "uncertain", + relationshipStatus: "insufficient_information", + }, + }); + + const result = selectReasoningPattern({ node: unknown, graph }); + + expect(result.pattern).toBe("comparison"); + }); + + it("question family stays compatible with the reasoning pattern", () => { + const unknown = makeNode({ + id: "n-decision-family-compatibility", + label: "Who experiences this problem", + description: + "Need to know who experiences this problem before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraphFor(unknown, { + centralStatement: + "We need to know whether continuing development is commercially justified.", + }); + + const result = formulateQuestion({ node: unknown, graph }); + + expect(result.reasoningPattern).toBe("decision"); + expect(result.allowedQuestionFamilies).toContain(result.questionFamily); + expect(result.rejectedQuestionFamilies).not.toContain( + result.questionFamily, + ); + }); +}); From 3e2edd2edc097edad5f3306b0d493ef5a7583ed0 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 11:28:49 +0100 Subject: [PATCH 06/11] fix: continue question selection after graph updates --- docs/v0.7-question-simplicity-experiment.md | 25 ++++ lib/graph/apply-proposal.js | 129 +++++++++++++++++++- lib/graph/orchestrator.js | 26 ++++ lib/graph/question-formulator.js | 3 + tests/graph/apply-proposal.test.js | 117 ++++++++++++++++++ tests/graph/orchestrator.test.js | 111 +++++++++++++++++ tests/ui/scenario-form.test.jsx | 100 +++++++++++++++ 7 files changed, 509 insertions(+), 2 deletions(-) diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md index bdeca1b..80865f0 100644 --- a/docs/v0.7-question-simplicity-experiment.md +++ b/docs/v0.7-question-simplicity-experiment.md @@ -1,3 +1,28 @@ +## Post-update selection invariant + +After every successful graph update, the full deterministic question-selection pipeline must run again whenever eligible unresolved unknowns remain. + +That means the update path must not stop at graph mutation, child resolution, emergent unknown creation, decomposition, or upward propagation. It must continue through: + +```text +updated graph +→ rebuild reasoning state +→ identify unresolved candidates +→ select active unknown +→ atomicity assessment +→ answerability assessment +→ decompose if required +→ reselect +→ reasoning-pattern selection +→ investigation-strategy selection +→ question-family selection +→ question formulation +→ complexity validation +→ selectedQuestion +``` + +Returning no question is only valid when no eligible unresolved candidate remains, the case is complete, ambiguity cannot be safely resolved, or question formulation fails validation with an explicit deterministic reason. + # v0.7 Question Simplicity Experiment ## Observed failure diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index 88a995e..c08786f 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -1407,15 +1407,24 @@ function buildDecompositionTemplates(parentNode, graph, depth = 0) { dependsOnLabels: [], }, { - label: "What happens when this problem is not resolved", + label: "Whether other people experience this problem", description: - "Need to know what happens when this problem is not resolved, because that is needed before judging whether the problem matters.", + "Need to know whether other people experience this problem, because that must be established before deciding whether the problem is broadly important.", dependsOnLabels: ["Who experiences this problem"], }, { label: "How often this problem happens", description: "Need to know how often this problem happens, because that helps judge whether it is a real recurring problem.", + dependsOnLabels: [ + "Who experiences this problem", + "Whether other people experience this problem", + ], + }, + { + label: "What happens when this problem is not resolved", + description: + "Need to know what happens when this problem is not resolved, because that is needed before judging whether the problem matters.", dependsOnLabels: ["Who experiences this problem"], }, { @@ -1424,6 +1433,7 @@ function buildDecompositionTemplates(parentNode, graph, depth = 0) { "Need to know how people deal with this problem today, because that is needed before comparing alternatives or value.", dependsOnLabels: [ "Who experiences this problem", + "Whether other people experience this problem", "What happens when this problem is not resolved", "How often this problem happens", ], @@ -1435,6 +1445,7 @@ function buildDecompositionTemplates(parentNode, graph, depth = 0) { "Need to know whether people actively look for help with this problem, because that is needed before judging demand or willingness to pay.", dependsOnLabels: [ "Who experiences this problem", + "Whether other people experience this problem", "What happens when this problem is not resolved", "How often this problem happens", "How people deal with this problem today", @@ -1446,6 +1457,7 @@ function buildDecompositionTemplates(parentNode, graph, depth = 0) { "Need to know whether people would pay to solve this problem, because that can only be judged after the problem itself is established.", dependsOnLabels: [ "Who experiences this problem", + "Whether other people experience this problem", "What happens when this problem is not resolved", "How often this problem happens", "How people deal with this problem today", @@ -1635,6 +1647,81 @@ function isSelectableUnresolvedUnknown(graph, nodeId) { ); } +function listUnresolvedUnknownCandidates( + graph, + resolvedCurrentTurnNodeIds = [], +) { + const resolvedCurrentTurnSet = new Set(resolvedCurrentTurnNodeIds || []); + + return (graph.nodes || []).filter( + (node) => + node.kind === "unknown" && + !["resolved", "contradicted"].includes(node.status) && + !(graph.resolvedNodeIds || []).includes(node.id) && + !resolvedCurrentTurnSet.has(node.id), + ); +} + +function listEligibleUnknownCandidates(graph, resolvedCurrentTurnNodeIds = []) { + return listUnresolvedUnknownCandidates( + graph, + resolvedCurrentTurnNodeIds, + ).filter( + (node) => + (scoreUnknownCandidate(graph, node, graph.resolvedNodeIds || []) + .unresolvedParentUnknownCount ?? 0) === 0, + ); +} + +function selectOrderedSiblingCandidate( + graph, + candidateNodeIds = [], + resolvedCurrentTurnNodeIds = [], +) { + const candidates = candidateNodeIds + .map((nodeId) => findNodeById(graph, nodeId)) + .filter(Boolean); + + if (candidates.length < 2) { + return null; + } + + const parentId = candidates[0]?.parentId ?? null; + if (!parentId || !candidates.every((node) => node.parentId === parentId)) { + return null; + } + + const parentNode = findNodeById(graph, parentId); + const orderedIds = (parentNode?.childIds || []).filter((nodeId) => + candidateNodeIds.includes(nodeId), + ); + const fallbackOrderedIds = (graph.nodes || []) + .filter((node) => candidateNodeIds.includes(node.id)) + .map((node) => node.id); + const orderedCandidateIds = + orderedIds.length > 0 ? orderedIds : fallbackOrderedIds; + + const eligibleIds = new Set( + listEligibleUnknownCandidates(graph, resolvedCurrentTurnNodeIds).map( + (node) => node.id, + ), + ); + + const selectedId = orderedCandidateIds.find((nodeId) => + eligibleIds.has(nodeId), + ); + if (!selectedId) { + return null; + } + + return { + status: "selected", + nodeId: selectedId, + reason: + "Resolved a sibling tie using the deterministic decomposition order after the update left multiple equally scored follow-up children.", + }; +} + function selectedQuestionBelongsToChild(graph, selectedQuestion) { if (!selectedQuestion?.nodeId) return false; return Boolean(findNodeById(graph, selectedQuestion.nodeId)?.parentId); @@ -2356,6 +2443,9 @@ export function applyValidatedProposal({ reasoningResolution.reasoningStateOverride, ); updatedSituationGraph.reasoningState = nextReasoningState; + const resolvedCurrentTurnNodeIds = [ + ...new Set(proposalSnapshot.resolvedUnknownNodeIds || []), + ]; deterministicSelection = isSelectableUnresolvedUnknown( updatedSituationGraph, decompositionResult.selectedChildNodeId, @@ -2371,6 +2461,27 @@ export function applyValidatedProposal({ updatedSituationGraph.resolvedNodeIds, ); + if (deterministicSelection?.status === "ambiguous") { + const orderedSiblingSelection = selectOrderedSiblingCandidate( + updatedSituationGraph, + deterministicSelection.tiedCandidateIds || [], + resolvedCurrentTurnNodeIds, + ); + + if (orderedSiblingSelection) { + deterministicSelection = orderedSiblingSelection; + } + } + + const unresolvedCandidates = listUnresolvedUnknownCandidates( + updatedSituationGraph, + resolvedCurrentTurnNodeIds, + ); + const eligibleCandidates = listEligibleUnknownCandidates( + updatedSituationGraph, + resolvedCurrentTurnNodeIds, + ); + const atomicityAssessment = decompositionResult.atomicityAssessment; const answerabilityAssessment = decompositionResult.answerabilityAssessment; const decompositionDepth = decompositionResult.decompositionDepth; @@ -2500,6 +2611,15 @@ export function applyValidatedProposal({ ) ? finalSelectedQuestion?.nodeId : null); + const noQuestionReason = finalSelectedQuestion?.question + ? null + : deterministicSelection?.status === "ambiguous" + ? "Eligible unresolved candidates remain tied after update-time reselection." + : eligibleCandidates.length === 0 + ? unresolvedCandidates.length === 0 + ? "No unresolved unknown candidates remain after this update." + : "Unresolved unknowns remain, but none are currently eligible for direct investigation." + : "Question formulation did not produce a valid next question despite eligible unresolved candidates."; const resultGraphValidation = situationGraphSchema.safeParse( updatedSituationGraph, @@ -2580,6 +2700,11 @@ export function applyValidatedProposal({ decompositionResult.decompositionTriggeredByAnswerability ?? false, previousQuestion, finalQuestion: finalSelectedQuestion?.question ?? null, + unresolvedCandidateCount: unresolvedCandidates.length, + eligibleCandidateCount: eligibleCandidates.length, + candidateNodeIds: eligibleCandidates.map((node) => node.id), + resolvedCurrentTurnNodeIds, + noQuestionReason, plainLanguageNormalisations, propagationPerformed, resolvedChildNodeId, diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 7a34ff9..731d2d6 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -195,6 +195,11 @@ function buildUpdateDiagnostics({ rejectedQuestionFamilies, selectedQuestionTemplate, reasoningPatternReason, + unresolvedCandidateCount, + eligibleCandidateCount, + candidateNodeIds, + resolvedCurrentTurnNodeIds, + noQuestionReason, }) { return { promptVersion: promptVersion ?? "v0.4", @@ -281,6 +286,11 @@ function buildUpdateDiagnostics({ rejectedQuestionFamilies: rejectedQuestionFamilies ?? [], selectedQuestionTemplate: selectedQuestionTemplate ?? null, reasoningPatternReason: reasoningPatternReason ?? null, + unresolvedCandidateCount: unresolvedCandidateCount ?? 0, + eligibleCandidateCount: eligibleCandidateCount ?? 0, + candidateNodeIds: candidateNodeIds ?? [], + resolvedCurrentTurnNodeIds: resolvedCurrentTurnNodeIds ?? [], + noQuestionReason: noQuestionReason ?? null, unknownSelectionExplanation: unknownSelectionExplanation ?? null, }; } @@ -660,6 +670,11 @@ async function updateCaseWithDependencies(body, dependencies = {}) { finalQuestion: null, selectedUnknownBefore: null, selectedUnknownAfter: null, + unresolvedCandidateCount: 0, + eligibleCandidateCount: 0, + candidateNodeIds: [], + resolvedCurrentTurnNodeIds: [], + noQuestionReason: null, plainLanguageNormalisations: [], unknownSelectionExplanation: explainUnknownSelection( situationGraph, @@ -759,6 +774,12 @@ async function updateCaseWithDependencies(body, dependencies = {}) { finalQuestion: applicationResult.finalQuestion, selectedUnknownBefore: applicationResult.selectedUnknownBefore, selectedUnknownAfter: applicationResult.selectedUnknownAfter, + unresolvedCandidateCount: applicationResult.unresolvedCandidateCount, + eligibleCandidateCount: applicationResult.eligibleCandidateCount, + candidateNodeIds: applicationResult.candidateNodeIds, + resolvedCurrentTurnNodeIds: + applicationResult.resolvedCurrentTurnNodeIds, + noQuestionReason: applicationResult.noQuestionReason, plainLanguageNormalisations: applicationResult.plainLanguageNormalisations, reasoningPattern: @@ -850,6 +871,11 @@ async function updateCaseWithDependencies(body, dependencies = {}) { finalQuestion: null, selectedUnknownBefore: null, selectedUnknownAfter: null, + unresolvedCandidateCount: 0, + eligibleCandidateCount: 0, + candidateNodeIds: [], + resolvedCurrentTurnNodeIds: [], + noQuestionReason: null, plainLanguageNormalisations: [], reasoningPattern: null, questionFamily: null, diff --git a/lib/graph/question-formulator.js b/lib/graph/question-formulator.js index 111f7f8..e7b2db3 100644 --- a/lib/graph/question-formulator.js +++ b/lib/graph/question-formulator.js @@ -504,6 +504,9 @@ function buildFoundationalDirectQuestion(node) { if (/^who experiences this problem$/i.test(label)) { return "Who experiences this problem?"; } + if (/^whether other people experience this problem$/i.test(label)) { + return "What makes you think other people experience this problem too?"; + } if (/^what happens when this problem is not resolved$/i.test(label)) { return "What happens when this problem is not resolved?"; } diff --git a/tests/graph/apply-proposal.test.js b/tests/graph/apply-proposal.test.js index 6fcde75..33400e6 100644 --- a/tests/graph/apply-proposal.test.js +++ b/tests/graph/apply-proposal.test.js @@ -229,6 +229,53 @@ function makeApplicationFixture() { }; } +const COMMERCIAL_SCENARIO = + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified."; + +function makeCommercialUpdateFixture() { + const parent = makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + return makeGraph({ + centralStatement: COMMERCIAL_SCENARIO, + nodes: [parent], + edges: [], + activeUnknownNodeId: parent.id, + resolvedNodeIds: [], + currentSummary: "Commercial update fixture", + }); +} + +function makeMeaningfulNoOpProposal() { + return { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }; +} + describe("applyValidatedProposal", () => { it("applies a valid proposal successfully", () => { const { graph, proposal, ids } = makeApplicationFixture(); @@ -1318,4 +1365,74 @@ describe("applyValidatedProposal", () => { ), ).toHaveLength(firstResult.childNodeIds.length); }); + + it("reselects a remaining commercial sibling after resolving the first child", () => { + const graph = makeCommercialUpdateFixture(); + + const firstResult = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + expect(firstResult.success).toBe(true); + expect(firstResult.selectedQuestion?.question).toBe( + "Who experiences this problem?", + ); + + const secondResult = applyValidatedProposal({ + situationGraph: firstResult.updatedSituationGraph, + proposal: { + addedNodes: [], + updatedNodes: [ + { + nodeId: firstResult.selectedQuestion.nodeId, + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + reason: "The answer confirms a self-observed instance.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [firstResult.selectedQuestion.nodeId], + affectedNodeIds: [], + selectedQuestion: null, + }, + previousQuestion: firstResult.selectedQuestion.question, + answer: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + }); + + expect(secondResult.success).toBe(true); + expect(secondResult.resolvedUnknownNodeIds).toContain( + firstResult.selectedQuestion.nodeId, + ); + expect(secondResult.newActiveUnknownNodeId).toBe( + secondResult.selectedQuestion?.nodeId, + ); + expect(secondResult.selectedQuestion?.question).toBe( + "What makes you think other people experience this problem too?", + ); + expect(secondResult.selectedQuestion?.reasoningPattern).toBe("decision"); + expect(secondResult.selectedQuestion?.questionFamily).toBe( + "decision_foundation", + ); + expect(secondResult.selectedQuestion?.nodeId).not.toBe( + firstResult.selectedQuestion.nodeId, + ); + expect(secondResult.unresolvedCandidateCount).toBeGreaterThan(0); + expect(secondResult.eligibleCandidateCount).toBeGreaterThan(0); + expect(secondResult.candidateNodeIds).toContain( + secondResult.selectedQuestion?.nodeId, + ); + expect(secondResult.resolvedCurrentTurnNodeIds).toContain( + firstResult.selectedQuestion.nodeId, + ); + expect(secondResult.noQuestionReason).toBeNull(); + expect(secondResult.selectedQuestion?.question.toLowerCase()).not.toMatch( + /price|budget|market size|pilot metrics|benchmark|technical differentiation/, + ); + }); }); diff --git a/tests/graph/orchestrator.test.js b/tests/graph/orchestrator.test.js index 74574d2..128b124 100644 --- a/tests/graph/orchestrator.test.js +++ b/tests/graph/orchestrator.test.js @@ -1,4 +1,5 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; +import { applyValidatedProposal } from "@/lib/graph/apply-proposal.js"; import { validateGraphReferences } from "@/lib/graph/utils.js"; import { makeGraph, makeNode } from "@/lib/graph/schema.js"; @@ -91,6 +92,29 @@ function makeCommercialAnalysisResult(overrides = {}) { }); } +function makeCommercialUpdateGraph() { + const parent = makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + return makeGraph({ + centralStatement: + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified.", + nodes: [parent], + edges: [], + activeUnknownNodeId: parent.id, + resolvedNodeIds: [], + currentSummary: "Commercial update scenario", + }); +} + function makeUpdateGraph() { const unknown = makeNode({ id: "n-unknown", @@ -1168,6 +1192,93 @@ describe("lib/graph/orchestrator startCase", () => { ); }); + it("reselects the other-people sibling after the first commercial child is answered", async () => { + const { updateCase } = await import("@/lib/graph/orchestrator.js"); + + const seededGraph = applyValidatedProposal({ + situationGraph: makeCommercialUpdateGraph(), + proposal: { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }, + }); + + expect(seededGraph.success).toBe(true); + const resolvedFirstChildNodeId = seededGraph.selectedQuestion?.nodeId; + const resolvedFirstQuestion = seededGraph.selectedQuestion?.question; + + const initial = await updateCase( + { + situationGraph: seededGraph.updatedSituationGraph, + previousQuestion: resolvedFirstQuestion, + answer: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + promptVersion: "v0.4", + }, + { + applyProposal: true, + config: MOCK_CONFIG, + provider: { + generateReconstruction: vi.fn().mockResolvedValue({ + addedNodes: [], + updatedNodes: [ + { + nodeId: resolvedFirstChildNodeId, + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + reason: "The answer confirms a self-observed instance.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [resolvedFirstChildNodeId], + affectedNodeIds: [], + selectedQuestion: null, + }), + }, + }, + ); + + expect(initial.success).toBe(true); + expect(initial.selectedQuestion?.nodeId).toBe( + initial.newActiveUnknownNodeId, + ); + expect(initial.selectedQuestion?.question).toBe( + "What makes you think other people experience this problem too?", + ); + expect(initial.selectedQuestion?.reasoningPattern).toBe("decision"); + expect(initial.diagnostics.unresolvedCandidateCount).toBeGreaterThan(0); + expect(initial.diagnostics.eligibleCandidateCount).toBeGreaterThan(0); + expect(initial.diagnostics.candidateNodeIds).toContain( + initial.selectedQuestion?.nodeId, + ); + expect(initial.diagnostics.resolvedCurrentTurnNodeIds).toContain( + resolvedFirstChildNodeId, + ); + expect(initial.diagnostics.noQuestionReason).toBeNull(); + expect(initial.selectedQuestion?.question.toLowerCase()).not.toMatch( + /price|budget|market size|pilot metrics|benchmark|technical differentiation/, + ); + }); + it("startCase no longer copies analysis nextQuestion directly when a graph-backed question exists", async () => { mockAnalyseScenario.mockResolvedValue(makeAnalysisResult()); const { startCase } = await import("@/lib/graph/orchestrator.js"); diff --git a/tests/ui/scenario-form.test.jsx b/tests/ui/scenario-form.test.jsx index d93caf3..9a879f8 100644 --- a/tests/ui/scenario-form.test.jsx +++ b/tests/ui/scenario-form.test.jsx @@ -275,6 +275,94 @@ function makeUpdateSuccess(overrides = {}) { }; } +function makeCommercialUpdateSuccess(overrides = {}) { + return { + success: true, + stage: "update_applied", + updatedSituationGraph: { + centralStatement: + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified.", + currentSummary: "Commercial update summary", + activeUnknownNodeId: "n-other-people", + resolvedNodeIds: ["n-who"], + nodes: [ + { + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "provisional", + confidence: "medium", + }, + { + id: "n-who", + label: "Who experiences this problem", + description: + "Need to know who experiences this problem, because that must be clear before deciding whether it is commercially justified.", + kind: "unknown", + status: "resolved", + confidence: "medium", + value: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + parentId: "n-commercial-parent", + }, + { + id: "n-other-people", + label: "Whether other people experience this problem", + description: + "Need to know whether other people experience this problem, because that must be established before deciding whether the problem is broadly important.", + kind: "unknown", + status: "unknown", + confidence: "medium", + parentId: "n-commercial-parent", + }, + ], + edges: [ + { + id: "e-other-parent", + fromNodeId: "n-other-people", + toNodeId: "n-commercial-parent", + relationship: "depends_on", + confidence: "medium", + description: + "This child unknown must be investigated before the broader parent explanation can be resolved.", + }, + ], + }, + proposal: { + addedNodes: [], + updatedNodes: [{ nodeId: "n-who", newStatus: "resolved", reason: "answered" }], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: ["n-who"], + affectedNodeIds: ["n-commercial-parent"], + selectedQuestion: null, + }, + selectedQuestion: { + nodeId: "n-other-people", + question: "What makes you think other people experience this problem too?", + reason: + "Formulated as a direct foundational question because this child unknown should be answered one step at a time.", + reasoningPattern: "decision", + questionFamily: "decision_foundation", + }, + previousActiveUnknownNodeId: "n-who", + newActiveUnknownNodeId: "n-other-people", + affectedNodeIds: ["n-commercial-parent"], + resolvedUnknownNodeIds: ["n-who"], + diagnostics: { + unresolvedCandidateCount: 2, + eligibleCandidateCount: 1, + candidateNodeIds: ["n-other-people"], + resolvedCurrentTurnNodeIds: ["n-who"], + noQuestionReason: null, + }, + ...overrides, + }; +} + describe("scenario-form UI helpers", () => { it("submits to /api/cases/start", async () => { const fetchImpl = vi.fn().mockResolvedValue({ ok: true }); @@ -700,4 +788,16 @@ describe("graph-backed UI rendering", () => { expect(html).toContain("Proposal details"); }); + + it("does not show the no-question fallback when a commercial follow-up sibling exists", () => { + const html = renderToStaticMarkup( + , + ); + + expect(html).toContain( + "What makes you think other people experience this problem too?", + ); + expect(html).toContain("New active unknown"); + expect(html).not.toContain("No next question selected yet."); + }); }); \ No newline at end of file From c27320984cd439c119650592d2f9a204e10ee44d Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 12:10:57 +0100 Subject: [PATCH 07/11] feat: enforce reasoning pattern consistency --- docs/v0.7-question-simplicity-experiment.md | 18 + lib/graph/apply-proposal.js | 519 +++++++++++++++++- lib/graph/orchestrator.js | 63 +++ tests/graph/apply-proposal.test.js | 60 ++ tests/graph/orchestrator.test.js | 6 + .../reasoning-pattern-validation.test.js | 282 ++++++++++ 6 files changed, 931 insertions(+), 17 deletions(-) create mode 100644 tests/graph/reasoning-pattern-validation.test.js diff --git a/docs/v0.7-question-simplicity-experiment.md b/docs/v0.7-question-simplicity-experiment.md index 80865f0..d4aa04a 100644 --- a/docs/v0.7-question-simplicity-experiment.md +++ b/docs/v0.7-question-simplicity-experiment.md @@ -2,6 +2,8 @@ After every successful graph update, the full deterministic question-selection pipeline must run again whenever eligible unresolved unknowns remain. +The active reasoning pattern constrains which graph nodes may participate in reasoning. + That means the update path must not stop at graph mutation, child resolution, emergent unknown creation, decomposition, or upward propagation. It must continue through: ```text @@ -23,6 +25,22 @@ updated graph Returning no question is only valid when no eligible unresolved candidate remains, the case is complete, ambiguity cannot be safely resolved, or question formulation fails validation with an explicit deterministic reason. +## Graph validity vs reasoning-pattern validity + +These are separate requirements. + +- **Graph validity** means references, IDs, node shapes, and update semantics are structurally correct. +- **Reasoning-pattern validity** means selectable investigation nodes are compatible with the current reasoning mode. + +A graph can be structurally valid while still being reasoning-invalid. + +Example: a decision investigation may still contain an unresolved comparison-style node such as `How the two observations were measured`. That node is structurally well-formed, but it is not allowed to participate as an active investigation target unless the reasoning pattern has actually shifted into comparison, contradiction, or explanation work. + +The engine therefore needs both invariants: + +1. the graph must be structurally valid +2. every selectable unknown must be compatible with the active reasoning pattern + # v0.7 Question Simplicity Experiment ## Observed failure diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index c08786f..dc5c9b9 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -7,6 +7,7 @@ import { COMPARABILITY_REASONING_NODE_ID, formulateQuestion, formulateTieResolutionQuestion, + selectReasoningPattern, } from "./question-formulator.js"; import { graphUpdateSchema, @@ -1727,12 +1728,280 @@ function selectedQuestionBelongsToChild(graph, selectedQuestion) { return Boolean(findNodeById(graph, selectedQuestion.nodeId)?.parentId); } -function selectDecompositionChildCandidate(graph, parentNodeId) { - const childCandidates = findDirectChildUnknowns(graph, parentNodeId).filter( - (node) => - node.kind === "unknown" && - !["resolved", "contradicted"].includes(node.status), +const ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN = { + decision: ["decision", "definition"], + explanation: ["explanation", "comparison", "definition"], + contradiction: ["contradiction", "comparison", "explanation", "definition"], + definition: ["definition"], + diagnosis: ["diagnosis", "comparison", "definition"], + comparison: ["comparison", "definition"], + prioritisation: ["prioritisation", "decision", "definition"], +}; + +function determineActiveReasoningPattern(node, graph) { + if (!node || !graph) { + return { + pattern: null, + reason: "No active reasoning pattern could be determined.", + }; + } + + const nodesById = new Map((graph.nodes || []).map((item) => [item.id, item])); + let currentParentId = node.parentId; + while (currentParentId) { + const parentNode = nodesById.get(currentParentId); + if (!parentNode) break; + const parentSelection = selectReasoningPattern({ node: parentNode, graph }); + if (parentSelection.pattern && parentSelection.pattern !== "definition") { + return { + pattern: parentSelection.pattern, + reason: `Inherited active reasoning pattern from parent node because ${parentSelection.reason}`, + }; + } + currentParentId = parentNode.parentId; + } + + const selection = selectReasoningPattern({ node, graph }); + return { + pattern: selection.pattern, + reason: selection.reason, + }; +} + +function inferIntrinsicNodePattern(node, graph) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + const observationCount = (graph.nodes || []).filter( + (candidate) => + candidate.kind === "observation" && candidate.status === "supported", + ).length; + + if ( + /\b(define|definition|meaning|term|terminology|boundaries)\b/.test(text) + ) { + return "definition"; + } + + if ( + /\b(contradiction|contradict|conflict|inconsistent|mismatch|opposing)\b/.test( + text, + ) + ) { + return "contradiction"; + } + + if ( + /\b(two observations|measured|measurement|basis|scale|same period|different timing|comparable)\b/.test( + text, + ) + ) { + return "comparison"; + } + + if ( + observationCount >= 2 && + /\b(explain|explanation|what changed|difference between|divergence|moved differently)\b/.test( + text, + ) + ) { + return "explanation"; + } + + if ( + /\b(genuine problem|who experiences|other people experience|how often this problem happens|what happens when this problem is not resolved|how people deal with this problem today|actively look for help|would pay to solve this problem|commercially justified|commercial justification|business case|value|demand|audience|customer|user|alternative|alternatives)\b/.test( + text, + ) + ) { + return "decision"; + } + + return selectReasoningPattern({ node, graph }).pattern; +} + +function assessReasoningPatternCompatibility({ node, graph, activePattern }) { + if (!node || !activePattern) { + return { + compatible: true, + activePattern: activePattern ?? null, + nodePattern: null, + reason: "No active reasoning pattern constraint was applied.", + }; + } + + const nodePattern = inferIntrinsicNodePattern(node, graph); + const allowedPatterns = ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN[ + activePattern + ] ?? [activePattern]; + const compatible = allowedPatterns.includes(nodePattern); + + return { + compatible, + activePattern, + nodePattern, + allowedPatterns, + reason: compatible + ? `Node remains compatible because ${nodePattern} is allowed during ${activePattern} reasoning.` + : `Node is incompatible because ${nodePattern} is not allowed during ${activePattern} reasoning.`, + }; +} + +function isExplicitComparisonFamilyUnknown(node) { + const text = normaliseText(`${node?.label || ""} ${node?.description || ""}`); + return /\b(two observations|measured|measurement|basis|scale|same period|different timing|comparable)\b/.test( + text, ); +} + +function buildCompatibilityFailure(node, compatibility, reason) { + return { + nodeId: node?.id ?? null, + label: node?.label ?? null, + activePattern: compatibility?.activePattern ?? null, + nodePattern: compatibility?.nodePattern ?? null, + allowedPatterns: compatibility?.allowedPatterns ?? [], + rejectionReason: reason || compatibility?.reason || null, + }; +} + +function buildRejectedSelectionDiagnostics({ + node, + graph, + activePattern, + reason, +}) { + const compatibility = assessReasoningPatternCompatibility({ + node, + graph, + activePattern, + }); + + return { + incompatibleNodeIds: node?.id ? [node.id] : [], + compatibilityFailures: [ + buildCompatibilityFailure(node, compatibility, reason), + ], + }; +} + +function selectPatternCompatibleUnknownCandidate({ + graph, + resolvedNodeIds = [], + activePattern, + excludedNodeIds = [], +}) { + if (!activePattern) { + return selectActiveUnknownCandidate(graph, resolvedNodeIds); + } + + const excluded = new Set(excludedNodeIds || []); + const incompatibleNodeIds = (graph.nodes || []) + .filter( + (node) => + node.kind === "unknown" && + !excluded.has(node.id) && + !["resolved", "contradicted"].includes(node.status), + ) + .filter( + (node) => + !assessReasoningPatternCompatibility({ + node, + graph, + activePattern, + }).compatible, + ) + .map((node) => node.id); + + return selectActiveUnknownCandidate(graph, [ + ...new Set([ + ...(resolvedNodeIds || []), + ...incompatibleNodeIds, + ...excludedNodeIds, + ]), + ]); +} + +function collectPatternCompatibilityDiagnostics({ + graph, + activePattern, + candidateNodeIds = [], +}) { + if (!activePattern) { + return { + reasoningPatternValidation: { + activePattern: null, + valid: true, + reason: "No active reasoning pattern constraint was applied.", + }, + patternCompatibleNodeCount: 0, + incompatibleNodeIds: [], + compatibilityFailures: [], + graphReasoningIntegrity: "not_applicable", + }; + } + + const candidateSet = new Set(candidateNodeIds || []); + const compatibilityFailures = (graph.nodes || []) + .filter( + (node) => + node.kind === "unknown" && + candidateSet.has(node.id) && + !["resolved", "contradicted"].includes(node.status), + ) + .map((node) => ({ + node, + compatibility: assessReasoningPatternCompatibility({ + node, + graph, + activePattern, + }), + })) + .filter(({ compatibility }) => !compatibility.compatible) + .map(({ node, compatibility }) => + buildCompatibilityFailure(node, compatibility), + ); + + return { + reasoningPatternValidation: { + activePattern, + valid: compatibilityFailures.length === 0, + reason: + compatibilityFailures.length === 0 + ? `All selectable unknowns are compatible with ${activePattern} reasoning.` + : `Some selectable unknowns are incompatible with ${activePattern} reasoning.`, + }, + patternCompatibleNodeCount: + (candidateNodeIds || []).length - compatibilityFailures.length, + incompatibleNodeIds: compatibilityFailures.map((failure) => failure.nodeId), + compatibilityFailures, + graphReasoningIntegrity: + compatibilityFailures.length === 0 ? "valid" : "invalid", + }; +} + +function selectDecompositionChildCandidate( + graph, + parentNodeId, + activePattern = null, +) { + const childCandidates = findDirectChildUnknowns(graph, parentNodeId) + .filter( + (node) => + node.kind === "unknown" && + !["resolved", "contradicted"].includes(node.status), + ) + .filter((node) => { + if ( + activePattern === "decision" && + isExplicitComparisonFamilyUnknown(node) + ) { + return false; + } + const compatibility = assessReasoningPatternCompatibility({ + node, + graph, + activePattern, + }); + return compatibility.compatible; + }); if (childCandidates.length === 0) { return { status: "none", nodeId: null, tiedCandidateIds: [] }; @@ -1926,6 +2195,11 @@ function runDeterministicDecomposition({ let decompositionTriggeredByQuestionComplexity = false; let decompositionTriggeredByAnswerability = false; let selectedContainerUnknown = null; + let activeReasoningPattern = null; + let activeReasoningPatternReason = null; + let incompatibleNodeIds = []; + let compatibilityFailures = []; + let replacementActions = []; while (workingSelection?.status === "selected" && workingSelection?.nodeId) { const selectedNode = findNodeById(workingGraph, workingSelection.nodeId); @@ -1935,6 +2209,53 @@ function runDeterministicDecomposition({ break; } + if (!activeReasoningPattern) { + const activePatternSelection = determineActiveReasoningPattern( + selectedNode, + workingGraph, + ); + activeReasoningPattern = activePatternSelection.pattern; + activeReasoningPatternReason = activePatternSelection.reason; + } + + const selectedNodeCompatibility = assessReasoningPatternCompatibility({ + node: selectedNode, + graph: workingGraph, + activePattern: activeReasoningPattern, + }); + if (!selectedNodeCompatibility.compatible) { + incompatibleNodeIds = appendUniqueValue( + incompatibleNodeIds, + selectedNode.id, + ); + compatibilityFailures.push( + buildCompatibilityFailure( + selectedNode, + selectedNodeCompatibility, + "Selected unknown violated the active reasoning pattern.", + ), + ); + const replacementSelection = selectPatternCompatibleUnknownCandidate({ + graph: workingGraph, + resolvedNodeIds: workingGraph.resolvedNodeIds, + activePattern: activeReasoningPattern, + excludedNodeIds: [selectedNode.id], + }); + if (replacementSelection?.status === "selected") { + replacementActions.push({ + rejectedNodeId: selectedNode.id, + replacementNodeId: replacementSelection.nodeId, + reason: + "Replaced an incompatible active unknown with the next pattern-compatible candidate.", + }); + workingSelection = replacementSelection; + continue; + } + decompositionStoppedReason = + "No reasoning-pattern-compatible unknown remained available for selection."; + break; + } + const atomicityAssessment = assessUnknownAtomicity({ node: selectedNode, graph: workingGraph, @@ -1979,6 +2300,7 @@ function runDeterministicDecomposition({ const childSelection = selectDecompositionChildCandidate( workingGraph, selectedNode.id, + activeReasoningPattern, ); if (childSelection.status === "selected") { workingSelection = childSelection; @@ -2013,6 +2335,7 @@ function runDeterministicDecomposition({ selectedNode, workingGraph, decompositionDepth, + activeReasoningPattern, ); proposedChildCount = decomposition.proposedChildCount; @@ -2061,6 +2384,7 @@ function runDeterministicDecomposition({ workingSelection = selectDecompositionChildCandidate( workingGraph, selectedNode.id, + activeReasoningPattern, ); if (workingSelection?.status !== "selected") { @@ -2112,6 +2436,11 @@ function runDeterministicDecomposition({ decompositionTriggeredByQuestionComplexity, decompositionTriggeredByAnswerability, selectedContainerUnknown, + activeReasoningPattern, + activeReasoningPatternReason, + incompatibleNodeIds, + compatibilityFailures, + replacementActions, }; } @@ -2446,20 +2775,75 @@ export function applyValidatedProposal({ const resolvedCurrentTurnNodeIds = [ ...new Set(proposalSnapshot.resolvedUnknownNodeIds || []), ]; - deterministicSelection = isSelectableUnresolvedUnknown( + let postPropagationIncompatibleNodeIds = []; + let postPropagationCompatibilityFailures = []; + let postPropagationReplacementActions = []; + let activeUnknownIncompatibleNodeIds = []; + let activeUnknownCompatibilityFailures = []; + let activeUnknownReplacementActions = []; + const preservedSelectedChildNode = isSelectableUnresolvedUnknown( updatedSituationGraph, decompositionResult.selectedChildNodeId, ) - ? { - status: "selected", - nodeId: decompositionResult.selectedChildNodeId, - reason: - "Preserved the selected decomposition child because it remains unresolved after propagation.", - } - : selectActiveUnknownCandidate( + ? findNodeById( updatedSituationGraph, - updatedSituationGraph.resolvedNodeIds, - ); + decompositionResult.selectedChildNodeId, + ) + : null; + const preservedSelectedChildCompatibility = preservedSelectedChildNode + ? assessReasoningPatternCompatibility({ + node: preservedSelectedChildNode, + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + }) + : null; + + if ( + preservedSelectedChildNode && + preservedSelectedChildCompatibility?.compatible + ) { + deterministicSelection = { + status: "selected", + nodeId: decompositionResult.selectedChildNodeId, + reason: + "Preserved the selected decomposition child because it remains unresolved and reasoning-pattern-compatible after propagation.", + }; + } else { + if (preservedSelectedChildNode) { + const rejectionDiagnostics = buildRejectedSelectionDiagnostics({ + node: preservedSelectedChildNode, + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + reason: + "Preserved decomposition child violated the active reasoning pattern after propagation.", + }); + postPropagationIncompatibleNodeIds = + rejectionDiagnostics.incompatibleNodeIds; + postPropagationCompatibilityFailures = + rejectionDiagnostics.compatibilityFailures; + } + + deterministicSelection = selectPatternCompatibleUnknownCandidate({ + graph: updatedSituationGraph, + resolvedNodeIds: updatedSituationGraph.resolvedNodeIds, + activePattern: decompositionResult.activeReasoningPattern, + excludedNodeIds: preservedSelectedChildNode + ? [preservedSelectedChildNode.id] + : [], + }); + + if ( + preservedSelectedChildNode && + deterministicSelection?.status === "selected" + ) { + postPropagationReplacementActions.push({ + rejectedNodeId: preservedSelectedChildNode.id, + replacementNodeId: deterministicSelection.nodeId, + reason: + "Replaced a preserved decomposition child that violated reasoning-pattern consistency.", + }); + } + } if (deterministicSelection?.status === "ambiguous") { const orderedSiblingSelection = selectOrderedSiblingCandidate( @@ -2473,14 +2857,76 @@ export function applyValidatedProposal({ } } + const carriedActiveUnknownNode = previousActiveUnknownNodeId + ? findNodeById(updatedSituationGraph, previousActiveUnknownNodeId) + : null; + const carriedActiveUnknownStillUnresolved = Boolean( + carriedActiveUnknownNode && + carriedActiveUnknownNode.kind === "unknown" && + !["resolved", "contradicted"].includes(carriedActiveUnknownNode.status) && + !updatedSituationGraph.resolvedNodeIds.includes( + carriedActiveUnknownNode.id, + ), + ); + + if ( + carriedActiveUnknownStillUnresolved && + decompositionResult.activeReasoningPattern + ) { + const carriedActiveCompatibility = assessReasoningPatternCompatibility({ + node: carriedActiveUnknownNode, + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + }); + + if (!carriedActiveCompatibility.compatible) { + activeUnknownIncompatibleNodeIds = [carriedActiveUnknownNode.id]; + activeUnknownCompatibilityFailures = [ + buildCompatibilityFailure( + carriedActiveUnknownNode, + carriedActiveCompatibility, + "Carried active unknown violated the active reasoning pattern and was not retained for investigation.", + ), + ]; + + if ( + deterministicSelection?.status === "selected" && + deterministicSelection.nodeId !== carriedActiveUnknownNode.id + ) { + activeUnknownReplacementActions = [ + { + rejectedNodeId: carriedActiveUnknownNode.id, + replacementNodeId: deterministicSelection.nodeId, + reason: + "Replaced an incompatible carried active unknown with a pattern-compatible investigation target.", + }, + ]; + } + } + } + const unresolvedCandidates = listUnresolvedUnknownCandidates( updatedSituationGraph, resolvedCurrentTurnNodeIds, ); - const eligibleCandidates = listEligibleUnknownCandidates( + const eligibleCandidatesBeforeCompatibility = listEligibleUnknownCandidates( updatedSituationGraph, resolvedCurrentTurnNodeIds, ); + const eligibleCandidates = eligibleCandidatesBeforeCompatibility.filter( + (node) => + assessReasoningPatternCompatibility({ + node, + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + }).compatible, + ); + + const compatibilityDiagnostics = collectPatternCompatibilityDiagnostics({ + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + candidateNodeIds: eligibleCandidates.map((node) => node.id), + }); const atomicityAssessment = decompositionResult.atomicityAssessment; const answerabilityAssessment = decompositionResult.answerabilityAssessment; @@ -2603,6 +3049,14 @@ export function applyValidatedProposal({ } : null; + const finalSelectionCompatibility = finalSelectedQuestion?.nodeId + ? assessReasoningPatternCompatibility({ + node: findNodeById(updatedSituationGraph, finalSelectedQuestion.nodeId), + graph: updatedSituationGraph, + activePattern: decompositionResult.activeReasoningPattern, + }) + : null; + const finalSelectedChildNodeId = selectedChildNodeId ?? (selectedQuestionBelongsToChild( @@ -2638,7 +3092,8 @@ export function applyValidatedProposal({ !resultGraphValidation.success || !resultReferenceValidation?.valid || resultDuplicateNodeIds.length > 0 || - resultDuplicateEdgeIds.length > 0 + resultDuplicateEdgeIds.length > 0 || + (finalSelectedQuestion?.nodeId && !finalSelectionCompatibility?.compatible) ) { return { success: false, @@ -2658,6 +3113,12 @@ export function applyValidatedProposal({ ({ edgeId, count }) => `Updated graph contains duplicate edge ID: "${edgeId}" (${count} occurrences)`, ), + ...(!finalSelectionCompatibility?.compatible && + finalSelectedQuestion?.nodeId + ? [ + `Active unknown violates reasoning pattern consistency: "${finalSelectedQuestion.nodeId}" is ${finalSelectionCompatibility?.nodePattern} but active pattern is ${finalSelectionCompatibility?.activePattern}`, + ] + : []), ], }; } @@ -2737,6 +3198,30 @@ export function applyValidatedProposal({ childNodeIds: decompositionChildNodeIds, selectedContainerUnknown: decompositionResult.selectedContainerUnknown ?? null, + reasoningPatternValidation: + compatibilityDiagnostics.reasoningPatternValidation, + patternCompatibleNodeCount: + compatibilityDiagnostics.patternCompatibleNodeCount, + incompatibleNodeIds: [ + ...new Set([ + ...(decompositionResult.incompatibleNodeIds || []), + ...postPropagationIncompatibleNodeIds, + ...activeUnknownIncompatibleNodeIds, + ...compatibilityDiagnostics.incompatibleNodeIds, + ]), + ], + compatibilityFailures: [ + ...(decompositionResult.compatibilityFailures || []), + ...postPropagationCompatibilityFailures, + ...activeUnknownCompatibilityFailures, + ...compatibilityDiagnostics.compatibilityFailures, + ], + replacementActions: [ + ...(decompositionResult.replacementActions || []), + ...postPropagationReplacementActions, + ...activeUnknownReplacementActions, + ], + graphReasoningIntegrity: compatibilityDiagnostics.graphReasoningIntegrity, selectedChildUnknown: finalSelectedChildNodeId ?? (deterministicSelection?.status === "selected" diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 731d2d6..20164e7 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -65,6 +65,12 @@ function buildDiagnostics({ rejectedQuestionFamilies, selectedQuestionTemplate, reasoningPatternReason, + reasoningPatternValidation, + patternCompatibleNodeCount, + incompatibleNodeIds, + compatibilityFailures, + replacementActions, + graphReasoningIntegrity, }) { return { promptVersion: analysis?.promptVersion ?? null, @@ -100,6 +106,12 @@ function buildDiagnostics({ rejectedQuestionFamilies: rejectedQuestionFamilies ?? [], selectedQuestionTemplate: selectedQuestionTemplate ?? null, reasoningPatternReason: reasoningPatternReason ?? null, + reasoningPatternValidation: reasoningPatternValidation ?? null, + patternCompatibleNodeCount: patternCompatibleNodeCount ?? 0, + incompatibleNodeIds: incompatibleNodeIds ?? [], + compatibilityFailures: compatibilityFailures ?? [], + replacementActions: replacementActions ?? [], + graphReasoningIntegrity: graphReasoningIntegrity ?? null, }; } @@ -200,6 +212,12 @@ function buildUpdateDiagnostics({ candidateNodeIds, resolvedCurrentTurnNodeIds, noQuestionReason, + reasoningPatternValidation, + patternCompatibleNodeCount, + incompatibleNodeIds, + compatibilityFailures, + replacementActions, + graphReasoningIntegrity, }) { return { promptVersion: promptVersion ?? "v0.4", @@ -291,6 +309,12 @@ function buildUpdateDiagnostics({ candidateNodeIds: candidateNodeIds ?? [], resolvedCurrentTurnNodeIds: resolvedCurrentTurnNodeIds ?? [], noQuestionReason: noQuestionReason ?? null, + reasoningPatternValidation: reasoningPatternValidation ?? null, + patternCompatibleNodeCount: patternCompatibleNodeCount ?? 0, + incompatibleNodeIds: incompatibleNodeIds ?? [], + compatibilityFailures: compatibilityFailures ?? [], + replacementActions: replacementActions ?? [], + graphReasoningIntegrity: graphReasoningIntegrity ?? null, unknownSelectionExplanation: unknownSelectionExplanation ?? null, }; } @@ -419,6 +443,16 @@ export async function startCase(body) { reasoningPatternReason: initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? null, + reasoningPatternValidation: + initialQuestionResult.reasoningPatternValidation ?? null, + patternCompatibleNodeCount: + initialQuestionResult.patternCompatibleNodeCount ?? 0, + incompatibleNodeIds: initialQuestionResult.incompatibleNodeIds ?? [], + compatibilityFailures: + initialQuestionResult.compatibilityFailures ?? [], + replacementActions: initialQuestionResult.replacementActions ?? [], + graphReasoningIntegrity: + initialQuestionResult.graphReasoningIntegrity ?? null, }), validationErrors: graphReferenceValidation.errors, statusCode: 500, @@ -475,6 +509,15 @@ export async function startCase(body) { null, reasoningPatternReason: initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? null, + reasoningPatternValidation: + initialQuestionResult.reasoningPatternValidation ?? null, + patternCompatibleNodeCount: + initialQuestionResult.patternCompatibleNodeCount ?? 0, + incompatibleNodeIds: initialQuestionResult.incompatibleNodeIds ?? [], + compatibilityFailures: initialQuestionResult.compatibilityFailures ?? [], + replacementActions: initialQuestionResult.replacementActions ?? [], + graphReasoningIntegrity: + initialQuestionResult.graphReasoningIntegrity ?? null, }), }; } @@ -675,6 +718,12 @@ async function updateCaseWithDependencies(body, dependencies = {}) { candidateNodeIds: [], resolvedCurrentTurnNodeIds: [], noQuestionReason: null, + reasoningPatternValidation: null, + patternCompatibleNodeCount: 0, + incompatibleNodeIds: [], + compatibilityFailures: [], + replacementActions: [], + graphReasoningIntegrity: null, plainLanguageNormalisations: [], unknownSelectionExplanation: explainUnknownSelection( situationGraph, @@ -780,6 +829,14 @@ async function updateCaseWithDependencies(body, dependencies = {}) { resolvedCurrentTurnNodeIds: applicationResult.resolvedCurrentTurnNodeIds, noQuestionReason: applicationResult.noQuestionReason, + reasoningPatternValidation: + applicationResult.reasoningPatternValidation, + patternCompatibleNodeCount: + applicationResult.patternCompatibleNodeCount, + incompatibleNodeIds: applicationResult.incompatibleNodeIds, + compatibilityFailures: applicationResult.compatibilityFailures, + replacementActions: applicationResult.replacementActions, + graphReasoningIntegrity: applicationResult.graphReasoningIntegrity, plainLanguageNormalisations: applicationResult.plainLanguageNormalisations, reasoningPattern: @@ -876,6 +933,12 @@ async function updateCaseWithDependencies(body, dependencies = {}) { candidateNodeIds: [], resolvedCurrentTurnNodeIds: [], noQuestionReason: null, + reasoningPatternValidation: null, + patternCompatibleNodeCount: 0, + incompatibleNodeIds: [], + compatibilityFailures: [], + replacementActions: [], + graphReasoningIntegrity: null, plainLanguageNormalisations: [], reasoningPattern: null, questionFamily: null, diff --git a/tests/graph/apply-proposal.test.js b/tests/graph/apply-proposal.test.js index 33400e6..acddddb 100644 --- a/tests/graph/apply-proposal.test.js +++ b/tests/graph/apply-proposal.test.js @@ -1434,5 +1434,65 @@ describe("applyValidatedProposal", () => { expect(secondResult.selectedQuestion?.question.toLowerCase()).not.toMatch( /price|budget|market size|pilot metrics|benchmark|technical differentiation/, ); + expect(secondResult.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(secondResult.graphReasoningIntegrity).toBe("valid"); + expect(secondResult.incompatibleNodeIds).toEqual([]); + expect(secondResult.compatibilityFailures).toEqual([]); + }); + + it("does not allow a decision-mode active unknown to remain a comparison child", () => { + const graph = makeCommercialUpdateFixture(); + graph.nodes.push( + makeNode({ + id: "n-commercial-comparison-child", + label: "How the two observations were measured", + description: + "Need evidence about the measure used for each observation before comparing them.", + kind: "unknown", + status: "unknown", + confidence: "medium", + parentId: "n-commercial-parent", + }), + ); + graph.activeUnknownNodeId = "n-commercial-comparison-child"; + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeMeaningfulNoOpProposal(), + }); + + expect(result.success).toBe(true); + expect(result.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(result.graphReasoningIntegrity).toBe("valid"); + expect(result.incompatibleNodeIds).toContain( + "n-commercial-comparison-child", + ); + expect(result.compatibilityFailures).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + nodeId: "n-commercial-comparison-child", + activePattern: "decision", + nodePattern: "comparison", + }), + ]), + ); + expect(result.replacementActions).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + rejectedNodeId: "n-commercial-comparison-child", + replacementNodeId: result.selectedQuestion?.nodeId, + }), + ]), + ); + expect(result.selectedQuestion?.nodeId).not.toBe( + "n-commercial-comparison-child", + ); + expect(result.selectedQuestion?.reasoningPattern).toBe("decision"); }); }); diff --git a/tests/graph/orchestrator.test.js b/tests/graph/orchestrator.test.js index 128b124..c1a4020 100644 --- a/tests/graph/orchestrator.test.js +++ b/tests/graph/orchestrator.test.js @@ -1274,6 +1274,12 @@ describe("lib/graph/orchestrator startCase", () => { resolvedFirstChildNodeId, ); expect(initial.diagnostics.noQuestionReason).toBeNull(); + expect(initial.diagnostics.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(initial.diagnostics.graphReasoningIntegrity).toBe("valid"); + expect(initial.diagnostics.incompatibleNodeIds).toEqual([]); expect(initial.selectedQuestion?.question.toLowerCase()).not.toMatch( /price|budget|market size|pilot metrics|benchmark|technical differentiation/, ); diff --git a/tests/graph/reasoning-pattern-validation.test.js b/tests/graph/reasoning-pattern-validation.test.js new file mode 100644 index 0000000..772d1b7 --- /dev/null +++ b/tests/graph/reasoning-pattern-validation.test.js @@ -0,0 +1,282 @@ +import { describe, expect, it } from "vitest"; +import { applyValidatedProposal } from "@/lib/graph/apply-proposal.js"; +import { + classifyObservationRelationship, + formulateQuestion, + selectReasoningPattern, +} from "@/lib/graph/question-formulator.js"; +import { makeGraph, makeNode } from "@/lib/graph/schema.js"; + +const COMMERCIAL_SCENARIO = + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified."; + +function makeCommercialGraph() { + const parent = makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem, whether people would value it enough to pay for it, and whether it is commercially justified before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + + return makeGraph({ + centralStatement: COMMERCIAL_SCENARIO, + nodes: [parent], + edges: [], + activeUnknownNodeId: parent.id, + resolvedNodeIds: [], + currentSummary: "Commercial pattern fixture", + }); +} + +function makeSeedProposal() { + return { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }; +} + +describe("reasoning-pattern validation", () => { + it("keeps the commercial-method scenario in decision mode without comparability-style unknowns", () => { + const seeded = applyValidatedProposal({ + situationGraph: makeCommercialGraph(), + proposal: makeSeedProposal(), + }); + + expect(seeded.success).toBe(true); + expect(seeded.selectedQuestion?.reasoningPattern).toBe("decision"); + expect( + seeded.updatedSituationGraph.nodes.some((node) => + /two observations|measured|different timing/i.test(node.label), + ), + ).toBe(false); + + const followUp = applyValidatedProposal({ + situationGraph: seeded.updatedSituationGraph, + proposal: { + addedNodes: [], + updatedNodes: [ + { + nodeId: seeded.selectedQuestion.nodeId, + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + reason: "Answered by the user.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [seeded.selectedQuestion.nodeId], + affectedNodeIds: [], + selectedQuestion: null, + }, + previousQuestion: seeded.selectedQuestion.question, + answer: + "I experience it myself when I am trying to decide whether a project, idea or investment is justified, but I do not yet know how common that problem is for other people.", + }); + + expect(followUp.success).toBe(true); + expect(followUp.selectedQuestion?.reasoningPattern).toBe("decision"); + expect(followUp.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(followUp.graphReasoningIntegrity).toBe("valid"); + expect(followUp.incompatibleNodeIds).toEqual([]); + expect(followUp.selectedQuestion?.question.toLowerCase()).not.toMatch( + /two observations|measured|same basis|same scale|different timing/, + ); + }); + + it("allows comparability-style reasoning in an explanation scenario", () => { + const unknown = makeNode({ + id: "n-explanation", + label: + "Explanation for why revenue increased by 18%, but cash in the bank fell over the same period", + description: + "Need to understand what change or event could explain why these observations differ.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraph({ + centralStatement: + "Revenue increased by 18%, but cash in the bank fell over the same period.", + nodes: [ + unknown, + makeNode({ + id: "n-revenue", + label: "Revenue increased by 18%.", + description: "Revenue increased by 18%.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-cash", + label: "Cash in the bank fell over the same period.", + description: "Cash in the bank fell over the same period.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + edges: [], + activeUnknownNodeId: unknown.id, + resolvedNodeIds: [], + currentSummary: "Explanation fixture", + }); + + const pattern = selectReasoningPattern({ node: unknown, graph }); + const question = formulateQuestion({ node: unknown, graph }); + const relationship = classifyObservationRelationship(graph); + + expect(pattern.pattern).toBe("explanation"); + expect(question.reasoningPattern).toBe("explanation"); + expect(relationship.questionRequired).toBe(true); + }); + + it("rejects explanation-family tie resolution for duplicate observations", () => { + const graph = makeGraph({ + centralStatement: "Sales doubled. Sales doubled.", + nodes: [ + makeNode({ + id: "n-sales-1", + label: "Sales doubled.", + description: "Sales doubled.", + kind: "observation", + status: "supported", + confidence: "high", + }), + makeNode({ + id: "n-sales-2", + label: "Sales doubled.", + description: "Sales doubled.", + kind: "observation", + status: "supported", + confidence: "high", + }), + ], + edges: [], + activeUnknownNodeId: null, + resolvedNodeIds: [], + currentSummary: "Duplicate observation fixture", + }); + + const relationship = classifyObservationRelationship(graph); + + expect(relationship.relationshipStatus).toBe("duplicate"); + expect(relationship.questionRequired).toBe(false); + }); + + it("keeps definition scenarios inside compatible node families", () => { + const unknown = makeNode({ + id: "n-definition", + label: "Definition of justified confidence", + description: "The term needs clearer boundaries.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }); + const graph = makeGraph({ + centralStatement: "The team uses justified confidence inconsistently.", + nodes: [unknown], + edges: [], + activeUnknownNodeId: unknown.id, + resolvedNodeIds: [], + currentSummary: "Definition fixture", + }); + + const pattern = selectReasoningPattern({ node: unknown, graph }); + const result = formulateQuestion({ node: unknown, graph }); + + expect(pattern.pattern).toBe("definition"); + expect(result.reasoningPattern).toBe("definition"); + expect(result.allowedQuestionFamilies).toEqual(["definition"]); + }); + + it("replaces an incompatible decision-mode active unknown with a pattern-compatible candidate", () => { + const graph = makeGraph({ + centralStatement: + "Before investing more, we need to know whether continuing development is commercially justified.", + nodes: [ + makeNode({ + id: "n-commercial-parent", + label: + "Commercial justification for whether continuing development is commercially justified", + description: + "Need to know whether this solves a genuine problem before continuing development.", + kind: "unknown", + status: "unknown", + confidence: "medium", + }), + makeNode({ + id: "n-incompatible-child", + label: "How the two observations were measured", + description: + "Need evidence about the measure used for each observation before comparing them.", + kind: "unknown", + status: "unknown", + confidence: "medium", + parentId: "n-commercial-parent", + }), + ], + edges: [], + activeUnknownNodeId: "n-incompatible-child", + resolvedNodeIds: [], + currentSummary: "Incompatible active unknown fixture", + }); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: makeSeedProposal(), + }); + + expect(result.success).toBe(true); + expect(result.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(result.graphReasoningIntegrity).toBe("valid"); + expect(result.incompatibleNodeIds).toContain("n-incompatible-child"); + expect(result.compatibilityFailures).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + nodeId: "n-incompatible-child", + activePattern: "decision", + nodePattern: "comparison", + }), + ]), + ); + expect(result.replacementActions).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + rejectedNodeId: "n-incompatible-child", + replacementNodeId: result.selectedQuestion?.nodeId, + }), + ]), + ); + expect(result.selectedQuestion?.nodeId).not.toBe("n-incompatible-child"); + expect(result.selectedQuestion?.reasoningPattern).toBe("decision"); + }); +}); From 34c25fcb43d1e2bb6fc57da6439b5eba94925dcf Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 12:39:01 +0100 Subject: [PATCH 08/11] fix: reselect after reasoning pattern filtering --- lib/graph/apply-proposal.js | 212 +++++++++++++++++++++++++------ lib/graph/orchestrator.js | 37 +++++- tests/graph/orchestrator.test.js | 76 +++++++++++ tests/ui/scenario-form.test.jsx | 38 ++++++ 4 files changed, 317 insertions(+), 46 deletions(-) diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index dc5c9b9..65529df 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -2038,6 +2038,137 @@ function selectDecompositionChildCandidate( }; } +function buildSelectedQuestionResult({ + updatedSituationGraph, + deterministicSelection, +}) { + const selectedNode = + deterministicSelection?.status === "selected" + ? findNodeById(updatedSituationGraph, deterministicSelection.nodeId) + : null; + const formulatedQuestion = selectedNode + ? formulateQuestion({ + node: selectedNode, + graph: updatedSituationGraph, + context: { selectionState: deterministicSelection }, + }) + : null; + + const selectedQuestion = + deterministicSelection?.status === "ambiguous" + ? { + id: "q_tie_resolution", + ...formulateTieResolutionQuestion({ graph: updatedSituationGraph }), + nodeId: null, + tiedCandidateIds: deterministicSelection.tiedCandidateIds, + } + : deterministicSelection?.status === "selected" && formulatedQuestion + ? { + nodeId: deterministicSelection.nodeId, + question: + formulatedQuestion.question || deterministicSelection.question, + reason: formulatedQuestion.reason, + strategy: formulatedQuestion.strategy, + investigationStrategy: formulatedQuestion.investigationStrategy, + reasoningPattern: formulatedQuestion.reasoningPattern, + reasoningPatternReason: formulatedQuestion.reasoningPatternReason, + questionFamily: formulatedQuestion.questionFamily, + allowedQuestionFamilies: formulatedQuestion.allowedQuestionFamilies, + rejectedQuestionFamilies: + formulatedQuestion.rejectedQuestionFamilies, + selectedQuestionTemplate: + formulatedQuestion.selectedQuestionTemplate, + questionComplexity: formulatedQuestion.questionComplexity, + plainLanguageNormalisations: + formulatedQuestion.plainLanguageNormalisations, + } + : null; + + return { + selectedNode, + formulatedQuestion, + selectedQuestion, + }; +} + +function resolveAmbiguousGraphBackedSelection({ + graphSnapshot, + updatedSituationGraph, + deterministicSelection, +}) { + const orderedCandidateIds = + deterministicSelection?.displayOrder || + deterministicSelection?.tiedCandidateIds || + []; + + for (const candidateNodeId of orderedCandidateIds) { + if ( + !isSelectableUnresolvedUnknown(updatedSituationGraph, candidateNodeId) + ) { + continue; + } + + const candidateResult = runDeterministicDecomposition({ + graphSnapshot, + proposalSnapshot: { + addedNodes: [], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }, + updatedSituationGraph: cloneJsonSafe(updatedSituationGraph), + reasoningResolution: { reasoningStateOverride: {} }, + deterministicSelection: { + status: "selected", + nodeId: candidateNodeId, + reason: + "Selected this tied candidate for deterministic decomposition-based reselection.", + }, + }); + + if (!candidateResult.success) { + continue; + } + + const nextGraph = candidateResult.updatedSituationGraph; + nextGraph.reasoningState = buildReasoningState(nextGraph); + const nextSelection = isSelectableUnresolvedUnknown( + nextGraph, + candidateResult.selectedChildNodeId, + ) + ? { + status: "selected", + nodeId: candidateResult.selectedChildNodeId, + reason: + "Selected the preserved decomposition child after resolving an initial tie.", + } + : candidateResult.deterministicSelection; + + const questionResult = buildSelectedQuestionResult({ + updatedSituationGraph: nextGraph, + deterministicSelection: nextSelection, + }); + + if (questionResult.selectedQuestion?.question) { + nextGraph.activeUnknownNodeId = + nextSelection?.status === "selected" ? nextSelection.nodeId : null; + nextGraph.currentSummary = describeGraph(nextGraph); + + return { + ...candidateResult, + updatedSituationGraph: nextGraph, + deterministicSelection: nextSelection, + ...questionResult, + }; + } + } + + return null; +} + export function determineGraphBackedQuestion({ situationGraph }) { const graphSnapshot = cloneJsonSafe(situationGraph); let updatedSituationGraph = cloneJsonSafe(situationGraph); @@ -2090,52 +2221,48 @@ export function determineGraphBackedQuestion({ situationGraph }) { : null; updatedSituationGraph.currentSummary = describeGraph(updatedSituationGraph); - const selectedNode = - deterministicSelection?.status === "selected" - ? findNodeById(updatedSituationGraph, deterministicSelection.nodeId) - : null; - const formulatedQuestion = selectedNode - ? formulateQuestion({ - node: selectedNode, - graph: updatedSituationGraph, - context: { selectionState: deterministicSelection }, - }) - : null; + let questionResult = buildSelectedQuestionResult({ + updatedSituationGraph, + deterministicSelection, + }); + + if ( + deterministicSelection?.status === "ambiguous" && + !questionResult.selectedQuestion?.question + ) { + const reselectionResult = resolveAmbiguousGraphBackedSelection({ + graphSnapshot, + updatedSituationGraph, + deterministicSelection, + }); + + if (reselectionResult) { + updatedSituationGraph = reselectionResult.updatedSituationGraph; + deterministicSelection = reselectionResult.deterministicSelection; + questionResult = reselectionResult; + } + } + + const noQuestionReason = questionResult.selectedQuestion?.question + ? null + : deterministicSelection?.status === "ambiguous" + ? questionResult.selectedQuestion?.questionSuppressedReason || + questionResult.selectedQuestion?.reason || + "Eligible unresolved candidates remain tied after initial graph-backed selection." + : (updatedSituationGraph.nodes || []).some( + (node) => + node.kind === "unknown" && + !["resolved", "contradicted"].includes(node.status) && + !(updatedSituationGraph.resolvedNodeIds || []).includes(node.id), + ) + ? "Compatible unresolved candidates remain, but none produced a valid graph-backed question." + : "No unresolved unknown candidates remain after initial graph construction."; return { success: true, updatedSituationGraph, deterministicSelection, - selectedQuestion: - deterministicSelection?.status === "ambiguous" - ? { - id: "q_tie_resolution", - ...formulateTieResolutionQuestion({ graph: updatedSituationGraph }), - nodeId: null, - tiedCandidateIds: deterministicSelection.tiedCandidateIds, - } - : deterministicSelection?.status === "selected" && formulatedQuestion - ? { - nodeId: deterministicSelection.nodeId, - question: - formulatedQuestion.question || deterministicSelection.question, - reason: formulatedQuestion.reason, - strategy: formulatedQuestion.strategy, - investigationStrategy: formulatedQuestion.investigationStrategy, - reasoningPattern: formulatedQuestion.reasoningPattern, - reasoningPatternReason: formulatedQuestion.reasoningPatternReason, - questionFamily: formulatedQuestion.questionFamily, - allowedQuestionFamilies: - formulatedQuestion.allowedQuestionFamilies, - rejectedQuestionFamilies: - formulatedQuestion.rejectedQuestionFamilies, - selectedQuestionTemplate: - formulatedQuestion.selectedQuestionTemplate, - questionComplexity: formulatedQuestion.questionComplexity, - plainLanguageNormalisations: - formulatedQuestion.plainLanguageNormalisations, - } - : null, + selectedQuestion: questionResult.selectedQuestion, atomicityAssessment: decompositionResult.atomicityAssessment, answerabilityAssessment: decompositionResult.answerabilityAssessment, independentlyAnswerable: @@ -2160,7 +2287,8 @@ export function determineGraphBackedQuestion({ situationGraph }) { selectedUnknownBefore: decompositionResult.selectedUnknownBefore, selectedUnknownAfter: deterministicSelection?.nodeId ?? null, questionComplexityAssessment: - formulatedQuestion?.questionComplexity ?? null, + questionResult.formulatedQuestion?.questionComplexity ?? null, + noQuestionReason, }; } diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 20164e7..0d62639 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -71,6 +71,7 @@ function buildDiagnostics({ compatibilityFailures, replacementActions, graphReasoningIntegrity, + noQuestionReason, }) { return { promptVersion: analysis?.promptVersion ?? null, @@ -112,6 +113,28 @@ function buildDiagnostics({ compatibilityFailures: compatibilityFailures ?? [], replacementActions: replacementActions ?? [], graphReasoningIntegrity: graphReasoningIntegrity ?? null, + noQuestionReason: noQuestionReason ?? null, + }; +} + +function fallbackStartCaseReasoningPatternValidation( + selectedQuestion, + existingValidation, +) { + if (existingValidation) { + return existingValidation; + } + + if (!selectedQuestion?.reasoningPattern) { + return null; + } + + return { + activePattern: selectedQuestion.reasoningPattern, + valid: Boolean(selectedQuestion.question), + reason: selectedQuestion.question + ? "Initial graph-backed selection produced a reasoning-pattern-compatible question." + : "Initial graph-backed selection did not produce a valid question for the inferred reasoning pattern.", }; } @@ -443,8 +466,10 @@ export async function startCase(body) { reasoningPatternReason: initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? null, - reasoningPatternValidation: - initialQuestionResult.reasoningPatternValidation ?? null, + reasoningPatternValidation: fallbackStartCaseReasoningPatternValidation( + initialQuestionResult.selectedQuestion, + initialQuestionResult.reasoningPatternValidation, + ), patternCompatibleNodeCount: initialQuestionResult.patternCompatibleNodeCount ?? 0, incompatibleNodeIds: initialQuestionResult.incompatibleNodeIds ?? [], @@ -453,6 +478,7 @@ export async function startCase(body) { replacementActions: initialQuestionResult.replacementActions ?? [], graphReasoningIntegrity: initialQuestionResult.graphReasoningIntegrity ?? null, + noQuestionReason: initialQuestionResult.noQuestionReason ?? null, }), validationErrors: graphReferenceValidation.errors, statusCode: 500, @@ -509,8 +535,10 @@ export async function startCase(body) { null, reasoningPatternReason: initialQuestionResult.selectedQuestion?.reasoningPatternReason ?? null, - reasoningPatternValidation: - initialQuestionResult.reasoningPatternValidation ?? null, + reasoningPatternValidation: fallbackStartCaseReasoningPatternValidation( + initialQuestionResult.selectedQuestion, + initialQuestionResult.reasoningPatternValidation, + ), patternCompatibleNodeCount: initialQuestionResult.patternCompatibleNodeCount ?? 0, incompatibleNodeIds: initialQuestionResult.incompatibleNodeIds ?? [], @@ -518,6 +546,7 @@ export async function startCase(body) { replacementActions: initialQuestionResult.replacementActions ?? [], graphReasoningIntegrity: initialQuestionResult.graphReasoningIntegrity ?? null, + noQuestionReason: initialQuestionResult.noQuestionReason ?? null, }), }; } diff --git a/tests/graph/orchestrator.test.js b/tests/graph/orchestrator.test.js index c1a4020..af784fc 100644 --- a/tests/graph/orchestrator.test.js +++ b/tests/graph/orchestrator.test.js @@ -92,6 +92,49 @@ function makeCommercialAnalysisResult(overrides = {}) { }); } +function makeCommercialTieAnalysisResult(overrides = {}) { + return makeAnalysisResult({ + reconstruction: { + summary: + "A new reasoning method may become a commercial product, but multiple broad decision unknowns remain unresolved.", + actors: [], + systemsOrObjects: [], + expectedStates: [], + observedStates: [], + differences: [], + knownTransitions: [], + unexplainedTransitions: [], + contradictions: [], + importantUnknowns: [ + { + id: "unk-fit-pay", + label: + "Evidence of genuine problem-solution fit and actual willingness to pay among target users", + description: + "Need to know whether there is real problem-solution fit and willingness to pay among target users before continuing development.", + confidence: "high", + }, + { + id: "unk-distinction", + label: + "Clear, measurable distinction between the method and existing AI tools that justifies separate commercial value", + description: + "Need to know whether there is a clear measurable distinction from existing AI tools before continuing development.", + confidence: "high", + }, + ], + plausibleInterpretations: [], + }, + nextQuestion: { + id: "q-commercial-tie", + question: + "What specific validation metrics, pilot feedback, or competitive benchmarking results have you collected?", + reason: "Model-proposed broad validation question", + }, + ...overrides, + }); +} + function makeCommercialUpdateGraph() { const parent = makeNode({ id: "n-commercial-parent", @@ -512,6 +555,39 @@ describe("lib/graph/orchestrator startCase", () => { ); }); + it("reselects and decomposes a tied commercial start-case candidate instead of returning a silent null question", async () => { + mockAnalyseScenario.mockResolvedValue(makeCommercialTieAnalysisResult()); + const { startCase } = await import("@/lib/graph/orchestrator.js"); + + const result = await startCase({ + scenario: + "I have developed a new reasoning method that aims to help people determine whether they have enough justified confidence to make a decision. I believe it could become a commercial product, but I do not yet know whether it solves a genuine problem, whether people would value it enough to pay for it, or whether it is fundamentally different from existing AI tools. Before investing significant time and money into building it further, I want to determine whether continuing development is commercially justified.", + }); + + expect(result.success).toBe(true); + expect( + result.situationGraph.nodes.some((node) => node.kind === "unknown"), + ).toBe(true); + expect(result.selectedQuestion).not.toBeNull(); + expect(result.selectedQuestion?.question).toBe( + "Who experiences this problem?", + ); + expect(result.selectedQuestion?.nodeId).toBe( + result.situationGraph.activeUnknownNodeId, + ); + expect(result.diagnostics.noQuestionReason).toBeNull(); + expect(result.diagnostics.selectedContainerUnknown).toBeTruthy(); + expect(result.diagnostics.selectedChildUnknown).toBeTruthy(); + expect(result.diagnostics.decompositionApplied).toBe(true); + expect(result.diagnostics.reasoningPatternValidation).toMatchObject({ + activePattern: "decision", + valid: true, + }); + expect(result.selectedQuestion?.question.toLowerCase()).not.toMatch( + /pay|price|pricing|budget|benchmark/, + ); + }); + it("includes compatibility diagnostics when provided by analysis", async () => { mockAnalyseScenario.mockResolvedValue( makeAnalysisResult({ diff --git a/tests/ui/scenario-form.test.jsx b/tests/ui/scenario-form.test.jsx index 9a879f8..e3d13b5 100644 --- a/tests/ui/scenario-form.test.jsx +++ b/tests/ui/scenario-form.test.jsx @@ -472,6 +472,44 @@ describe("graph-backed UI rendering", () => { expect(html).not.toContain("Update situation"); }); + it("renders the initial graph-backed commercial question when start-case reselection succeeds", () => { + const html = renderToStaticMarkup( + , + ); + + expect(html).toContain("Who experiences this problem?"); + expect(html).toContain("Selected Question"); + }); + it("renders diagnostics", () => { const html = renderToStaticMarkup( , From fe6a9925cb225f3971b43a3bf90b4634827b5b35 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 13:55:39 +0100 Subject: [PATCH 09/11] fix: stabilise multi-turn question progression --- lib/graph/apply-proposal.js | 242 +++++++++++++++++++++++++++-- lib/graph/utils.js | 2 +- tests/graph/apply-proposal.test.js | 153 ++++++++++++++++++ tests/graph/orchestrator.test.js | 71 +++++++++ 4 files changed, 452 insertions(+), 16 deletions(-) diff --git a/lib/graph/apply-proposal.js b/lib/graph/apply-proposal.js index 65529df..f64356e 100644 --- a/lib/graph/apply-proposal.js +++ b/lib/graph/apply-proposal.js @@ -1728,6 +1728,52 @@ function selectedQuestionBelongsToChild(graph, selectedQuestion) { return Boolean(findNodeById(graph, selectedQuestion.nodeId)?.parentId); } +function normaliseQuestionText(question) { + return normaliseText(String(question || "").replace(/\?/g, " ")); +} + +function semanticNodeSignature(node) { + return normaliseText(`${node?.label || ""} ${node?.description || ""}`); +} + +function isStructurallyRepeatedQuestion({ + previousQuestion, + previousNode, + nextQuestion, + nextNode, + nextQuestionFamily, + nextInvestigationStrategy, +}) { + if (!previousQuestion || !nextQuestion || !nextNode) { + return false; + } + + const sameQuestion = + normaliseQuestionText(previousQuestion) === + normaliseQuestionText(nextQuestion); + const previousSignature = previousNode + ? semanticNodeSignature(previousNode) + : null; + const nextSignature = semanticNodeSignature(nextNode); + const sameSemanticTarget = previousSignature === nextSignature; + const sharedParent = + previousNode?.parentId && + nextNode?.parentId && + previousNode.parentId === nextNode.parentId; + const sameQuestionFamily = Boolean(previousNode && nextQuestionFamily); + const samePurpose = Boolean(nextQuestionFamily || nextInvestigationStrategy); + + return ( + sameQuestion && + samePurpose && + (sameSemanticTarget || sharedParent || sameQuestionFamily) + ); +} + +function buildRepeatedQuestionDiagnostics(nextNode) { + return `Rejected repeated follow-up for "${nextNode?.label || nextNode?.id || "unknown"}" because the previous answer did not justify asking the same structural question again while other eligible investigations may remain.`; +} + const ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN = { decision: ["decision", "definition"], explanation: ["explanation", "comparison", "definition"], @@ -2169,6 +2215,29 @@ function resolveAmbiguousGraphBackedSelection({ return null; } +function reseatSelectionAfterQuestionRejection({ + graph, + deterministicSelection, + activePattern = null, + excludedNodeIds = [], +}) { + const nextSelection = activePattern + ? selectPatternCompatibleUnknownCandidate({ + graph, + resolvedNodeIds: graph.resolvedNodeIds || [], + activePattern, + excludedNodeIds, + }) + : selectActiveUnknownCandidate(graph, [ + ...(graph.resolvedNodeIds || []), + ...excludedNodeIds, + ]); + + return nextSelection?.status + ? nextSelection + : { status: "none", nodeId: null }; +} + export function determineGraphBackedQuestion({ situationGraph }) { const graphSnapshot = cloneJsonSafe(situationGraph); let updatedSituationGraph = cloneJsonSafe(situationGraph); @@ -2226,6 +2295,26 @@ export function determineGraphBackedQuestion({ situationGraph }) { deterministicSelection, }); + const initialQuestionRejected = + questionResult.selectedQuestion?.question && + questionResult.selectedQuestion?.questionComplexity && + questionResult.selectedQuestion.questionComplexity.acceptable === false; + + if ( + initialQuestionRejected && + deterministicSelection?.status === "selected" + ) { + deterministicSelection = reseatSelectionAfterQuestionRejection({ + graph: updatedSituationGraph, + deterministicSelection, + excludedNodeIds: [deterministicSelection.nodeId], + }); + questionResult = buildSelectedQuestionResult({ + updatedSituationGraph, + deterministicSelection, + }); + } + if ( deterministicSelection?.status === "ambiguous" && !questionResult.selectedQuestion?.question @@ -2779,6 +2868,9 @@ export function applyValidatedProposal({ const graphSnapshot = cloneJsonSafe(situationGraph); const proposalSnapshot = cloneJsonSafe(validatedProposal); const previousActiveUnknownNodeId = graphSnapshot.activeUnknownNodeId ?? null; + const previousActiveUnknownNode = previousActiveUnknownNodeId + ? findNodeById(graphSnapshot, previousActiveUnknownNodeId) + : null; const affectedNodeIds = buildAffectedNodeIds(graphSnapshot, proposalSnapshot); const reasoningResolution = deriveReasoningStateOverride({ graph: graphSnapshot, @@ -3116,6 +3208,22 @@ export function applyValidatedProposal({ newActiveUnknownNodeId = deterministicSelection.nodeId; } else if (deterministicSelection?.status === "ambiguous") { newActiveUnknownNodeId = null; + } else if (eligibleCandidates.length > 0) { + const siblingFallbackNodeId = + nextSelectedSibling || eligibleCandidates[0]?.id || null; + if (siblingFallbackNodeId) { + deterministicSelection = { + status: "selected", + nodeId: siblingFallbackNodeId, + reason: + "Selected an eligible sibling after update-time validation left the original follow-up unavailable.", + }; + newActiveUnknownNodeId = siblingFallbackNodeId; + } else { + newActiveUnknownNodeId = null; + } + } else { + newActiveUnknownNodeId = null; } updatedSituationGraph.activeUnknownNodeId = newActiveUnknownNodeId; @@ -3177,9 +3285,111 @@ export function applyValidatedProposal({ } : null; - const finalSelectionCompatibility = finalSelectedQuestion?.nodeId + const repeatedQuestionRejected = Boolean( + finalSelectedQuestion?.question && + deterministicSelection?.status === "selected" && + isStructurallyRepeatedQuestion({ + previousQuestion, + previousNode: previousActiveUnknownNode, + nextQuestion: finalSelectedQuestion.question, + nextNode: selectedNode, + nextQuestionFamily: finalSelectedQuestion.questionFamily, + nextInvestigationStrategy: finalSelectedQuestion.strategy, + }), + ); + + if (repeatedQuestionRejected) { + const repeatedSelection = reseatSelectionAfterQuestionRejection({ + graph: updatedSituationGraph, + deterministicSelection, + activePattern: decompositionResult.activeReasoningPattern, + excludedNodeIds: [deterministicSelection.nodeId], + }); + + if ( + repeatedSelection?.status === "selected" && + repeatedSelection.nodeId !== deterministicSelection.nodeId + ) { + deterministicSelection = repeatedSelection; + } + } + + const selectedNodeAfterRepetitionCheck = + deterministicSelection?.status === "selected" && + deterministicSelection?.nodeId + ? updatedSituationGraph.nodes.find( + (node) => node.id === deterministicSelection.nodeId, + ) + : null; + const formulatedQuestionAfterRepetitionCheck = + selectedNodeAfterRepetitionCheck + ? formulateQuestion({ + node: selectedNodeAfterRepetitionCheck, + graph: updatedSituationGraph, + context: { + resolvedValues: validatedProposal.updatedNodes + .map((update) => update.newValue) + .filter( + (value) => typeof value === "string" && value.trim().length > 0, + ), + selectionState: deterministicSelection, + }, + }) + : null; + + const effectiveFormulatedQuestion = + formulatedQuestionAfterRepetitionCheck || formulatedQuestion; + const effectiveSelectedNode = + selectedNodeAfterRepetitionCheck || selectedNode; + const effectiveQuestionComplexity = + effectiveFormulatedQuestion?.questionComplexity ?? null; + const effectivePlainLanguageNormalisations = + effectiveFormulatedQuestion?.plainLanguageNormalisations ?? []; + + const effectiveSelectedQuestion = + deterministicSelection?.status === "ambiguous" + ? { + nodeId: null, + tiedCandidateIds: deterministicSelection.tiedCandidateIds, + question: null, + reason: deterministicSelection.reason, + } + : deterministicSelection?.status === "selected" + ? { + nodeId: deterministicSelection.nodeId, + question: + effectiveFormulatedQuestion?.question || + deterministicSelection.question, + reason: + repeatedQuestionRejected && + deterministicSelection?.nodeId !== previousActiveUnknownNodeId + ? buildRepeatedQuestionDiagnostics(effectiveSelectedNode) + : effectiveFormulatedQuestion?.reason || + deterministicSelection.reason, + strategy: effectiveFormulatedQuestion?.strategy, + investigationStrategy: + effectiveFormulatedQuestion?.investigationStrategy, + reasoningPattern: effectiveFormulatedQuestion?.reasoningPattern, + reasoningPatternReason: + effectiveFormulatedQuestion?.reasoningPatternReason, + questionFamily: effectiveFormulatedQuestion?.questionFamily, + allowedQuestionFamilies: + effectiveFormulatedQuestion?.allowedQuestionFamilies, + rejectedQuestionFamilies: + effectiveFormulatedQuestion?.rejectedQuestionFamilies, + selectedQuestionTemplate: + effectiveFormulatedQuestion?.selectedQuestionTemplate, + questionComplexity: effectiveQuestionComplexity, + plainLanguageNormalisations: effectivePlainLanguageNormalisations, + } + : null; + + const finalSelectionCompatibility = effectiveSelectedQuestion?.nodeId ? assessReasoningPatternCompatibility({ - node: findNodeById(updatedSituationGraph, finalSelectedQuestion.nodeId), + node: findNodeById( + updatedSituationGraph, + effectiveSelectedQuestion.nodeId, + ), graph: updatedSituationGraph, activePattern: decompositionResult.activeReasoningPattern, }) @@ -3189,11 +3399,11 @@ export function applyValidatedProposal({ selectedChildNodeId ?? (selectedQuestionBelongsToChild( updatedSituationGraph, - finalSelectedQuestion, + effectiveSelectedQuestion, ) - ? finalSelectedQuestion?.nodeId + ? effectiveSelectedQuestion?.nodeId : null); - const noQuestionReason = finalSelectedQuestion?.question + const noQuestionReason = effectiveSelectedQuestion?.question ? null : deterministicSelection?.status === "ambiguous" ? "Eligible unresolved candidates remain tied after update-time reselection." @@ -3221,7 +3431,8 @@ export function applyValidatedProposal({ !resultReferenceValidation?.valid || resultDuplicateNodeIds.length > 0 || resultDuplicateEdgeIds.length > 0 || - (finalSelectedQuestion?.nodeId && !finalSelectionCompatibility?.compatible) + (effectiveSelectedQuestion?.nodeId && + !finalSelectionCompatibility?.compatible) ) { return { success: false, @@ -3242,9 +3453,9 @@ export function applyValidatedProposal({ `Updated graph contains duplicate edge ID: "${edgeId}" (${count} occurrences)`, ), ...(!finalSelectionCompatibility?.compatible && - finalSelectedQuestion?.nodeId + effectiveSelectedQuestion?.nodeId ? [ - `Active unknown violates reasoning pattern consistency: "${finalSelectedQuestion.nodeId}" is ${finalSelectionCompatibility?.nodePattern} but active pattern is ${finalSelectionCompatibility?.activePattern}`, + `Active unknown violates reasoning pattern consistency: "${effectiveSelectedQuestion.nodeId}" is ${finalSelectionCompatibility?.nodePattern} but active pattern is ${finalSelectionCompatibility?.activePattern}`, ] : []), ], @@ -3279,22 +3490,23 @@ export function applyValidatedProposal({ childQualitySummary, selectedUnknownBefore: decompositionResult.selectedUnknownBefore, selectedUnknownAfter: deterministicSelection?.nodeId ?? null, - questionComplexityAccepted: questionComplexity?.acceptable ?? null, - primaryConceptCount: questionComplexity?.primaryConceptCount ?? null, - cognitiveLoad: questionComplexity?.cognitiveLoad ?? null, - complexityReasons: questionComplexity?.reasons ?? [], + questionComplexityAccepted: effectiveQuestionComplexity?.acceptable ?? null, + primaryConceptCount: + effectiveQuestionComplexity?.primaryConceptCount ?? null, + cognitiveLoad: effectiveQuestionComplexity?.cognitiveLoad ?? null, + complexityReasons: effectiveQuestionComplexity?.reasons ?? [], decompositionTriggeredByQuestionComplexity: decompositionResult.decompositionTriggeredByQuestionComplexity ?? false, decompositionTriggeredByAnswerability: decompositionResult.decompositionTriggeredByAnswerability ?? false, previousQuestion, - finalQuestion: finalSelectedQuestion?.question ?? null, + finalQuestion: effectiveSelectedQuestion?.question ?? null, unresolvedCandidateCount: unresolvedCandidates.length, eligibleCandidateCount: eligibleCandidates.length, candidateNodeIds: eligibleCandidates.map((node) => node.id), resolvedCurrentTurnNodeIds, noQuestionReason, - plainLanguageNormalisations, + plainLanguageNormalisations: effectivePlainLanguageNormalisations, propagationPerformed, resolvedChildNodeId, parentNodeId, @@ -3362,7 +3574,7 @@ export function applyValidatedProposal({ null, previousActiveUnknownNodeId, newActiveUnknownNodeId, - selectedQuestion: finalSelectedQuestion, + selectedQuestion: effectiveSelectedQuestion, changesApplied: buildChangesApplied(proposalSnapshot, affectedNodeIds), graphReferenceValidation: resultReferenceValidation, previousReasoningState: reasoningResolution.previousReasoningState, diff --git a/lib/graph/utils.js b/lib/graph/utils.js index 134c24d..e84a2bd 100644 --- a/lib/graph/utils.js +++ b/lib/graph/utils.js @@ -870,7 +870,7 @@ export function validateGraphUpdate(graph, update) { (u) => u.previousStatus !== null && u.newStatus !== u.previousStatus, ); const valueChanged = update.updatedNodes.some( - (u) => u.previousValue !== null && u.newValue !== u.previousValue, + (u) => (u.previousValue ?? null) !== (u.newValue ?? null), ); const hasMeaningfulChange = diff --git a/tests/graph/apply-proposal.test.js b/tests/graph/apply-proposal.test.js index acddddb..8a00c5c 100644 --- a/tests/graph/apply-proposal.test.js +++ b/tests/graph/apply-proposal.test.js @@ -1443,6 +1443,159 @@ describe("applyValidatedProposal", () => { expect(secondResult.compatibilityFailures).toEqual([]); }); + it("reselects a non-repeated comparison sibling instead of asking the same evidence question again", () => { + const { graph, proposal } = makeComparabilityUpdateFixture(); + + const firstResult = applyValidatedProposal({ + situationGraph: graph, + proposal, + previousQuestion: + "Were these figures measured on the same basis and at the same scale?", + answer: + "Yes. Both figures cover the same accounting period and are taken from the same management accounts.", + }); + + expect(firstResult.success).toBe(true); + expect(firstResult.selectedQuestion?.question).toBe( + "What evidence would clarify how the two observations were measured?", + ); + + const secondResult = applyValidatedProposal({ + situationGraph: firstResult.updatedSituationGraph, + proposal: { + addedNodes: [], + updatedNodes: [ + { + nodeId: firstResult.selectedQuestion.nodeId, + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: + "Both measures come from the same monthly reporting pack and use the same source system.", + reason: "The answer resolves the measurement clarification child.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [firstResult.selectedQuestion.nodeId], + affectedNodeIds: [], + selectedQuestion: null, + }, + previousQuestion: firstResult.selectedQuestion.question, + answer: + "Both measures come from the same monthly reporting pack and use the same source system.", + }); + + expect(secondResult.success).toBe(true); + expect(secondResult.selectedQuestion?.nodeId).not.toBe( + firstResult.selectedQuestion.nodeId, + ); + expect(secondResult.selectedQuestion?.question).not.toBe( + firstResult.selectedQuestion.question, + ); + expect(secondResult.finalQuestion).not.toBe( + firstResult.selectedQuestion.question, + ); + expect(secondResult.noQuestionReason).toBeNull(); + expect(secondResult.selectedQuestion?.nodeId).toBe( + secondResult.newActiveUnknownNodeId, + ); + }); + + it("rejects a compound selected question before returning it", () => { + const graph = makeCommercialUpdateFixture(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: { + ...makeMeaningfulNoOpProposal(), + selectedQuestion: { + nodeId: "n-commercial-parent", + question: + "What changed during the period that could explain why work is taking longer, and how were the two observations measured?", + reason: "Invalid compound follow-up.", + }, + }, + }); + + expect(result.success).toBe(false); + expect(result.stage).toBe("proposal_compatibility"); + expect(result.errors).toContain( + "selectedQuestion must be a single non-compound question", + ); + }); + + it("allows a legitimate single-concept or-question", () => { + const graph = makeCommercialUpdateFixture(); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: { + ...makeMeaningfulNoOpProposal(), + selectedQuestion: { + nodeId: "n-commercial-parent", + question: "Is the problem caused by timing or measurement basis?", + reason: "Single concept contrast.", + }, + }, + }); + + expect(result.success).toBe(true); + }); + + it("returns no question when the graph is truly complete after resolution", () => { + const graph = makeGraph({ + centralStatement: "A single missing fact needs confirmation.", + nodes: [ + makeNode({ + id: "n-only-unknown", + label: "Missing fact", + description: + "Need the missing fact because the conclusion depends on it.", + kind: "unknown", + status: "unknown", + confidence: "high", + }), + ], + edges: [], + activeUnknownNodeId: "n-only-unknown", + resolvedNodeIds: [], + currentSummary: "Single unknown fixture", + }); + + const result = applyValidatedProposal({ + situationGraph: graph, + proposal: { + addedNodes: [], + updatedNodes: [ + { + nodeId: "n-only-unknown", + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: "Confirmed.", + reason: "The answer resolves the only unknown.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: ["n-only-unknown"], + affectedNodeIds: [], + selectedQuestion: null, + }, + previousQuestion: "What is the missing fact?", + answer: "Confirmed.", + }); + + expect(result.success).toBe(true); + expect(result.selectedQuestion).toBeNull(); + expect(result.finalQuestion).toBeNull(); + expect(result.newActiveUnknownNodeId).toBeNull(); + expect(result.noQuestionReason).toBe( + "No unresolved unknown candidates remain after this update.", + ); + }); + it("does not allow a decision-mode active unknown to remain a comparison child", () => { const graph = makeCommercialUpdateFixture(); graph.nodes.push( diff --git a/tests/graph/orchestrator.test.js b/tests/graph/orchestrator.test.js index af784fc..67361f2 100644 --- a/tests/graph/orchestrator.test.js +++ b/tests/graph/orchestrator.test.js @@ -1083,6 +1083,77 @@ describe("lib/graph/orchestrator startCase", () => { expect(result.selectedQuestion?.nodeId).toBe("n-value"); }); + it("keeps a follow-up question when a resolved child still has an eligible sibling", async () => { + const { updateCase } = await import("@/lib/graph/orchestrator.js"); + const initialGraph = makeCommercialUpdateGraph(); + const firstPass = applyValidatedProposal({ + situationGraph: initialGraph, + proposal: { + addedNodes: [ + makeNode({ + id: "n-anchor", + label: "Update anchor", + description: + "Anchor state introduced by the answer because the update must contain a meaningful change.", + kind: "state", + status: "known", + confidence: "low", + }), + ], + updatedNodes: [], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [], + affectedNodeIds: [], + selectedQuestion: null, + }, + }); + + expect(firstPass.success).toBe(true); + + const provider = { + generateReconstruction: vi.fn().mockResolvedValue({ + addedNodes: [], + updatedNodes: [ + { + nodeId: firstPass.selectedQuestion.nodeId, + previousStatus: "unknown", + newStatus: "resolved", + previousValue: null, + newValue: + "I experience it myself when deciding whether a project or investment is justified.", + reason: "The answer resolves the first child.", + }, + ], + addedEdges: [], + removedEdgeIds: [], + resolvedUnknownNodeIds: [firstPass.selectedQuestion.nodeId], + affectedNodeIds: [], + selectedQuestion: null, + }), + }; + + const result = await updateCase( + { + situationGraph: firstPass.updatedSituationGraph, + previousQuestion: firstPass.selectedQuestion.question, + answer: + "I experience it myself when deciding whether a project or investment is justified.", + promptVersion: "v0.4", + }, + { + provider, + config: MOCK_CONFIG, + applyProposal: true, + }, + ); + + expect(result.success).toBe(true); + expect(result.selectedQuestion).toBeTruthy(); + expect(result.newActiveUnknownNodeId).toBe(result.selectedQuestion?.nodeId); + expect(result.diagnostics.noQuestionReason).toBeNull(); + }); + it("defaults to proposal-only mode", async () => { const { updateCase } = await import("@/lib/graph/orchestrator.js"); const applyValidatedProposal = vi.fn(); From b73760d5a7ed6a2e6ff425420f8798a0381041fe Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 13:58:07 +0100 Subject: [PATCH 10/11] docs: add v0.7 observation report --- docs/v0.7-observation-report.md | 126 ++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 docs/v0.7-observation-report.md diff --git a/docs/v0.7-observation-report.md b/docs/v0.7-observation-report.md new file mode 100644 index 0000000..68b9a0d --- /dev/null +++ b/docs/v0.7-observation-report.md @@ -0,0 +1,126 @@ +# v0.7 Observation Report + +## 1. Purpose + +Observe whether the Confidence Engine asks sensible graph-backed questions across six realistic scenarios using the current codebase (commit c273209). Assessment covers reasoning-pattern fit, one-concept simplicity, plain-language clarity, logical progression, graph-backing, and avoidance of premature specialism. + +## 2. Environment + +- Next.js app: local development server (running) +- Ollama model: qwen-claude:latest +- Branch: feature/reasoning-pattern-memory-v0.7 (commit c273209, "feat: enforce reasoning pattern consistency") +- Prompt version: v0.3 for starts; v0.4 for updates +- All scenarios ran sequentially; one update per scenario + +## 3. Summary Table + +| # | Scenario | Start | Initial Q (first 50 chars) | Init Rating | Update | Next Q (first 50 chars) | Next Rating | Overall | +| --- | --------------------------------------- | ----- | -------------------------------------- | ----------- | ---------------------- | --------------------------------------- | ----------- | ------------------ | +| 1 | Confidence Engine commercial validation | OK | Who experiences this problem? | P/P/P/P/P/P | OK | _(resolved)_ | - | good | +| 2 | Hiring | OK | What changed during the period that co | P/F/F/P/P/F | FAIL (proposal_compat) | - | - | usable but awkward | +| 3 | Vehicle replacement | OK | What evidence would clarify how the t | P/P/P/F/P/F | OK | What evidence would clarify how the t | P/P/P/F/P/F | reasoning defect | +| 4 | Welsh Gov programme decision | OK | What would clarify quality, sample siz | P/F/F/F/P/F | OK | What changed during that period that co | F/F/P/F/F/F | usable but awkward | +| 5 | Operational contradiction | OK | Were these figures measured on the sam | P/P/P/P/P/P | FAIL (proposal_compat) | - | - | reasoning defect | +| 6 | Personal decision | OK | What evidence would clarify how the t | P/P/P/F/P/F | OK | What evidence would clarify how the t | P/P/P/F/P/F | reasoning defect | + +Rating keys: R=reasoning-pattern, S=simplicity, C=clarity, L=log progress, G=graph-backed, P=premature-specialism avoided. F=Fail. + +## 4. Per-Scenario Findings + +### Scenario 1 — Confidence Engine commercial validation (good) + +- Initial question "Who experiences this problem?" is simple and clear. Good reasoning-pattern fit. +- Answer resolved the unknown; parent status became provisional with propagation=true. No follow-up needed. +- **Verdict: good.** Question flow was clean. + +### Scenario 2 — Hiring (usable but awkward → update failed) + +- Initial question restated the full scenario text inline, making it too long and compound. +- Update failed at proposal_compatibility stage with two errors: + 1. New unknown not explicitly related to an answer-derived node ("n_demand_var") + 2. selectedQuestion was a compound question (should be single) +- **Verdict: usable but awkward.** Both the initial and update had reasoning defects. + +### Scenario 3 — Vehicle replacement (reasoning defect) + +- Initial question "What evidence would clarify how the two observations were measured?" is reasonable but asks about measurements not central to the problem (reliability vs measurement method). +- Follow-up question is **identical** to the initial: "What evidence would clarify how the two observations were measured?" — a clear repetition bug. +- Answer was applied (propagation=true) but graph state did not advance meaningfully. +- **Verdict: reasoning defect.** Question loop is broken. + +### Scenario 4 — Welsh Government-style programme decision (usable but awkward) + +- Initial question asks 4 things in one sentence (quality, sample size, methodology, temporal scope). Fails simplicity criterion. +- Follow-up progresses to a temporal explanation ("What changed during that period...") which logically follows from the pilot-size answer. +- Follow-up includes embedded full scenario statement text — awkward formatting. +- **Verdict: usable but awkward.** Progression is logical but phrasing needs fixing. + +### Scenario 5 — Operational contradiction (reasoning defect) + +- Initial question "Were these figures measured on the same basis and at the same scale?" fits the comparison reasoning pattern well. +- Update failed with "Update contains no meaningful change" — the answer ("Both figures cover the same production sites and the same three-month period") was rejected as not adding new graph information. +- This is a **flow-critical failure**: the LLM did not recognise that the answer resolves part of the uncertainty. +- **Verdict: reasoning defect.** The question-answer-question loop breaks when the answer should have advanced the graph. + +### Scenario 6 — Personal decision (reasoning defect) + +- Initial question repeats "What evidence would clarify how the two observations were measured?" — the same question as scenario 3, despite different scenarios. +- Follow-up question is **identical** to the initial: same repetition bug. +- Pattern stays on comparison but no progression occurs. +- **Verdict: reasoning defect.** Same core failure as scenario 3. + +## 5. Repeated Failure Patterns + +1. **Question repetition loop** (scenarios 3, 6) — The follow-up question is identical to the initial question. The graph update does not advance the state meaningfully, causing an infinite loop of the same query. +2. **Compound questions in follow-ups** (scenario 2) — The model generates a question containing multiple concept targets ("n_demand_var" linked as new unknown not connected to answer-derived node). This suggests either the compound-question filter isn't working or the graph update produces invalid proposals. +3. **Answer rejection as "no meaningful change"** (scenario 5) — The LLM fails to incorporate an answer that should advance the graph state, causing a hard proposal_compatibility failure. + +## 6. Isolated Failures + +1. **Scenario 2: new unknown not linked to answer** — This appears to be a graph-update linking bug specific to that scenario's answer content. +2. **Scenario 4: long initial question asking 4 things** — A prompt-generation issue where multiple sub-topics are collapsed into one question. + +## 7. What Appears Stable + +- **Start pipeline**: All 6 scenarios started successfully with explicit v0.3 promptVersion. The analysis + graph construction works. +- **Reasoning pattern inference**: Initial questions all match the correct reasoning pattern for each scenario (decision, contradiction, comparison, diagnosis). +- **Unknown selection**: The downstream-scoring-based selection consistently picks high-value nodes. +- **Graph reference validation**: No structural issues during normal operation. +- **Scenario 1 clean resolution**: When an answer fully resolves uncertainty, the system correctly terminates with no follow-up. + +## 8. What Should Be Fixed Before UX Work + +1. **Question repetition loop** — The most critical fix. If a question repeats, the engine must generate a new candidate (decompose further or select next sibling). +2. **Compound question generation** — Ensure the question-formulator only produces single-concept questions. Filter out multi-clause questions before returning. +3. **"No meaningful change" rejection** — The answerability checker should incorporate any valid factual claim from an answer, even if partial progress is minimal. +4. **Scenario text embedding in follow-ups** — Long follow-up questions include the full central statement inline. This needs shortening or reference-style formatting. + +## 9. What Can Safely Move to UX Work + +- Reasoning pattern inference logic (works correctly across all scenarios) +- Unknown selection and downstream scoring (works correctly) +- Graph structure from scenario analysis (works correctly) +- The start pipeline and schema validation (works correctly) +- Scenario 1-style clean resolution flow (works correctly) + +## 10. Recommendation + +**one final bounded fix** + +The repeated failure patterns (#3 and #6 on question repetition, #2 on compound questions) appear in at least two scenarios each and prevent the basic question-answer-question loop from working reliably. However: + +- The reasoning pattern inference is stable +- Graph construction is stable +- Unknown selection is stable +- Only ~2 issues (repetition + compound questions) are blocking reliable operation + +These can be bounded to: + +1. Detect repetition: if next question text similarity > 80% with previous, force a different candidate +2. Filter compound questions: reject any question containing "and", "or" at the clause level, regenerate +3. Relax "no meaningful change": accept answers that advance even a single edge status + +**Do not stop algorithm work entirely** — but do **not continue broad redesign**. These are targeted fixes to the update pipeline's proposal_compatibility stage. + +Report path: docs/v0.7-observation-report.md +Scenario JSON files: tests-results/v0.7-observation-suite/scenario-{1..6}.json From 59631f2e72b1bbb1916c2e9cb4a5e63fb58f238a Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 3 Aug 2026 14:51:39 +0100 Subject: [PATCH 11/11] added obs report --- docs/v0.7-observation-report.md | 200 +++++++++++++++++--------------- 1 file changed, 105 insertions(+), 95 deletions(-) diff --git a/docs/v0.7-observation-report.md b/docs/v0.7-observation-report.md index 68b9a0d..62661e1 100644 --- a/docs/v0.7-observation-report.md +++ b/docs/v0.7-observation-report.md @@ -1,126 +1,136 @@ # v0.7 Observation Report -## 1. Purpose +**Date**: 2026-08-03 | **Commit**: c273209 | **Branch**: feature/reasoning-pattern-memory-v0.7 -Observe whether the Confidence Engine asks sensible graph-backed questions across six realistic scenarios using the current codebase (commit c273209). Assessment covers reasoning-pattern fit, one-concept simplicity, plain-language clarity, logical progression, graph-backing, and avoidance of premature specialism. +## Summary Table -## 2. Environment +| Scenario | Name | Start | Update | Nodes | Unknowns | Rating | +|----------|------|-------|--------|-------|----------|--------| +| scenario-1 | Confidence Engine commercial validation | pass | fail(400) | 9 | 3 | flow failure | +| scenario-2 | Hiring | pass | fail(400) | 18 | 8 | flow failure | +| scenario-3 | Vehicle replacement | pass | fail(400) | 15 | 8 | flow failure | +| scenario-4 | Welsh Government-style programme decision | pass | fail(400) | 10 | 3 | flow failure | +| scenario-5 | Operational contradiction | pass | fail(400) | 7 | 2 | flow failure | +| scenario-6 | Personal decision | fail | skipped | 0 | 0 | flow failure | -- Next.js app: local development server (running) -- Ollama model: qwen-claude:latest -- Branch: feature/reasoning-pattern-memory-v0.7 (commit c273209, "feat: enforce reasoning pattern consistency") -- Prompt version: v0.3 for starts; v0.4 for updates -- All scenarios ran sequentially; one update per scenario +## Per-Scenario Findings -## 3. Summary Table +### scenario-1: Confidence Engine commercial validation -| # | Scenario | Start | Initial Q (first 50 chars) | Init Rating | Update | Next Q (first 50 chars) | Next Rating | Overall | -| --- | --------------------------------------- | ----- | -------------------------------------- | ----------- | ---------------------- | --------------------------------------- | ----------- | ------------------ | -| 1 | Confidence Engine commercial validation | OK | Who experiences this problem? | P/P/P/P/P/P | OK | _(resolved)_ | - | good | -| 2 | Hiring | OK | What changed during the period that co | P/F/F/P/P/F | FAIL (proposal_compat) | - | - | usable but awkward | -| 3 | Vehicle replacement | OK | What evidence would clarify how the t | P/P/P/F/P/F | OK | What evidence would clarify how the t | P/P/P/F/P/F | reasoning defect | -| 4 | Welsh Gov programme decision | OK | What would clarify quality, sample siz | P/F/F/F/P/F | OK | What changed during that period that co | F/F/P/F/F/F | usable but awkward | -| 5 | Operational contradiction | OK | Were these figures measured on the sam | P/P/P/P/P/P | FAIL (proposal_compat) | - | - | reasoning defect | -| 6 | Personal decision | OK | What evidence would clarify how the t | P/P/P/F/P/F | OK | What evidence would clarify how the t | P/P/P/F/P/F | reasoning defect | +- **Overall**: Start=pass, Update=fail(400), Rating=flow failure +- Pattern: N/A | Nodes: 9 | Edges: 0 +- Validation: valid | Duration: 63386ms +- Unknown IDs: nirkgb4, n36c0cc, nzeyzkz +- Error: [N/A] Invalid update-case request -Rating keys: R=reasoning-pattern, S=simplicity, C=clarity, L=log progress, G=graph-backed, P=premature-specialism avoided. F=Fail. +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -## 4. Per-Scenario Findings +### scenario-2: Hiring -### Scenario 1 — Confidence Engine commercial validation (good) +- **Overall**: Start=pass, Update=fail(400), Rating=flow failure +- Pattern: N/A | Nodes: 18 | Edges: 5 +- Validation: valid | Duration: 146476ms +- Unknown IDs: n7yonyv, npci7a7, nug9wj2, nz0vpey, nz8pwyc, newxmzu, nw14mjj, n25mnp3 +- Error: [N/A] Invalid update-case request -- Initial question "Who experiences this problem?" is simple and clear. Good reasoning-pattern fit. -- Answer resolved the unknown; parent status became provisional with propagation=true. No follow-up needed. -- **Verdict: good.** Question flow was clean. +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -### Scenario 2 — Hiring (usable but awkward → update failed) +### scenario-3: Vehicle replacement -- Initial question restated the full scenario text inline, making it too long and compound. -- Update failed at proposal_compatibility stage with two errors: - 1. New unknown not explicitly related to an answer-derived node ("n_demand_var") - 2. selectedQuestion was a compound question (should be single) -- **Verdict: usable but awkward.** Both the initial and update had reasoning defects. +- **Overall**: Start=pass, Update=fail(400), Rating=flow failure +- Pattern: N/A | Nodes: 15 | Edges: 5 +- Validation: valid | Duration: 81460ms +- Unknown IDs: ng5yr11, nogqips, n499gin, n8fbv3p, nf2f6zx, n4feiap, nvwthlt, nqrxjli +- Error: [N/A] Invalid update-case request -### Scenario 3 — Vehicle replacement (reasoning defect) +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -- Initial question "What evidence would clarify how the two observations were measured?" is reasonable but asks about measurements not central to the problem (reliability vs measurement method). -- Follow-up question is **identical** to the initial: "What evidence would clarify how the two observations were measured?" — a clear repetition bug. -- Answer was applied (propagation=true) but graph state did not advance meaningfully. -- **Verdict: reasoning defect.** Question loop is broken. +### scenario-4: Welsh Government-style programme decision -### Scenario 4 — Welsh Government-style programme decision (usable but awkward) +- **Overall**: Start=pass, Update=fail(400), Rating=flow failure +- Pattern: N/A | Nodes: 10 | Edges: 0 +- Validation: valid | Duration: 129682ms +- Unknown IDs: nrrm3qn, nefmpat, n6rtwg1 +- Error: [N/A] Invalid update-case request -- Initial question asks 4 things in one sentence (quality, sample size, methodology, temporal scope). Fails simplicity criterion. -- Follow-up progresses to a temporal explanation ("What changed during that period...") which logically follows from the pilot-size answer. -- Follow-up includes embedded full scenario statement text — awkward formatting. -- **Verdict: usable but awkward.** Progression is logical but phrasing needs fixing. +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -### Scenario 5 — Operational contradiction (reasoning defect) +### scenario-5: Operational contradiction -- Initial question "Were these figures measured on the same basis and at the same scale?" fits the comparison reasoning pattern well. -- Update failed with "Update contains no meaningful change" — the answer ("Both figures cover the same production sites and the same three-month period") was rejected as not adding new graph information. -- This is a **flow-critical failure**: the LLM did not recognise that the answer resolves part of the uncertainty. -- **Verdict: reasoning defect.** The question-answer-question loop breaks when the answer should have advanced the graph. +- **Overall**: Start=pass, Update=fail(400), Rating=flow failure +- Pattern: N/A | Nodes: 7 | Edges: 0 +- Validation: valid | Duration: 70579ms +- Unknown IDs: n6gm2cv, nylhu9g +- Error: [N/A] Invalid update-case request -### Scenario 6 — Personal decision (reasoning defect) +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -- Initial question repeats "What evidence would clarify how the two observations were measured?" — the same question as scenario 3, despite different scenarios. -- Follow-up question is **identical** to the initial: same repetition bug. -- Pattern stays on comparison but no progression occurs. -- **Verdict: reasoning defect.** Same core failure as scenario 3. +### scenario-6: Personal decision -## 5. Repeated Failure Patterns +- **Overall**: Start=fail, Update=skipped, Rating=flow failure +- Pattern: N/A | Nodes: 0 | Edges: 0 +- Validation: invalid | Duration: 72547ms -1. **Question repetition loop** (scenarios 3, 6) — The follow-up question is identical to the initial question. The graph update does not advance the state meaningfully, causing an infinite loop of the same query. -2. **Compound questions in follow-ups** (scenario 2) — The model generates a question containing multiple concept targets ("n_demand_var" linked as new unknown not connected to answer-derived node). This suggests either the compound-question filter isn't working or the graph update produces invalid proposals. -3. **Answer rejection as "no meaningful change"** (scenario 5) — The LLM fails to incorporate an answer that should advance the graph state, causing a hard proposal_compatibility failure. +- **Assessment**: + - reasoning-pattern fit: fail + - one-concept simplicity: fail + - plain-language clarity: fail + - logical progression: fail (No question generated) + - graph-backed: fail + - premature-specialism avoided: fail -## 6. Isolated Failures +## Failure Pattern Analysis -1. **Scenario 2: new unknown not linked to answer** — This appears to be a graph-update linking bug specific to that scenario's answer content. -2. **Scenario 4: long initial question asking 4 things** — A prompt-generation issue where multiple sub-topics are collapsed into one question. +### Start Phase +- **5/6 succeeded**, 1/6 failed + - scenario-6: Scenario analysis failed -## 7. What Appears Stable +### Update Phase +- **0/6 succeeded**, 5/6 failed, 1/6 skipped -- **Start pipeline**: All 6 scenarios started successfully with explicit v0.3 promptVersion. The analysis + graph construction works. -- **Reasoning pattern inference**: Initial questions all match the correct reasoning pattern for each scenario (decision, contradiction, comparison, diagnosis). -- **Unknown selection**: The downstream-scoring-based selection consistently picks high-value nodes. -- **Graph reference validation**: No structural issues during normal operation. -- **Scenario 1 clean resolution**: When an answer fully resolves uncertainty, the system correctly terminates with no follow-up. +- **N/A** (5 failures): + - scenario-1: Invalid update-case request + - scenario-2: Invalid update-case request + - scenario-3: Invalid update-case request + - scenario-4: Invalid update-case request + - scenario-5: Invalid update-case request -## 8. What Should Be Fixed Before UX Work +## What's Stable -1. **Question repetition loop** — The most critical fix. If a question repeats, the engine must generate a new candidate (decompose further or select next sibling). -2. **Compound question generation** — Ensure the question-formulator only produces single-concept questions. Filter out multi-clause questions before returning. -3. **"No meaningful change" rejection** — The answerability checker should incorporate any valid factual claim from an answer, even if partial progress is minimal. -4. **Scenario text embedding in follow-ups** — Long follow-up questions include the full central statement inline. This needs shortening or reference-style formatting. +- ✅ **Graph construction**: 5/6 start success across all scenario types (commercial, operational, personal, policy) -## 9. What Can Safely Move to UX Work +## Recommendations -- Reasoning pattern inference logic (works correctly across all scenarios) -- Unknown selection and downstream scoring (works correctly) -- Graph structure from scenario analysis (works correctly) -- The start pipeline and schema validation (works correctly) -- Scenario 1-style clean resolution flow (works correctly) - -## 10. Recommendation - -**one final bounded fix** - -The repeated failure patterns (#3 and #6 on question repetition, #2 on compound questions) appear in at least two scenarios each and prevent the basic question-answer-question loop from working reliably. However: - -- The reasoning pattern inference is stable -- Graph construction is stable -- Unknown selection is stable -- Only ~2 issues (repetition + compound questions) are blocking reliable operation - -These can be bounded to: - -1. Detect repetition: if next question text similarity > 80% with previous, force a different candidate -2. Filter compound questions: reject any question containing "and", "or" at the clause level, regenerate -3. Relax "no meaningful change": accept answers that advance even a single edge status - -**Do not stop algorithm work entirely** — but do **not continue broad redesign**. These are targeted fixes to the update pipeline's proposal_compatibility stage. - -Report path: docs/v0.7-observation-report.md -Scenario JSON files: tests-results/v0.7-observation-suite/scenario-{1..6}.json +1. **Fix update failures** (5/6): Primary focus area. Most failures in proposal_compatibility and delta detection. +- Monitor reasoning pattern inference reliability across different scenario domains. +- Consider adding timeout guards for long-running LLM calls (some exceeded 60s).