confidence-engine/docs/v0.7-observation-report.md

v0.7 Observation Report

1. Purpose

Observe whether the Confidence Engine asks sensible graph-backed questions across six realistic scenarios using the current codebase (commit c273209). Assessment covers reasoning-pattern fit, one-concept simplicity, plain-language clarity, logical progression, graph-backing, and avoidance of premature specialism.

2. Environment

• Next.js app: local development server (running)
• Ollama model: qwen-claude:latest
• Branch: feature/reasoning-pattern-memory-v0.7 (commit c273209, "feat: enforce reasoning pattern consistency")
• Prompt version: v0.3 for starts; v0.4 for updates
• All scenarios ran sequentially; one update per scenario

3. Summary Table

#	Scenario	Start	Initial Q (first 50 chars)	Init Rating	Update	Next Q (first 50 chars)	Next Rating	Overall
1	Confidence Engine commercial validation	OK	Who experiences this problem?	P/P/P/P/P/P	OK	(resolved)	-	good
2	Hiring	OK	What changed during the period that co	P/F/F/P/P/F	FAIL (proposal_compat)	-	-	usable but awkward
3	Vehicle replacement	OK	What evidence would clarify how the t	P/P/P/F/P/F	OK	What evidence would clarify how the t	P/P/P/F/P/F	reasoning defect
4	Welsh Gov programme decision	OK	What would clarify quality, sample siz	P/F/F/F/P/F	OK	What changed during that period that co	F/F/P/F/F/F	usable but awkward
5	Operational contradiction	OK	Were these figures measured on the sam	P/P/P/P/P/P	FAIL (proposal_compat)	-	-	reasoning defect
6	Personal decision	OK	What evidence would clarify how the t	P/P/P/F/P/F	OK	What evidence would clarify how the t	P/P/P/F/P/F	reasoning defect

Rating keys: R=reasoning-pattern, S=simplicity, C=clarity, L=log progress, G=graph-backed, P=premature-specialism avoided. F=Fail.

4. Per-Scenario Findings

Scenario 1 — Confidence Engine commercial validation (good)

• Initial question "Who experiences this problem?" is simple and clear. Good reasoning-pattern fit.
• Answer resolved the unknown; parent status became provisional with propagation=true. No follow-up needed.
• Verdict: good. Question flow was clean.

Scenario 2 — Hiring (usable but awkward → update failed)

• Initial question restated the full scenario text inline, making it too long and compound.
• Update failed at proposal_compatibility stage with two errors:
  1. New unknown not explicitly related to an answer-derived node ("n_demand_var")
  2. selectedQuestion was a compound question (should be single)
• Verdict: usable but awkward. Both the initial and update had reasoning defects.

Scenario 3 — Vehicle replacement (reasoning defect)

• Initial question "What evidence would clarify how the two observations were measured?" is reasonable but asks about measurements not central to the problem (reliability vs measurement method).
• Follow-up question is identical to the initial: "What evidence would clarify how the two observations were measured?" — a clear repetition bug.
• Answer was applied (propagation=true) but graph state did not advance meaningfully.
• Verdict: reasoning defect. Question loop is broken.

Scenario 4 — Welsh Government-style programme decision (usable but awkward)

• Initial question asks 4 things in one sentence (quality, sample size, methodology, temporal scope). Fails simplicity criterion.
• Follow-up progresses to a temporal explanation ("What changed during that period...") which logically follows from the pilot-size answer.
• Follow-up includes embedded full scenario statement text — awkward formatting.
• Verdict: usable but awkward. Progression is logical but phrasing needs fixing.

Scenario 5 — Operational contradiction (reasoning defect)

• Initial question "Were these figures measured on the same basis and at the same scale?" fits the comparison reasoning pattern well.
• Update failed with "Update contains no meaningful change" — the answer ("Both figures cover the same production sites and the same three-month period") was rejected as not adding new graph information.
• This is a flow-critical failure: the LLM did not recognise that the answer resolves part of the uncertainty.
• Verdict: reasoning defect. The question-answer-question loop breaks when the answer should have advanced the graph.

Scenario 6 — Personal decision (reasoning defect)

• Initial question repeats "What evidence would clarify how the two observations were measured?" — the same question as scenario 3, despite different scenarios.
• Follow-up question is identical to the initial: same repetition bug.
• Pattern stays on comparison but no progression occurs.
• Verdict: reasoning defect. Same core failure as scenario 3.

5. Repeated Failure Patterns

1. Question repetition loop (scenarios 3, 6) — The follow-up question is identical to the initial question. The graph update does not advance the state meaningfully, causing an infinite loop of the same query.
2. Compound questions in follow-ups (scenario 2) — The model generates a question containing multiple concept targets ("n_demand_var" linked as new unknown not connected to answer-derived node). This suggests either the compound-question filter isn't working or the graph update produces invalid proposals.
3. Answer rejection as "no meaningful change" (scenario 5) — The LLM fails to incorporate an answer that should advance the graph state, causing a hard proposal_compatibility failure.

6. Isolated Failures

1. Scenario 2: new unknown not linked to answer — This appears to be a graph-update linking bug specific to that scenario's answer content.
2. Scenario 4: long initial question asking 4 things — A prompt-generation issue where multiple sub-topics are collapsed into one question.

7. What Appears Stable

• Start pipeline: All 6 scenarios started successfully with explicit v0.3 promptVersion. The analysis + graph construction works.
• Reasoning pattern inference: Initial questions all match the correct reasoning pattern for each scenario (decision, contradiction, comparison, diagnosis).
• Unknown selection: The downstream-scoring-based selection consistently picks high-value nodes.
• Graph reference validation: No structural issues during normal operation.
• Scenario 1 clean resolution: When an answer fully resolves uncertainty, the system correctly terminates with no follow-up.

8. What Should Be Fixed Before UX Work

1. Question repetition loop — The most critical fix. If a question repeats, the engine must generate a new candidate (decompose further or select next sibling).
2. Compound question generation — Ensure the question-formulator only produces single-concept questions. Filter out multi-clause questions before returning.
3. "No meaningful change" rejection — The answerability checker should incorporate any valid factual claim from an answer, even if partial progress is minimal.
4. Scenario text embedding in follow-ups — Long follow-up questions include the full central statement inline. This needs shortening or reference-style formatting.

9. What Can Safely Move to UX Work

• Reasoning pattern inference logic (works correctly across all scenarios)
• Unknown selection and downstream scoring (works correctly)
• Graph structure from scenario analysis (works correctly)
• The start pipeline and schema validation (works correctly)
• Scenario 1-style clean resolution flow (works correctly)

10. Recommendation

one final bounded fix

The repeated failure patterns (#3 and #6 on question repetition, #2 on compound questions) appear in at least two scenarios each and prevent the basic question-answer-question loop from working reliably. However:

• The reasoning pattern inference is stable
• Graph construction is stable
• Unknown selection is stable
• Only ~2 issues (repetition + compound questions) are blocking reliable operation

These can be bounded to:

1. Detect repetition: if next question text similarity > 80% with previous, force a different candidate
2. Filter compound questions: reject any question containing "and", "or" at the clause level, regenerate
3. Relax "no meaningful change": accept answers that advance even a single edge status

Do not stop algorithm work entirely — but do not continue broad redesign. These are targeted fixes to the update pipeline's proposal_compatibility stage.

Report path: docs/v0.7-observation-report.md
Scenario JSON files: tests-results/v0.7-observation-suite/scenario-{1..6}.json