# v0.7 Observation Report

## 1. Purpose

Observe whether the Confidence Engine asks sensible graph-backed questions across six realistic scenarios using the current codebase (commit c273209). Assessment covers reasoning-pattern fit, one-concept simplicity, plain-language clarity, logical progression, graph-backing, and avoidance of premature specialism.

## 2. Environment

- Next.js app: local development server (running)
- Ollama model: qwen-claude:latest
- Branch: feature/reasoning-pattern-memory-v0.7 (commit c273209, "feat: enforce reasoning pattern consistency")
- Prompt version: v0.3 for starts; v0.4 for updates
- All scenarios ran sequentially; one update per scenario

## 3. Summary Table

| #   | Scenario                                | Start | Initial Q (first 50 chars)             | Init Rating | Update                 | Next Q (first 50 chars)                 | Next Rating | Overall            |
| --- | --------------------------------------- | ----- | -------------------------------------- | ----------- | ---------------------- | --------------------------------------- | ----------- | ------------------ |
| 1   | Confidence Engine commercial validation | OK    | Who experiences this problem?          | P/P/P/P/P/P | OK                     | _(resolved)_                            | -           | good               |
| 2   | Hiring                                  | OK    | What changed during the period that co | P/F/F/P/P/F | FAIL (proposal_compat) | -                                       | -           | usable but awkward |
| 3   | Vehicle replacement                     | OK    | What evidence would clarify how the t  | P/P/P/F/P/F | OK                     | What evidence would clarify how the t   | P/P/P/F/P/F | reasoning defect   |
| 4   | Welsh Gov programme decision            | OK    | What would clarify quality, sample siz | P/F/F/F/P/F | OK                     | What changed during that period that co | F/F/P/F/F/F | usable but awkward |
| 5   | Operational contradiction               | OK    | Were these figures measured on the sam | P/P/P/P/P/P | FAIL (proposal_compat) | -                                       | -           | reasoning defect   |
| 6   | Personal decision                       | OK    | What evidence would clarify how the t  | P/P/P/F/P/F | OK                     | What evidence would clarify how the t   | P/P/P/F/P/F | reasoning defect   |

Rating keys: R=reasoning-pattern, S=simplicity, C=clarity, L=log progress, G=graph-backed, P=premature-specialism avoided. F=Fail.

## 4. Per-Scenario Findings

### Scenario 1 — Confidence Engine commercial validation (good)

- Initial question "Who experiences this problem?" is simple and clear. Good reasoning-pattern fit.
- Answer resolved the unknown; parent status became provisional with propagation=true. No follow-up needed.
- **Verdict: good.** Question flow was clean.

### Scenario 2 — Hiring (usable but awkward → update failed)

- Initial question restated the full scenario text inline, making it too long and compound.
- Update failed at proposal_compatibility stage with two errors:
  1. New unknown not explicitly related to an answer-derived node ("n_demand_var")
  2. selectedQuestion was a compound question (should be single)
- **Verdict: usable but awkward.** Both the initial and update had reasoning defects.

### Scenario 3 — Vehicle replacement (reasoning defect)

- Initial question "What evidence would clarify how the two observations were measured?" is reasonable but asks about measurements not central to the problem (reliability vs measurement method).
- Follow-up question is **identical** to the initial: "What evidence would clarify how the two observations were measured?" — a clear repetition bug.
- Answer was applied (propagation=true) but graph state did not advance meaningfully.
- **Verdict: reasoning defect.** Question loop is broken.

### Scenario 4 — Welsh Government-style programme decision (usable but awkward)

- Initial question asks 4 things in one sentence (quality, sample size, methodology, temporal scope). Fails simplicity criterion.
- Follow-up progresses to a temporal explanation ("What changed during that period...") which logically follows from the pilot-size answer.
- Follow-up includes embedded full scenario statement text — awkward formatting.
- **Verdict: usable but awkward.** Progression is logical but phrasing needs fixing.

### Scenario 5 — Operational contradiction (reasoning defect)

- Initial question "Were these figures measured on the same basis and at the same scale?" fits the comparison reasoning pattern well.
- Update failed with "Update contains no meaningful change" — the answer ("Both figures cover the same production sites and the same three-month period") was rejected as not adding new graph information.
- This is a **flow-critical failure**: the LLM did not recognise that the answer resolves part of the uncertainty.
- **Verdict: reasoning defect.** The question-answer-question loop breaks when the answer should have advanced the graph.

### Scenario 6 — Personal decision (reasoning defect)

- Initial question repeats "What evidence would clarify how the two observations were measured?" — the same question as scenario 3, despite different scenarios.
- Follow-up question is **identical** to the initial: same repetition bug.
- Pattern stays on comparison but no progression occurs.
- **Verdict: reasoning defect.** Same core failure as scenario 3.

## 5. Repeated Failure Patterns

1. **Question repetition loop** (scenarios 3, 6) — The follow-up question is identical to the initial question. The graph update does not advance the state meaningfully, causing an infinite loop of the same query.
2. **Compound questions in follow-ups** (scenario 2) — The model generates a question containing multiple concept targets ("n_demand_var" linked as new unknown not connected to answer-derived node). This suggests either the compound-question filter isn't working or the graph update produces invalid proposals.
3. **Answer rejection as "no meaningful change"** (scenario 5) — The LLM fails to incorporate an answer that should advance the graph state, causing a hard proposal_compatibility failure.

## 6. Isolated Failures

1. **Scenario 2: new unknown not linked to answer** — This appears to be a graph-update linking bug specific to that scenario's answer content.
2. **Scenario 4: long initial question asking 4 things** — A prompt-generation issue where multiple sub-topics are collapsed into one question.

## 7. What Appears Stable

- **Start pipeline**: All 6 scenarios started successfully with explicit v0.3 promptVersion. The analysis + graph construction works.
- **Reasoning pattern inference**: Initial questions all match the correct reasoning pattern for each scenario (decision, contradiction, comparison, diagnosis).
- **Unknown selection**: The downstream-scoring-based selection consistently picks high-value nodes.
- **Graph reference validation**: No structural issues during normal operation.
- **Scenario 1 clean resolution**: When an answer fully resolves uncertainty, the system correctly terminates with no follow-up.

## 8. What Should Be Fixed Before UX Work

1. **Question repetition loop** — The most critical fix. If a question repeats, the engine must generate a new candidate (decompose further or select next sibling).
2. **Compound question generation** — Ensure the question-formulator only produces single-concept questions. Filter out multi-clause questions before returning.
3. **"No meaningful change" rejection** — The answerability checker should incorporate any valid factual claim from an answer, even if partial progress is minimal.
4. **Scenario text embedding in follow-ups** — Long follow-up questions include the full central statement inline. This needs shortening or reference-style formatting.

## 9. What Can Safely Move to UX Work

- Reasoning pattern inference logic (works correctly across all scenarios)
- Unknown selection and downstream scoring (works correctly)
- Graph structure from scenario analysis (works correctly)
- The start pipeline and schema validation (works correctly)
- Scenario 1-style clean resolution flow (works correctly)

## 10. Recommendation

**one final bounded fix**

The repeated failure patterns (#3 and #6 on question repetition, #2 on compound questions) appear in at least two scenarios each and prevent the basic question-answer-question loop from working reliably. However:

- The reasoning pattern inference is stable
- Graph construction is stable
- Unknown selection is stable
- Only ~2 issues (repetition + compound questions) are blocking reliable operation

These can be bounded to:

1. Detect repetition: if next question text similarity > 80% with previous, force a different candidate
2. Filter compound questions: reject any question containing "and", "or" at the clause level, regenerate
3. Relax "no meaningful change": accept answers that advance even a single edge status

**Do not stop algorithm work entirely** — but do **not continue broad redesign**. These are targeted fixes to the update pipeline's proposal_compatibility stage.

Report path: docs/v0.7-observation-report.md  
Scenario JSON files: tests-results/v0.7-observation-suite/scenario-{1..6}.json