robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
3.5 KiB
v0.1 vs v0.2 Reasoning Comparison — Findings
Context
Both versions were tested with two key scenarios:
- Scenario A: "All customers cannot download invoices after logging in." (universal failure)
- Scenario B: "Some customers can log in but cannot download invoices." (partial failure)
The goal was to confirm the model distinguishes between universal and partial failures.
Results — v0.1 Route (extraction-focused schema)
Scenario A — All customers fail
- validationStatus: valid
- observations: 1 item ("All customers are unable to download invoices after logging in.")
- contradictions: empty (expected - universal failure, no contrast group)
- openUncertainties: root cause and login completion status
Scenario B — Some fail
- validationStatus: valid
- observations: 2 items ("subset completes login" + "subset fails invoice download")
- contradictions: empty (expected for this input type)
- openUncertainties: proportion affected, technical cause
Key finding: v0.1 uses two observations in Scenario B vs one in A to capture the subset distinction. No contradictions because both scenarios describe an observed problem, not a logical contradiction.
Results — v0.2 Route (reasoning classification schema)
Scenario A — All customers fail
- validationStatus: valid
- primaryType: observed_problem + fault_report (secondary)
- differences: empty (expected - universal failure has no contrast group)
- importantUnknowns: error message, recent changes to services
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
Scenario B — Some fail
- validationStatus: valid
- primaryType: observed_problem + fault_report (secondary)
- differences (1): "The failure is limited to some customers, implying a difference between affected and unaffected user accounts"
- importantUnknowns: what distinguishes affected from unaffected accounts
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
Key finding: v0.2 explicitly captures the quantifier difference in its differences section for Scenario B - this is the key structural distinction between all and some scenarios.
Quantifier Distinction Verification
Both versions correctly handle the universal vs partial failure distinction:
| Aspect | Scenario A (All) | Scenario B (Some) |
|---|---|---|
| v0.1 observations | 1 (universal) | 2 (login OK + download fail) |
| v0.1 contradictions | 0 (expected) | 0 (expected) |
| v0.2 primaryType | observed_problem | observed_problem |
| v0.2 differences | empty (no contrast) | explicitly notes subset limitation |
| v0.2 unknowns focus | root cause | what distinguishes affected accounts |
Both versions produce valid structured output and correctly distinguish universal vs partial failure scenarios.
Prompt Fix Summary
The v0.2 prompt template (prompts/reconstruct-v0.2.md) was updated to include an explicit JSON output schema section that:
- Specifies exact camelCase key names matching the Zod schema
- Lists all valid enum values for primaryType and reasoningModes
- Defines the complete nested structure for reconstruction, evidence, and nextQuestion
- Includes critical rules preventing snake_case keys or invented top-level fields
Before fix: Model output had input_classification, reasoning_mode, anchors - all invalid per Zod schema -> validationStatus: invalid After fix: Model output has inputClassification, reconstruction, evidence, nextQuestion with correct nested structure -> validationStatus: valid