robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
68 lines
3.5 KiB
Markdown
68 lines
3.5 KiB
Markdown
# v0.1 vs v0.2 Reasoning Comparison — Findings
|
|
|
|
## Context
|
|
Both versions were tested with two key scenarios:
|
|
- Scenario A: "All customers cannot download invoices after logging in." (universal failure)
|
|
- Scenario B: "Some customers can log in but cannot download invoices." (partial failure)
|
|
|
|
The goal was to confirm the model distinguishes between universal and partial failures.
|
|
|
|
## Results — v0.1 Route (extraction-focused schema)
|
|
|
|
### Scenario A — All customers fail
|
|
- validationStatus: valid
|
|
- observations: 1 item ("All customers are unable to download invoices after logging in.")
|
|
- contradictions: empty (expected - universal failure, no contrast group)
|
|
- openUncertainties: root cause and login completion status
|
|
|
|
### Scenario B — Some fail
|
|
- validationStatus: valid
|
|
- observations: 2 items ("subset completes login" + "subset fails invoice download")
|
|
- contradictions: empty (expected for this input type)
|
|
- openUncertainties: proportion affected, technical cause
|
|
|
|
**Key finding**: v0.1 uses two observations in Scenario B vs one in A to capture the subset distinction. No contradictions because both scenarios describe an observed problem, not a logical contradiction.
|
|
|
|
## Results — v0.2 Route (reasoning classification schema)
|
|
|
|
### Scenario A — All customers fail
|
|
- validationStatus: valid
|
|
- primaryType: observed_problem + fault_report (secondary)
|
|
- differences: empty (expected - universal failure has no contrast group)
|
|
- importantUnknowns: error message, recent changes to services
|
|
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
|
|
|
|
### Scenario B — Some fail
|
|
- validationStatus: valid
|
|
- primaryType: observed_problem + fault_report (secondary)
|
|
- differences (1): "The failure is limited to some customers, implying a difference between affected and unaffected user accounts"
|
|
- importantUnknowns: what distinguishes affected from unaffected accounts
|
|
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
|
|
|
|
**Key finding**: v0.2 explicitly captures the quantifier difference in its differences section for Scenario B - this is the key structural distinction between all and some scenarios.
|
|
|
|
## Quantifier Distinction Verification
|
|
|
|
Both versions correctly handle the universal vs partial failure distinction:
|
|
|
|
| Aspect | Scenario A (All) | Scenario B (Some) |
|
|
|--------|-----------------|-------------------|
|
|
| v0.1 observations | 1 (universal) | 2 (login OK + download fail) |
|
|
| v0.1 contradictions | 0 (expected) | 0 (expected) |
|
|
| v0.2 primaryType | observed_problem | observed_problem |
|
|
| v0.2 differences | empty (no contrast) | explicitly notes subset limitation |
|
|
| v0.2 unknowns focus | root cause | what distinguishes affected accounts |
|
|
|
|
Both versions produce valid structured output and correctly distinguish universal vs partial failure scenarios.
|
|
|
|
## Prompt Fix Summary
|
|
|
|
The v0.2 prompt template (prompts/reconstruct-v0.2.md) was updated to include an explicit JSON output schema section that:
|
|
1. Specifies exact camelCase key names matching the Zod schema
|
|
2. Lists all valid enum values for primaryType and reasoningModes
|
|
3. Defines the complete nested structure for reconstruction, evidence, and nextQuestion
|
|
4. Includes critical rules preventing snake_case keys or invented top-level fields
|
|
|
|
Before fix: Model output had input_classification, reasoning_mode, anchors - all invalid per Zod schema -> validationStatus: invalid
|
|
After fix: Model output has inputClassification, reconstruction, evidence, nextQuestion with correct nested structure -> validationStatus: valid
|