robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
60 lines
1.4 KiB
JSON
60 lines
1.4 KiB
JSON
{
|
|
"id": "diag-03",
|
|
"description": "Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.",
|
|
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
|
"responseDurationMs": 1,
|
|
"actualPrimaryType": "causal_claim",
|
|
"actualReasoningModes": [
|
|
"establish_baseline",
|
|
"identify_difference"
|
|
],
|
|
"technical": {
|
|
"schemaValid": true,
|
|
"classificationMatch": false,
|
|
"reasoningModeMatch": true,
|
|
"nextQuestionPresent": true,
|
|
"pass": false,
|
|
"errors": []
|
|
},
|
|
"reasoningQuality": {
|
|
"requiredConcepts": {
|
|
"pass": false,
|
|
"details": [
|
|
{
|
|
"concept": "sales decline",
|
|
"found": false
|
|
},
|
|
{
|
|
"concept": "price increase",
|
|
"found": false
|
|
},
|
|
{
|
|
"concept": "revenue increase",
|
|
"found": false
|
|
},
|
|
{
|
|
"concept": "CFO report",
|
|
"found": false
|
|
}
|
|
]
|
|
},
|
|
"unsupportedInferencesAbsent": {
|
|
"pass": true,
|
|
"details": [
|
|
{
|
|
"concept": "price was set too high",
|
|
"absent": true
|
|
},
|
|
{
|
|
"concept": "competitors gained market share",
|
|
"absent": true
|
|
},
|
|
{
|
|
"concept": "revenue data is wrong",
|
|
"absent": true
|
|
}
|
|
]
|
|
},
|
|
"pass": false
|
|
}
|
|
} |