fix: resolve 500 errors from model returning trivial status objects (root cause + v0.2 prompt fix)

Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:

1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
   generateReconstruction() called buildPrompt(scenario) on input that was
   already a fully-built prompt string from analyseScenario(). This wrapped the
   v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
   output that the model could not parse as structured JSON.
   Fix: Pass scenario through directly (it is ALREADY a built prompt).

2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
   The v0.2 prompt template said 'matching the structure exactly' but never
   defined what that structure was. The model invented its own field names
   (input_classification, reasoning_mode, anchors) with snake_case instead of
   camelCase, which failed Zod validation -> 500 errors.
   Fix: Added explicit JSON schema section with exact key names, enum values,
   and nested structure matching the Zod validation layer.

Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
This commit is contained in:
2026-08-01 08:57:28 +01:00
parent 18ac3f37ec
commit 956fc2e31e
91 changed files with 17691 additions and 280 deletions
@@ -0,0 +1,8 @@
{
"qwen-claude:latest": {
"availability": "NOT_AVAILABLE_ON_SERVER"
},
"qwen3.6:35b-a3b": {
"availability": "NOT_AVAILABLE_ON_SERVER"
}
}
@@ -0,0 +1,142 @@
{
"qwen-claude:latest": {
"test_A_qwen_claude": {
"testDescription": "Plain instruction test — should return CHAT_WORKS",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": true,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 30,
"thinkingPresent": true,
"parsedContentKeys": [
"response"
],
"looksLikeStatusAck": false,
"rawPreview": "{\n \"response\": \"CHAT_WORKS\"\n}"
},
"test_A_qwen_claude_generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "CHAT_WORKS",
"responseLooksLikeStructuredJSON": false,
"rawPreview": "CHAT_WORKS"
},
"test_B_qwen_claude": {
"testDescription": "JSON schema test — should return exact object",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": false,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 38,
"thinkingPresent": true,
"parsedContentKeys": [
"message"
],
"looksLikeStatusAck": false,
"rawPreview": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}"
},
"test_B_qwen_claude_generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}",
"responseLooksLikeStructuredJSON": true,
"rawPreview": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}"
},
"test_C_qwen_claude": {
"testDescription": "Minimal reconstruction schema — structured output test",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": false,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 1145,
"thinkingPresent": true,
"parsedContentKeys": [
"analysis",
"meaningful_difference",
"next_question"
],
"looksLikeStatusAck": false,
"rawPreview": "{\"analysis\":\"The workflow breaks into two distinct stages: authentication (login) and resource access (invoice download). Since login succeeds for the affected users, identity verification and session creation are functioning correctly. The failure occurs downstream in the layer responsible for loca"
},
"test_C_qwen_claude_generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "**Analysis of the Situation**\nThe issue represents a partial, post-authentication failure: identity verification works for a subset of users, but access to or delivery of a specific resource (invoices",
"responseLooksLikeStructuredJSON": false,
"rawPreview": "**Analysis of the Situation**\nThe issue represents a partial, post-authentication failure: identity verification works for a subset of users, but access to or delivery of a specific resource (invoices"
}
},
"qwen3.6:35b-a3b": {
"test_A_qwen___": {
"testDescription": "Plain instruction test — should return CHAT_WORKS",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": true,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 30,
"thinkingPresent": true,
"parsedContentKeys": [
"response"
],
"looksLikeStatusAck": false,
"rawPreview": "{\n \"response\": \"CHAT_WORKS\"\n}"
},
"test_A_qwen____generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "CHAT_WORKS",
"responseLooksLikeStructuredJSON": false,
"rawPreview": "CHAT_WORKS"
},
"test_B_qwen___": {
"testDescription": "JSON schema test — should return exact object",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": false,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 38,
"thinkingPresent": true,
"parsedContentKeys": [
"message"
],
"looksLikeStatusAck": false,
"rawPreview": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}"
},
"test_B_qwen____generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}",
"responseLooksLikeStructuredJSON": true,
"rawPreview": "{\"message\": \"STRUCTURED_OUTPUT_WORKS\"}"
},
"test_C_qwen___": {
"testDescription": "Minimal reconstruction schema — structured output test",
"endpoint": "/api/chat",
"format": "json",
"hasSystemMessage": false,
"httpStatus": 200,
"messageContentType": "string",
"messageContentLength": 843,
"thinkingPresent": true,
"parsedContentKeys": [
"analysis",
"meaningful_difference",
"next_question"
],
"looksLikeStatusAck": false,
"rawPreview": "{\"analysis\": \"The scenario indicates that authentication is functioning correctly, but a downstream access or resource-retrieval step is failing for a subset of users. This separates the problem from credential or session validity and points toward post-login factors such as authorization scope, doc"
},
"test_C_qwen____generate": {
"endpoint": "/api/generate",
"httpStatus": 200,
"responseFirst200": "**Meaningful Difference:** \nSuccessful login verifies *account-level authentication and baseline platform access*, while downloading an invoice requires *invoice-specific data availability, retrieval",
"responseLooksLikeStructuredJSON": false,
"rawPreview": "**Meaningful Difference:** \nSuccessful login verifies *account-level authentication and baseline platform access*, while downloading an invoice requires *invoice-specific data availability, retrieval"
}
}
}