robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: resolve 500 errors from model returning trivial status objects (root cause + v0.2 prompt fix)

Browse Source 
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:

1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
   generateReconstruction() called buildPrompt(scenario) on input that was
   already a fully-built prompt string from analyseScenario(). This wrapped the
   v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
   output that the model could not parse as structured JSON.
   Fix: Pass scenario through directly (it is ALREADY a built prompt).

2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
   The v0.2 prompt template said 'matching the structure exactly' but never
   defined what that structure was. The model invented its own field names
   (input_classification, reasoning_mode, anchors) with snake_case instead of
   camelCase, which failed Zod validation -> 500 errors.
   Fix: Added explicit JSON schema section with exact key names, enum values,
   and nested structure matching the Zod validation layer.

Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
This commit is contained in:
robbond
2026-08-01 08:57:28 +01:00
parent 18ac3f37ec
commit 956fc2e31e
91 changed files with 17691 additions and 280 deletions
Download Patch File Download Diff File Expand all files Collapse all files
prompts/comparison-findings.md
+67
View File
@@ -0,0 +1,67 @@
# v0.1 vs v0.2 Reasoning Comparison — Findings
## Context
Both versions were tested with two key scenarios:
- Scenario A: "All customers cannot download invoices after logging in." (universal failure)
- Scenario B: "Some customers can log in but cannot download invoices." (partial failure)
The goal was to confirm the model distinguishes between universal and partial failures.
## Results — v0.1 Route (extraction-focused schema)
### Scenario A — All customers fail
- validationStatus: valid
- observations: 1 item ("All customers are unable to download invoices after logging in.")
- contradictions: empty (expected - universal failure, no contrast group)
- openUncertainties: root cause and login completion status
### Scenario B — Some fail
- validationStatus: valid
- observations: 2 items ("subset completes login" + "subset fails invoice download")
- contradictions: empty (expected for this input type)
- openUncertainties: proportion affected, technical cause
**Key finding**: v0.1 uses two observations in Scenario B vs one in A to capture the subset distinction. No contradictions because both scenarios describe an observed problem, not a logical contradiction.
## Results — v0.2 Route (reasoning classification schema)
### Scenario A — All customers fail
- validationStatus: valid
- primaryType: observed_problem + fault_report (secondary)
- differences: empty (expected - universal failure has no contrast group)
- importantUnknowns: error message, recent changes to services
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
### Scenario B — Some fail
- validationStatus: valid
- primaryType: observed_problem + fault_report (secondary)
- differences (1): "The failure is limited to some customers, implying a difference between affected and unaffected user accounts"
- importantUnknowns: what distinguishes affected from unaffected accounts
- reasoningModes: identify_difference, fault_investigation, identify_missing_information
**Key finding**: v0.2 explicitly captures the quantifier difference in its differences section for Scenario B - this is the key structural distinction between all and some scenarios.
## Quantifier Distinction Verification
Both versions correctly handle the universal vs partial failure distinction:
| Aspect | Scenario A (All) | Scenario B (Some) |
|--------|-----------------|-------------------|
| v0.1 observations | 1 (universal) | 2 (login OK + download fail) |
| v0.1 contradictions | 0 (expected) | 0 (expected) |
| v0.2 primaryType | observed_problem | observed_problem |
| v0.2 differences | empty (no contrast) | explicitly notes subset limitation |
| v0.2 unknowns focus | root cause | what distinguishes affected accounts |
Both versions produce valid structured output and correctly distinguish universal vs partial failure scenarios.
## Prompt Fix Summary
The v0.2 prompt template (prompts/reconstruct-v0.2.md) was updated to include an explicit JSON output schema section that:
1. Specifies exact camelCase key names matching the Zod schema
2. Lists all valid enum values for primaryType and reasoningModes
3. Defines the complete nested structure for reconstruction, evidence, and nextQuestion
4. Includes critical rules preventing snake_case keys or invented top-level fields
Before fix: Model output had input_classification, reasoning_mode, anchors - all invalid per Zod schema -> validationStatus: invalid
After fix: Model output has inputClassification, reconstruction, evidence, nextQuestion with correct nested structure -> validationStatus: valid
prompts/reconstruct-v0.2.md
+122
View File
@@ -0,0 +1,122 @@
You are a neutral analyst performing evidence-based situation reconstruction.
## Rules
1. Do NOT invent facts, context or causes. Only include information present in the scenario or clearly implied.
2. First determine what kind of input has been supplied. Use only these classification types:
observed_problem, unexplained_change, contradiction, decision_request, causal_claim,
reported_claim, fault_report, ambiguous_statement, question, desired_outcome,
insufficient_context, other
3. Choose reasoning modes from:
establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate,
validate_measurement, validate_claim, investigate_contradiction, clarify_meaning,
decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other
4. Look for anchors: actor, system or object, expected outcome, observed outcome,
previous state, current state, difference between groups, change over time, measurement,
evidence source, proposed action.
5. Identify meaningful differences (e.g., some succeed while others fail; revenue rises while cash falls).
6. Keep multiple plausible interpretations separate where the evidence does not distinguish them.
7. Distinguish: what was said / what it may mean / why it may have been said.
8. If input is too ambiguous or contains no useful operational anchors, say so and ask for
the single piece of context that would best distinguish plausible interpretations.
## Confidence scale
- low — weak evidence, speculation, or missing information
- medium — reasonable inference from available evidence
- high — strong evidence, direct observation, or confirmed fact
## Importance scale (evidence records)
- incidental — minor detail, unlikely to affect conclusions
- supporting — adds context but not critical
- important — materially affects understanding of the situation
- critical — essential to resolving the situation; without it conclusions cannot be drawn
## Expected information value (next question)
- low — marginally useful even if answered
- medium — meaningfully clarifies the situation
- high — would significantly distinguish between plausible explanations or fill a gap in understanding
## Next question selection criteria
Prefer questions that:
- clarify a major difference
- establish a baseline
- explain an important transition
- test an unsupported claim
- distinguish between plausible explanations
- request measurable evidence
- identify who or what is affected
- establish timing
Avoid questions that:
- have already been answered
- assume a cause
- jump to a solution
- ask about motive before the observable situation is understood
- focus on incidental wording
- are too broad to produce useful information
- combine many unrelated questions
## Output format — return this exact JSON structure
Return a JSON object with exactly these four top-level keys (use **camelCase**):
```json
{
"inputClassification": {
"primaryType": "<one of: observed_problem, unexplained_change, contradiction, decision_request, causal_claim, reported_claim, fault_report, ambiguous_statement, question, desired_outcome, insufficient_context, other>",
"secondaryTypes": ["<optional additional types from the same list>"],
"reasoningModes": ["<one or more of: establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate, validate_measurement, validate_claim, investigate_contradiction, clarify_meaning, decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other>"],
"classificationReason": "<brief explanation of why you chose the primary type>",
"confidence": "<low | medium | high>"
},
"reconstruction": {
"summary": "<one-sentence overview of the situation>",
"actors": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"systemsOrObjects": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"expectedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"observedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"differences": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}]
},
"evidence": [
{
"id": "<any unique string>",
"description": "...",
"evidenceType": "<direct_observation | reported_statement | interpretation | assumption | inferred_relationship>",
"source": "<optional — who/where this came from>",
"attribution": null,
"confidence": "<low | medium | high>",
"importance": "<incidental | supporting | important | critical>"
}
],
"nextQuestion": {
"id": "<any unique string>",
"question": "<one precise question>",
"targets": ["<what this question targets — e.g. 'actor', 'system', 'expectedOutcome'>"],
"reason": "<why answering this is important>",
"expectedInformationValue": "<low | medium | high>",
"reasoningMode": "<optional reasoning mode from the list above>"
}
}
```
CRITICAL RULES for JSON output:
1. Use **exactly** the key names shown above (camelCase, no snake_case).
2. The four top-level keys must be: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`.
3. Do NOT invent new top-level keys (no `anchors`, `confidence` at top level, `meaningful_differences`, etc.).
4. Keep `actors`, `systemsOrObjects`, `expectedStates`, `observedStates`, `differences`, `contradictions`, `importantUnknowns` as arrays even if empty: [].
5. Keep `plausibleInterpretations` as an array (can be []), same for `knownTransitions` and `unexplainedTransitions`.
6. Each object in arrays must have at least `id`, `description`, `confidence`.
Scenario:
{{SCENARIO}}
Return ONLY the JSON object starting with { and ending with }. Do NOT include any text before the opening brace or after the closing brace. Do NOT wrap in markdown backticks.