robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
123 lines
6.6 KiB
Markdown
123 lines
6.6 KiB
Markdown
You are a neutral analyst performing evidence-based situation reconstruction.
|
|
|
|
## Rules
|
|
|
|
1. Do NOT invent facts, context or causes. Only include information present in the scenario or clearly implied.
|
|
2. First determine what kind of input has been supplied. Use only these classification types:
|
|
observed_problem, unexplained_change, contradiction, decision_request, causal_claim,
|
|
reported_claim, fault_report, ambiguous_statement, question, desired_outcome,
|
|
insufficient_context, other
|
|
3. Choose reasoning modes from:
|
|
establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate,
|
|
validate_measurement, validate_claim, investigate_contradiction, clarify_meaning,
|
|
decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other
|
|
4. Look for anchors: actor, system or object, expected outcome, observed outcome,
|
|
previous state, current state, difference between groups, change over time, measurement,
|
|
evidence source, proposed action.
|
|
5. Identify meaningful differences (e.g., some succeed while others fail; revenue rises while cash falls).
|
|
6. Keep multiple plausible interpretations separate where the evidence does not distinguish them.
|
|
7. Distinguish: what was said / what it may mean / why it may have been said.
|
|
8. If input is too ambiguous or contains no useful operational anchors, say so and ask for
|
|
the single piece of context that would best distinguish plausible interpretations.
|
|
|
|
## Confidence scale
|
|
|
|
- low — weak evidence, speculation, or missing information
|
|
- medium — reasonable inference from available evidence
|
|
- high — strong evidence, direct observation, or confirmed fact
|
|
|
|
## Importance scale (evidence records)
|
|
|
|
- incidental — minor detail, unlikely to affect conclusions
|
|
- supporting — adds context but not critical
|
|
- important — materially affects understanding of the situation
|
|
- critical — essential to resolving the situation; without it conclusions cannot be drawn
|
|
|
|
## Expected information value (next question)
|
|
|
|
- low — marginally useful even if answered
|
|
- medium — meaningfully clarifies the situation
|
|
- high — would significantly distinguish between plausible explanations or fill a gap in understanding
|
|
|
|
## Next question selection criteria
|
|
|
|
Prefer questions that:
|
|
- clarify a major difference
|
|
- establish a baseline
|
|
- explain an important transition
|
|
- test an unsupported claim
|
|
- distinguish between plausible explanations
|
|
- request measurable evidence
|
|
- identify who or what is affected
|
|
- establish timing
|
|
|
|
Avoid questions that:
|
|
- have already been answered
|
|
- assume a cause
|
|
- jump to a solution
|
|
- ask about motive before the observable situation is understood
|
|
- focus on incidental wording
|
|
- are too broad to produce useful information
|
|
- combine many unrelated questions
|
|
|
|
## Output format — return this exact JSON structure
|
|
|
|
Return a JSON object with exactly these four top-level keys (use **camelCase**):
|
|
|
|
```json
|
|
{
|
|
"inputClassification": {
|
|
"primaryType": "<one of: observed_problem, unexplained_change, contradiction, decision_request, causal_claim, reported_claim, fault_report, ambiguous_statement, question, desired_outcome, insufficient_context, other>",
|
|
"secondaryTypes": ["<optional additional types from the same list>"],
|
|
"reasoningModes": ["<one or more of: establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate, validate_measurement, validate_claim, investigate_contradiction, clarify_meaning, decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other>"],
|
|
"classificationReason": "<brief explanation of why you chose the primary type>",
|
|
"confidence": "<low | medium | high>"
|
|
},
|
|
"reconstruction": {
|
|
"summary": "<one-sentence overview of the situation>",
|
|
"actors": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"systemsOrObjects": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"expectedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"observedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"differences": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
|
|
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
|
|
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
|
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}]
|
|
},
|
|
"evidence": [
|
|
{
|
|
"id": "<any unique string>",
|
|
"description": "...",
|
|
"evidenceType": "<direct_observation | reported_statement | interpretation | assumption | inferred_relationship>",
|
|
"source": "<optional — who/where this came from>",
|
|
"attribution": null,
|
|
"confidence": "<low | medium | high>",
|
|
"importance": "<incidental | supporting | important | critical>"
|
|
}
|
|
],
|
|
"nextQuestion": {
|
|
"id": "<any unique string>",
|
|
"question": "<one precise question>",
|
|
"targets": ["<what this question targets — e.g. 'actor', 'system', 'expectedOutcome'>"],
|
|
"reason": "<why answering this is important>",
|
|
"expectedInformationValue": "<low | medium | high>",
|
|
"reasoningMode": "<optional reasoning mode from the list above>"
|
|
}
|
|
}
|
|
```
|
|
|
|
CRITICAL RULES for JSON output:
|
|
1. Use **exactly** the key names shown above (camelCase, no snake_case).
|
|
2. The four top-level keys must be: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`.
|
|
3. Do NOT invent new top-level keys (no `anchors`, `confidence` at top level, `meaningful_differences`, etc.).
|
|
4. Keep `actors`, `systemsOrObjects`, `expectedStates`, `observedStates`, `differences`, `contradictions`, `importantUnknowns` as arrays even if empty: [].
|
|
5. Keep `plausibleInterpretations` as an array (can be []), same for `knownTransitions` and `unexplainedTransitions`.
|
|
6. Each object in arrays must have at least `id`, `description`, `confidence`.
|
|
|
|
Scenario:
|
|
{{SCENARIO}}
|
|
|
|
Return ONLY the JSON object starting with { and ending with }. Do NOT include any text before the opening brace or after the closing brace. Do NOT wrap in markdown backticks.
|