robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
36 lines
15 KiB
JSON
36 lines
15 KiB
JSON
{"id":"tc-001","input":"All customers cannot download their invoices.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["customers","invoices","download","access_issue"],"shouldNotInfer":[],"notes":"Full scope problem — every customer is affected. Should not infer root cause."}
|
|
{"id":"tc-002","input":"Some customers cannot download their invoices.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["some_customers","invoices","download"],"shouldNotInfer":["root_cause","payment_system_failure"],"notes":"Partial scope — subset of users affected. The word 'some' is the key distinction from tc-001."}
|
|
{"id":"tc-003","input":"Complaints increased by 35%.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["complaints","increase","35_percent"],"shouldNotInfer":["cause_of_complaints","customer_dissatisfaction_is_worse"],"notes":"Change in isolation — need baseline to understand significance."}
|
|
{"id":"tc-004","input":"Complaints increased by 35% while production increased by 40%.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["identify_difference","validate_measurement"],"shouldIdentify":["complaints_increase","production_increase","relative_rates"],"shouldNotInfer":["production_quality_declined"],"notes":"Paired with tc-003 — the production context changes meaning significantly."}
|
|
{"id":"tc-005","input":"Sales are falling.","expectedPrimaryTypes":["observed_problem","unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["sales_decline","direction_negative"],"shouldNotInfer":["cause_of_fall","competitor_action"],"notes":"Vague claim — need baseline, timeline, and definition of 'falling'."}
|
|
{"id":"tc-006","input":"Sales fell sharply immediately after the price increase.","expectedPrimaryTypes":["observed_problem","causal_claim"],"expectedReasoningModes":["investigate_contradiction","test_possible_explanations"],"shouldIdentify":["sales_decline","price_increase","temporal_correlation"],"shouldNotInfer":["price_increase_caused_the_fall"],"notes":"Paired with tc-005 — adds temporal anchor and proposed cause."}
|
|
{"id":"tc-007","input":"The quarterly revenue exceeded targets but net profit declined by 12%.","expectedPrimaryTypes":["contradiction","unexplained_change"],"expectedReasoningModes":["investigate_contradiction","identify_missing_information"],"shouldIdentify":["revenue_above_target","profit_decline","divergence"],"shouldNotInfer":["cost_overrun_is_the_cause"],"notes":"Apparent contradiction — revenue up but profit down. Missing cost breakdown."}
|
|
{"id":"tc-008","input":"Revenue from the premium tier dropped while total revenue grew.","expectedPrimaryTypes":["observed_problem","unexplained_change"],"expectedReasoningModes":["decompose_aggregate","identify_difference"],"shouldIdentify":["premium_tier_decline","total_revenue_growth","segment_cannibalization_risk"],"shouldNotInfer":["pricing_change_occurred"],"notes":"Aggregate masking — total growth hides segment decline."}
|
|
{"id":"tc-009","input":"We need to improve our customer retention rate.","expectedPrimaryTypes":["decision_request","desired_outcome"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["retention_improvement_desired","current_state_unknown"],"shouldNotInfer":["retention_rate_is_low","churn_has_increased"],"notes":"Desired outcome without stating the problem. Need to know if retention is actually bad."}
|
|
{"id":"tc-010","input":"The system latency went from 200ms to 5 seconds on Tuesday.","expectedPrimaryTypes":["unexplained_change","observed_problem"],"expectedReasoningModes":["reconstruct_transition","identify_missing_information"],"shouldIdentify":["latency_baseline_200ms","latency_spike_5s","timestamp_tuesday"],"shouldNotInfer":["database_cause","release_cause"],"notes":"Specific measurement with timing anchor. Should identify transition but not infer cause."}
|
|
{"id":"tc-011","input":"The new release should fix the login issue.","expectedPrimaryTypes":["decision_request","causal_claim"],"expectedReasoningModes":["validate_claim","investigate_contradiction"],"shouldIdentify":["proposed_solution","login_issue","solution_claim"],"shouldNotInfer":["login_issue_is_real","release_will_work"],"notes":"Proposed solution before problem is fully understood. Assumes the issue and fix are connected."}
|
|
{"id":"tc-012","input":"I think therefore I am.","expectedPrimaryTypes":["ambiguous_statement","question"],"expectedReasoningModes":["clarify_meaning","identify_missing_information"],"shouldIdentify":["philosophical_statement","insufficient_operational_context"],"shouldNotInfer":["business_problem_exists","actionable_insight_possible"],"notes":"Ambiguous philosophical statement. Should not try to find operational meaning."}
|
|
{"id":"tc-013","input":"I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.","expectedPrimaryTypes":["question","ambiguous_statement"],"expectedReasoningModes":["clarify_meaning"],"shouldIdentify":["meta_context","testing_hypothesis","self_reference"],"shouldNotInfer":[],"notes":"Paired with tc-12 — the meta-context changes classification entirely."}
|
|
{"id":"tc-014","input":"The warehouse manager reported that inventory counts don't match the system.","expectedPrimaryTypes":["reported_claim","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction"],"shouldIdentify":["warehouse_manager_report","inventory_mismatch","system_discrepancy","source_attribution"],"shouldNotInfer":["theft_occurred","software_bug"],"notes":"Reported claim — must distinguish what was said from what it means."}
|
|
{"id":"tc-015","input":"We've seen a 35% increase in customer complaints.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["complaints_increase","percentage_metric"],"shouldNotInfer":["product_quality_declined","customer_satisfaction_drop"],"notes":"Needs baseline — is this absolute or relative? Over what period?"}
|
|
{"id":"tc-016","input":"The number of active users increased by 500%, from 4 to 2,001.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_measurement","decompose_aggregate"],"shouldIdentify":["active_users_metric","absolute_vs_relative_growth","small_base_problem"],"shouldNotInfer":["product_success"],"notes":"Misleading absolute count where rate matters. Small base inflates percentage."}
|
|
{"id":"tc-017","input":"User engagement metrics improved but the support ticket backlog grew by 200%.","expectedPrimaryTypes":["contradiction"],"expectedReasoningModes":["investigate_contradiction","identify_difference"],"shouldIdentify":["engagement_improvement","support_backlog_growth","divergent_metrics"],"shouldNotInfer":["users_are_angry","product_quality_is_worse"],"notes":"Two metrics telling opposite stories. Could mean engagement is superficial."}
|
|
{"id":"tc-018","input":"The manufacturing team needs better quality control.","expectedPrimaryTypes":["decision_request","fault_report"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["manufacturing_team","quality_control_desired"],"shouldNotInfer":["quality_is_bad","defect_rate_is_high"],"notes":"Solution proposed without problem specification. What specific quality issue?"}
|
|
{"id":"tc-019","input":"All users in the EU region are getting a 403 error when trying to access the dashboard.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["eu_region","403_error","access_denied","geographic_scope"],"shouldNotInfer":["gdpr_cause","regulatory_change"],"notes":"Geographic subset fault. Should not infer GDPR as cause without evidence."}
|
|
{"id":"tc-020","input":"Some users in the EU region are getting a 403 error when trying to access the dashboard.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference","decompose_aggregate"],"shouldIdentify":["eu_region_subset","403_error","partial_reachability"],"shouldNotInfer":["all_eu_users_affected"],"notes":"Paired with tc-19 — 'some' vs 'all' is the material difference."}
|
|
{"id":"tc-021","input":"Production output was 1,200 units last month and 1,180 units this month.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_measurement","establish_baseline"],"shouldIdentify":["production_output","month_over_month_decline","absolute_difference"],"shouldNotInfer":["efficiency_loss_occurred","equipment_failure"],"notes":"Small absolute change needs context — 1.7% drop might be normal variation."}
|
|
{"id":"tc-022","input":"The CFO reported that the company's cash position is healthy.","expectedPrimaryTypes":["reported_claim"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["cfo_statement","cash_position_claim","source_attribution_cfo"],"shouldNotInfer":["cash_is_healthy","financial_stability_is_real"],"notes":"Reported opinion — must distinguish what was said from reality."}
|
|
{"id":"tc-023","input":"We have enough funding to operate for 18 months.","expectedPrimaryTypes":["decision_request","observed_problem"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["funding_period","operational_sustainability","burn_rate_unknown"],"shouldNotInfer":["no_risk_exists"],"notes":"Claim about sustainability without burn rate context."}
|
|
{"id":"tc-024","input":"The new feature was deployed at 3am and user complaints tripled the next day.","expectedPrimaryTypes":["causal_claim","unexplained_change"],"expectedReasoningModes":["test_possible_explanations","reconstruct_transition"],"shouldIdentify":["feature_deployment","timing_3am","complaint_tripling","temporal_relationship"],"shouldNotInfer":["deployment_caused_complaints"],"notes":"Temporal proximity ≠ causation. Should identify both events but not claim cause."}
|
|
{"id":"tc-025","input":"We need to launch a mobile app to capture market share.","expectedPrimaryTypes":["decision_request","desired_outcome"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["mobile_app_proposed","market_share_desired"],"shouldNotInfer":["no_mobile_app_exists","competitors_have_apps"],"notes":"Desired outcome without problem statement. What evidence supports this decision?"}
|
|
{"id":"tc-026","input":"The system has been running for 90 days without failure since the migration.","expectedPrimaryTypes":["observed_problem"],"expectedReasoningModes":["validate_claim","establish_baseline"],"shouldIdentify":["uptime_90_days","post_migration_context","baseline_established"],"shouldNotInfer":["system_is_stable_forever"],"notes":"Positive claim about system stability with temporal anchor."}
|
|
{"id":"tc-027","input":"No one has submitted the required compliance report despite multiple reminders.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_missing_information","investigate_contradiction"],"shouldIdentify":["compliance_report","multiple_reminders","non_submission","absent_action"],"shouldNotInfer":["deliberate_refusal","negligence"],"notes":"Expected-but-missing information. Action was required but absent."}
|
|
{"id":"tc-028","input":"The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.","expectedPrimaryTypes":["observed_problem","contradiction"],"expectedReasoningModes":["validate_measurement","decompose_aggregate"],"shouldIdentify":["audit_findings","incorrect_reports_rate_3_of_10","data_accuracy_issue"],"shouldNotInfer":["intentional_falsification","systemic_failure"],"notes":"Aggregate data — 30% error rate requires context about severity."}
|
|
{"id":"tc-029","input":"We should implement the new CRM because our competitors have one.","expectedPrimaryTypes":["decision_request","causal_claim"],"expectedReasoningModes":["test_possible_explanations","validate_claim"],"shouldIdentify":["crm_proposal","competitor_comparison","competitive_pressure"],"shouldNotInfer":["crm_will_help","we_lack_crm","competitors_success_is_from_crm"],"notes":"FOMO-driven decision request without problem analysis."}
|
|
{"id":"tc-030","input":"The server response time was acceptable last quarter but degraded this month.","expectedPrimaryTypes":["unexplained_change","observed_problem"],"expectedReasoningModes":["reconstruct_transition","identify_missing_information"],"shouldIdentify":["response_time_baseline_acceptable","degradation_timeline","quarter_to_month_comparison"],"shouldNotInfer":["load_increase_occurred"],"notes":"Baseline comparison with transition over time. Need specifics."}
|
|
{"id":"tc-031","input":"The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.","expectedPrimaryTypes":["contradiction","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction","identify_missing_information"],"shouldIdentify":["regulatory_requirement","data_location_violation","cross_border_backup"],"shouldNotInfer":["compliance_failure_is_certain"],"notes":"Regulatory conflict — requires verification of both claim and current state."}
|
|
{"id":"tc-032","input":"External analysts expect our industry to decline by 15% next year due to regulatory changes.","expectedPrimaryTypes":["causal_claim","reported_claim"],"expectedReasoningModes":["validate_claim","test_possible_explanations"],"shouldIdentify":["industry_decline_prediction","external_source","regulatory_cause","15_percent_forecast"],"shouldNotInfer":["decline_will_occur"],"notes":"External prediction — must treat as claim, not fact."}
|
|
{"id":"tc-033","input":"The database schema was changed on Friday but the reports are still working.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_claim","test_possible_explanations"],"shouldIdentify":["schema_change","reports_working_post_change","unexpected_continuity"],"shouldNotInfer":["change_was_harmless"],"notes":"Expected impact did not occur — should flag as unexplained."}
|
|
{"id":"tc-034","input":"Some team members say the new process is better while others say it's slower.","expectedPrimaryTypes":["contradiction","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction","identify_missing_information"],"shouldIdentify":["subjective_split","new_process_evaluation","conflicting_opinions","measurement_gap"],"shouldNotInfer":["process_is_better_or_worse"],"notes":"Conflicting subjective claims — need measurable criteria."}
|
|
{"id":"tc-035","input":"The application works fine on Chrome but not on Safari.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["chrome_compatibility","safari_incompatibility","browser_specific_issue"],"shouldNotInfer":["webkit_bug"],"notes":"Browser-specific fault. Should identify the difference but not the technical cause."}
|