robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: resolve 500 errors from model returning trivial status objects (root cause + v0.2 prompt fix)

Browse Source 
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:

1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
   generateReconstruction() called buildPrompt(scenario) on input that was
   already a fully-built prompt string from analyseScenario(). This wrapped the
   v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
   output that the model could not parse as structured JSON.
   Fix: Pass scenario through directly (it is ALREADY a built prompt).

2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
   The v0.2 prompt template said 'matching the structure exactly' but never
   defined what that structure was. The model invented its own field names
   (input_classification, reasoning_mode, anchors) with snake_case instead of
   camelCase, which failed Zod validation -> 500 errors.
   Fix: Added explicit JSON schema section with exact key names, enum values,
   and nested structure matching the Zod validation layer.

Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
This commit is contained in:
robbond
2026-08-01 08:57:28 +01:00
parent 18ac3f37ec
commit 956fc2e31e
91 changed files with 17691 additions and 280 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests-results/evaluation-2026-07-31T18-14-22.json
+653
View File
@@ -0,0 +1,653 @@
{
"timestamp": "2026-07-31T18:14:22.632Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "28.6%",
"reasoningModeMatchRate": "48.6%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "1",
"fullPassRate": "28.6%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 4,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"identify_difference",
"investigate_contradiction"
],
"errors": []
}
]
}
tests-results/evaluation-2026-07-31T18-16-32.json
+652
View File
@@ -0,0 +1,652 @@
{
"timestamp": "2026-07-31T18:16:32.255Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "37.1%",
"reasoningModeMatchRate": "71.4%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "1",
"fullPassRate": "37.1%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 4,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
}
]
}
tests-results/evaluation-2026-07-31T18-23-11.json
+633
View File
@@ -0,0 +1,633 @@
{
"timestamp": "2026-07-31T18:23:11.669Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "48.6%",
"reasoningModeMatchRate": "48.6%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "1",
"fullPassRate": "48.6%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 6,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "fault_report",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "other",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
}
]
}
tests-results/evaluation-2026-07-31T18-34-17.json
+639
View File
@@ -0,0 +1,639 @@
{
"timestamp": "2026-07-31T18:34:17.855Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "82.9%",
"reasoningModeMatchRate": "51.4%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "1",
"fullPassRate": "82.9%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 5,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 2,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": false,
"schemaValid": true,
"classificationMatch": false,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
}
]
}
tests-results/evaluation-2026-07-31T18-59-37.json
+644
View File
@@ -0,0 +1,644 @@
{
"timestamp": "2026-07-31T18:59:37.227Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "100.0%",
"reasoningModeMatchRate": "57.1%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "0",
"fullPassRate": "100.0%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 4,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
}
]
}
tests-results/evaluation-2026-07-31T19-00-09.json
+644
View File
@@ -0,0 +1,644 @@
{
"timestamp": "2026-07-31T19:00:09.575Z",
"provider": "mock",
"promptVersion": "v0.2",
"casesRun": 35,
"summary": {
"schemaValidityRate": "100.0%",
"classificationMatchRate": "100.0%",
"reasoningModeMatchRate": "57.1%",
"requiredConceptMatchRate": "0.0%",
"unsupportedInferenceFailures": "0",
"averageResponseDurationMs": "1",
"fullPassRate": "100.0%"
},
"testCaseResults": [
{
"id": "tc-001",
"input": "All customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 6,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-002",
"input": "Some customers cannot download their invoices.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 2,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-003",
"input": "Complaints increased by 35%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-004",
"input": "Complaints increased by 35% while production increased by 40%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-005",
"input": "Sales are falling.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-006",
"input": "Sales fell sharply immediately after the price increase.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-007",
"input": "The quarterly revenue exceeded targets but net profit declined by 12%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-008",
"input": "Revenue from the premium tier dropped while total revenue grew.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-009",
"input": "We need to improve our customer retention rate.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-010",
"input": "The system latency went from 200ms to 5 seconds on Tuesday.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-011",
"input": "The new release should fix the login issue.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-012",
"input": "I think therefore I am.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-013",
"input": "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "ambiguous_statement",
"actualReasoningModes": [
"identify_difference",
"clarify_meaning"
],
"errors": []
},
{
"id": "tc-014",
"input": "The warehouse manager reported that inventory counts don't match the system.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-015",
"input": "We've seen a 35% increase in customer complaints.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-016",
"input": "The number of active users increased by 500%, from 4 to 2,001.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-017",
"input": "User engagement metrics improved but the support ticket backlog grew by 200%.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "contradiction",
"actualReasoningModes": [
"investigate_contradiction",
"identify_difference"
],
"errors": []
},
{
"id": "tc-018",
"input": "The manufacturing team needs better quality control.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-019",
"input": "All users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-020",
"input": "Some users in the EU region are getting a 403 error when trying to access the dashboard.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-021",
"input": "Production output was 1,200 units last month and 1,180 units this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-022",
"input": "The CFO reported that the company's cash position is healthy.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "reported_claim",
"actualReasoningModes": [
"identify_difference",
"validate_claim"
],
"errors": []
},
{
"id": "tc-023",
"input": "We have enough funding to operate for 18 months.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-024",
"input": "The new feature was deployed at 3am and user complaints tripled the next day.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"establish_baseline",
"identify_difference"
],
"errors": []
},
{
"id": "tc-025",
"input": "We need to launch a mobile app to capture market share.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-026",
"input": "The system has been running for 90 days without failure since the migration.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-027",
"input": "No one has submitted the required compliance report despite multiple reminders.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-028",
"input": "The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-029",
"input": "We should implement the new CRM because our competitors have one.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "decision_request",
"actualReasoningModes": [
"identify_difference",
"decision_support",
"identify_missing_information"
],
"errors": []
},
{
"id": "tc-030",
"input": "The server response time was acceptable last quarter but degraded this month.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-031",
"input": "The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-032",
"input": "External analysts expect our industry to decline by 15% next year due to regulatory changes.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "causal_claim",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-033",
"input": "The database schema was changed on Friday but the reports are still working.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "unexplained_change",
"actualReasoningModes": [
"identify_difference",
"establish_baseline",
"validate_measurement"
],
"errors": []
},
{
"id": "tc-034",
"input": "Some team members say the new process is better while others say it's slower.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": false,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 1,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
},
{
"id": "tc-035",
"input": "The application works fine on Chrome but not on Safari.",
"pass": true,
"schemaValid": true,
"classificationMatch": true,
"reasoningModeMatch": true,
"requiredConceptMatch": false,
"unsupportedInferenceAbsence": true,
"nextQuestionPresent": true,
"responseDurationMs": 0,
"actualPrimaryType": "observed_problem",
"actualReasoningModes": [
"identify_difference"
],
"errors": []
}
]
}
tests-results/evaluation-2026-08-01T05-17-18.json
+1735
View File
File diff suppressed because it is too large Load Diff
tests-results/evaluation-2026-08-01T05-22-52.json
+1735
View File
File diff suppressed because it is too large Load Diff
tests-results/evaluation-2026-08-01T05-30-52.json
+1735
View File
File diff suppressed because it is too large Load Diff
tests-results/evaluation-2026-08-01T05-31-49.json
+1735
View File
File diff suppressed because it is too large Load Diff
tests-results/evaluation-2026-08-01T06-02-26.json
+1735
View File
File diff suppressed because it is too large Load Diff