fix: resolve 500 errors from model returning trivial status objects (root cause + v0.2 prompt fix)
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"id": "diag-01",
|
||||
"description": "Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 6,
|
||||
"actualPrimaryType": "unexplained_change",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "complaints",
|
||||
"found": true
|
||||
},
|
||||
{
|
||||
"concept": "warehouse",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "baseline comparison",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "quality issue",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "staff turnover",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "training gap",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-01
|
||||
|
||||
Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We've seen a spike in complaints from our warehouse team this month compared to last month.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: unexplained_change
|
||||
- **Actual Reasoning Modes**: establish_baseline, identify_difference
|
||||
- **Response Duration**: 6ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): warehouse, baseline comparison
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"id": "diag-02",
|
||||
"description": "Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 2,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "app crashes",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "photo upload",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "some customers",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "all users affected",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "server-side bug",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "Android only",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-02
|
||||
|
||||
Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some customers reported that the new app crashes when uploading photos.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: observed_problem
|
||||
- **Actual Reasoning Modes**: identify_difference
|
||||
- **Response Duration**: 2ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): app crashes, photo upload, some customers
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"id": "diag-03",
|
||||
"description": "Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "causal_claim",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": false,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "sales decline",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "price increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "revenue increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "CFO report",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "price was set too high",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "competitors gained market share",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "revenue data is wrong",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-03
|
||||
|
||||
Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: causal_claim
|
||||
- **Actual Reasoning Modes**: establish_baseline, identify_difference
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
classification mismatch
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): sales decline, price increase, revenue increase, CFO report
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"id": "diag-04",
|
||||
"description": "Decision request — forward-looking, needs missing info identification.",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 0,
|
||||
"actualPrimaryType": "decision_request",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"decision_support",
|
||||
"identify_missing_information"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "marketplace app",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "Southeast Asia",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "competitor gap",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "this will definitely succeed",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we have the resources",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "competitors are struggling",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-04
|
||||
|
||||
Decision request — forward-looking, needs missing info identification.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: decision_request
|
||||
- **Actual Reasoning Modes**: identify_difference, decision_support, identify_missing_information
|
||||
- **Response Duration**: 0ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): marketplace app, Southeast Asia, competitor gap
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"id": "diag-05",
|
||||
"description": "Unexpected continuity — changed context but no outcome change.",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "unexplained_change",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"establish_baseline",
|
||||
"validate_measurement"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "supplier change",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "three months ago",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "same defect rate",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "new supplier is worse",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "old supplier was better",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "quality process is broken",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-05
|
||||
|
||||
Unexpected continuity — changed context but no outcome change.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Our production line changed suppliers three months ago but still delivers the same defect rate as before.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: unexplained_change
|
||||
- **Actual Reasoning Modes**: identify_difference, establish_baseline, validate_measurement
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): supplier change, three months ago, same defect rate
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"id": "diag-06",
|
||||
"description": "Quantified improvement — needs context about measurement period and baseline conditions.",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": false,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "completion rate",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "45%",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "62%",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "onboarding",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "all improvements are due to the redesign",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the old flow was bad",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "users prefer the new design",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-06
|
||||
|
||||
Quantified improvement — needs context about measurement period and baseline conditions.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
From 45% to 62%, the completion rate for our onboarding flow improved significantly.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: observed_problem
|
||||
- **Actual Reasoning Modes**: identify_difference
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
classification mismatch
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): completion rate, 45%, 62%, onboarding
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"id": "diag-07",
|
||||
"description": "Single reported claim — needs validation, not acceptance as fact.",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "reported_claim",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"validate_claim"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "pricing complexity",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "small business",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "user claim",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the pricing is actually complex",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "other small businesses agree",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we should simplify pricing",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-07
|
||||
|
||||
Single reported claim — needs validation, not acceptance as fact.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
A user claimed that our pricing model is too complex for small businesses.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: reported_claim
|
||||
- **Actual Reasoning Modes**: identify_difference, validate_claim
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): pricing complexity, small business, user claim
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"id": "diag-08",
|
||||
"description": "Meta-test — self-referential ambiguous statement. Should trigger clarification mode.",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "ambiguous_statement",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"clarify_meaning"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "philosophical",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "ambiguous",
|
||||
"found": true
|
||||
},
|
||||
{
|
||||
"concept": "meaning clarification",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the phrase was wrong",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the colleague is hostile",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we should avoid philosophical language",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-08
|
||||
|
||||
Meta-test — self-referential ambiguous statement. Should trigger clarification mode.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: ambiguous_statement
|
||||
- **Actual Reasoning Modes**: identify_difference, clarify_meaning
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): philosophical, meaning clarification
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"id": "diag-09",
|
||||
"description": "Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "causal_claim",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "deployment",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "complaint volume increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "tripled",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "47 cases",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the deployment caused the complaints",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the bug report was insufficient",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "rollback is needed",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-09
|
||||
|
||||
Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
After the deployment last week, our complaint volume tripled to 47 cases per day.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: causal_claim
|
||||
- **Actual Reasoning Modes**: establish_baseline, identify_difference
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): deployment, complaint volume increase, tripled, 47 cases
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"id": "diag-10",
|
||||
"description": "Paired with diag-01 — distinguishes subset complaints from aggregate claims.",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "production issues",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "delivery speed",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "complaint types",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "production is worse than delivery",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the delivery team needs training",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "both teams are underperforming equally",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Diagnostic Case: diag-10
|
||||
|
||||
Paired with diag-01 — distinguishes subset complaints from aggregate claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some complaints involve production issues, but others say the delivery team is slow.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ✅ PASS (1/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: observed_problem
|
||||
- **Actual Reasoning Modes**: establish_baseline, identify_difference
|
||||
- **Response Duration**: 1ms
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
missing required concept(s): production issues, delivery speed, complaint types
|
||||
@@ -0,0 +1,585 @@
|
||||
{
|
||||
"timestamp": "2026-08-01T06:01:56.070Z",
|
||||
"provider": "mock",
|
||||
"promptVersion": "v0.2",
|
||||
"casesRun": 10,
|
||||
"summary": {
|
||||
"technical": {
|
||||
"schemaValidityRate": "100.0%",
|
||||
"classificationMatchRate": "80.0%",
|
||||
"nextQuestionPresentRate": "100.0%",
|
||||
"passRate": "80.0%"
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConceptMatchRate": "0.0%",
|
||||
"unsupportedInferenceFailures": "0",
|
||||
"passRate": "0.0%"
|
||||
},
|
||||
"combinedPassRate": "0.0%",
|
||||
"averageResponseDurationMs": "2"
|
||||
},
|
||||
"testCaseResults": [
|
||||
{
|
||||
"id": "diag-01",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 6,
|
||||
"actualPrimaryType": "unexplained_change",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "complaints",
|
||||
"found": true
|
||||
},
|
||||
{
|
||||
"concept": "warehouse",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "baseline comparison",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "quality issue",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "staff turnover",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "training gap",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-02",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 2,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "app crashes",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "photo upload",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "some customers",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "all users affected",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "server-side bug",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "Android only",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-03",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "causal_claim",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": false,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "sales decline",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "price increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "revenue increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "CFO report",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "price was set too high",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "competitors gained market share",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "revenue data is wrong",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-04",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 0,
|
||||
"actualPrimaryType": "decision_request",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"decision_support",
|
||||
"identify_missing_information"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "marketplace app",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "Southeast Asia",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "competitor gap",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "this will definitely succeed",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we have the resources",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "competitors are struggling",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-05",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "unexplained_change",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"establish_baseline",
|
||||
"validate_measurement"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "supplier change",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "three months ago",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "same defect rate",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "new supplier is worse",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "old supplier was better",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "quality process is broken",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-06",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": false,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "completion rate",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "45%",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "62%",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "onboarding",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "all improvements are due to the redesign",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the old flow was bad",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "users prefer the new design",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-07",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "reported_claim",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"validate_claim"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "pricing complexity",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "small business",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "user claim",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the pricing is actually complex",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "other small businesses agree",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we should simplify pricing",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-08",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "ambiguous_statement",
|
||||
"actualReasoningModes": [
|
||||
"identify_difference",
|
||||
"clarify_meaning"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "philosophical",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "ambiguous",
|
||||
"found": true
|
||||
},
|
||||
{
|
||||
"concept": "meaning clarification",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the phrase was wrong",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the colleague is hostile",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "we should avoid philosophical language",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-09",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "causal_claim",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "deployment",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "complaint volume increase",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "tripled",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "47 cases",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "the deployment caused the complaints",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the bug report was insufficient",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "rollback is needed",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-10",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 1,
|
||||
"actualPrimaryType": "observed_problem",
|
||||
"actualReasoningModes": [
|
||||
"establish_baseline",
|
||||
"identify_difference"
|
||||
],
|
||||
"technical": {
|
||||
"schemaValid": true,
|
||||
"classificationMatch": true,
|
||||
"reasoningModeMatch": true,
|
||||
"nextQuestionPresent": true,
|
||||
"pass": true,
|
||||
"errors": []
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": false,
|
||||
"details": [
|
||||
{
|
||||
"concept": "production issues",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "delivery speed",
|
||||
"found": false
|
||||
},
|
||||
{
|
||||
"concept": "complaint types",
|
||||
"found": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": [
|
||||
{
|
||||
"concept": "production is worse than delivery",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "the delivery team needs training",
|
||||
"absent": true
|
||||
},
|
||||
{
|
||||
"concept": "both teams are underperforming equally",
|
||||
"absent": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-01",
|
||||
"description": "Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 1785564462515,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-01
|
||||
|
||||
Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We've seen a spike in complaints from our warehouse team this month compared to last month.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462515ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-02",
|
||||
"description": "Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 1785564462519,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-02
|
||||
|
||||
Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some customers reported that the new app crashes when uploading photos.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462519ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-03",
|
||||
"description": "Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 1785564462519,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-03
|
||||
|
||||
Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462519ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-04",
|
||||
"description": "Decision request — forward-looking, needs missing info identification.",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 1785564462520,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-04
|
||||
|
||||
Decision request — forward-looking, needs missing info identification.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462520ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-05",
|
||||
"description": "Unexpected continuity — changed context but no outcome change.",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 1785564462520,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-05
|
||||
|
||||
Unexpected continuity — changed context but no outcome change.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Our production line changed suppliers three months ago but still delivers the same defect rate as before.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462520ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-06",
|
||||
"description": "Quantified improvement — needs context about measurement period and baseline conditions.",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-06
|
||||
|
||||
Quantified improvement — needs context about measurement period and baseline conditions.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
From 45% to 62%, the completion rate for our onboarding flow improved significantly.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462521ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-07",
|
||||
"description": "Single reported claim — needs validation, not acceptance as fact.",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-07
|
||||
|
||||
Single reported claim — needs validation, not acceptance as fact.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
A user claimed that our pricing model is too complex for small businesses.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462521ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-08",
|
||||
"description": "Meta-test — self-referential ambiguous statement. Should trigger clarification mode.",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-08
|
||||
|
||||
Meta-test — self-referential ambiguous statement. Should trigger clarification mode.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462521ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-09",
|
||||
"description": "Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-09
|
||||
|
||||
Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
After the deployment last week, our complaint volume tripled to 47 cases per day.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462521ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-10",
|
||||
"description": "Paired with diag-01 — distinguishes subset complaints from aggregate claims.",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 1785564462522,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-10
|
||||
|
||||
Paired with diag-01 — distinguishes subset complaints from aggregate claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some complaints involve production issues, but others say the delivery team is slow.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 1785564462522ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
{
|
||||
"timestamp": "2026-08-01T06:07:42.532Z",
|
||||
"provider": "ollama-real",
|
||||
"promptVersion": "v0.2",
|
||||
"casesRun": 10,
|
||||
"summary": {
|
||||
"technical": {
|
||||
"schemaValidityRate": "0.0%",
|
||||
"classificationMatchRate": "0.0%",
|
||||
"nextQuestionPresentRate": "0.0%",
|
||||
"passRate": "0.0%"
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConceptMatchRate": "100.0%",
|
||||
"unsupportedInferenceFailures": "0",
|
||||
"passRate": "0.0%"
|
||||
},
|
||||
"combinedPassRate": "0.0%",
|
||||
"averageResponseDurationMs": "1785564462520"
|
||||
},
|
||||
"testCaseResults": [
|
||||
{
|
||||
"id": "diag-01",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 1785564462515,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-02",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 1785564462519,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-03",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 1785564462519,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-04",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 1785564462520,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-05",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 1785564462520,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-06",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-07",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-08",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-09",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 1785564462521,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-10",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 1785564462522,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Invalid server configuration",
|
||||
"Invalid server configuration"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-01",
|
||||
"description": "Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 19459,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"received\",\"message\":\"Please provide a specific request or data to process.\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-01
|
||||
|
||||
Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We've seen a spike in complaints from our warehouse team this month compared to last month.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 19459ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"id": "diag-02",
|
||||
"description": "Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 14693,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Model returned output that could not be parsed as valid JSON.\n\nAPI used: /api/generate\n/api/chat supported: false\nRaw model output:\n{}`\n\nPossible causes:\n- This Ollama version does not support format:json. The model is producing free-form text.\n- Try a larger model (llama3.1, mistral-large) which follows JSON instructions better\n- Shorten your scenario to under 500 words\n- Consider upgrading Ollama: https://ollama.com/download",
|
||||
"Model returned output that could not be parsed as valid JSON.\n\nAPI used: /api/generate\n/api/chat supported: false\nRaw model output:\n{}`\n\nPossible causes:\n- This Ollama version does not support format:json. The model is producing free-form text.\n- Try a larger model (llama3.1, mistral-large) which follows JSON instructions better\n- Shorten your scenario to under 500 words\n- Consider upgrading Ollama: https://ollama.com/download"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-02
|
||||
|
||||
Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some customers reported that the new app crashes when uploading photos.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 14693ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-03",
|
||||
"description": "Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 13516,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"success\",\"message\":\"Input received and processed.\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-03
|
||||
|
||||
Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 13516ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-04",
|
||||
"description": "Decision request — forward-looking, needs missing info identification.",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 48851,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"state\":\"pending\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-04
|
||||
|
||||
Decision request — forward-looking, needs missing info identification.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 48851ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-05",
|
||||
"description": "Unexpected continuity — changed context but no outcome change.",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 16508,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-05
|
||||
|
||||
Unexpected continuity — changed context but no outcome change.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Our production line changed suppliers three months ago but still delivers the same defect rate as before.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 16508ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-06",
|
||||
"description": "Quantified improvement — needs context about measurement period and baseline conditions.",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 15608,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"ok\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-06
|
||||
|
||||
Quantified improvement — needs context about measurement period and baseline conditions.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
From 45% to 62%, the completion rate for our onboarding flow improved significantly.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 15608ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-07",
|
||||
"description": "Single reported claim — needs validation, not acceptance as fact.",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 15750,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"success\",\"message\":\"Input received\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-07
|
||||
|
||||
Single reported claim — needs validation, not acceptance as fact.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
A user claimed that our pricing model is too complex for small businesses.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 15750ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-08",
|
||||
"description": "Meta-test — self-referential ambiguous statement. Should trigger clarification mode.",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 15383,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"success\",\"message\":\"JSON object returned as requested\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-08
|
||||
|
||||
Meta-test — self-referential ambiguous statement. Should trigger clarification mode.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 15383ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-09",
|
||||
"description": "Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 15643,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{\"status\":\"ok\"}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-09
|
||||
|
||||
Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
After the deployment last week, our complaint volume tripled to 47 cases per day.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 15643ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "diag-10",
|
||||
"description": "Paired with diag-01 — distinguishes subset complaints from aggregate claims.",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 15401,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"rawOutput": "{}",
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# Diagnostic Case: diag-10
|
||||
|
||||
Paired with diag-01 — distinguishes subset complaints from aggregate claims.
|
||||
|
||||
## Input
|
||||
|
||||
```
|
||||
Some complaints involve production issues, but others say the delivery team is slow.
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
- **Technical**: ❌ FAIL (0/4 sub-checks pass)
|
||||
- **Reasoning Quality**: ❌ FAIL (0/2 sub-checks pass)
|
||||
- **Actual Primary Type**: N/A
|
||||
- **Actual Reasoning Modes**: N/A
|
||||
- **Response Duration**: 15401ms
|
||||
|
||||
### Technical Failures
|
||||
|
||||
schema invalid, classification mismatch, no next question
|
||||
|
||||
### Reasoning Quality Failures
|
||||
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
{
|
||||
"timestamp": "2026-08-01T06:11:41.538Z",
|
||||
"provider": "ollama-real",
|
||||
"promptVersion": "v0.2",
|
||||
"casesRun": 10,
|
||||
"summary": {
|
||||
"technical": {
|
||||
"schemaValidityRate": "0.0%",
|
||||
"classificationMatchRate": "0.0%",
|
||||
"nextQuestionPresentRate": "0.0%",
|
||||
"passRate": "0.0%"
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConceptMatchRate": "100.0%",
|
||||
"unsupportedInferenceFailures": "0",
|
||||
"passRate": "0.0%"
|
||||
},
|
||||
"combinedPassRate": "0.0%",
|
||||
"averageResponseDurationMs": "19081"
|
||||
},
|
||||
"testCaseResults": [
|
||||
{
|
||||
"id": "diag-01",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"responseDurationMs": 19459,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-02",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"responseDurationMs": 14693,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"Model returned output that could not be parsed as valid JSON.\n\nAPI used: /api/generate\n/api/chat supported: false\nRaw model output:\n{}`\n\nPossible causes:\n- This Ollama version does not support format:json. The model is producing free-form text.\n- Try a larger model (llama3.1, mistral-large) which follows JSON instructions better\n- Shorten your scenario to under 500 words\n- Consider upgrading Ollama: https://ollama.com/download",
|
||||
"Model returned output that could not be parsed as valid JSON.\n\nAPI used: /api/generate\n/api/chat supported: false\nRaw model output:\n{}`\n\nPossible causes:\n- This Ollama version does not support format:json. The model is producing free-form text.\n- Try a larger model (llama3.1, mistral-large) which follows JSON instructions better\n- Shorten your scenario to under 500 words\n- Consider upgrading Ollama: https://ollama.com/download"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-03",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"responseDurationMs": 13516,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-04",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"responseDurationMs": 48851,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-05",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"responseDurationMs": 16508,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-06",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"responseDurationMs": 15608,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-07",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"responseDurationMs": 15750,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-08",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"responseDurationMs": 15383,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-09",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"responseDurationMs": 15643,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "diag-10",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"responseDurationMs": 15401,
|
||||
"actualPrimaryType": null,
|
||||
"actualReasoningModes": [],
|
||||
"technical": {
|
||||
"schemaValid": false,
|
||||
"classificationMatch": false,
|
||||
"reasoningModeMatch": false,
|
||||
"nextQuestionPresent": false,
|
||||
"pass": false,
|
||||
"errors": [
|
||||
"inputClassification: Required",
|
||||
"reconstruction: Required",
|
||||
"evidence: Required",
|
||||
"nextQuestion: Required"
|
||||
]
|
||||
},
|
||||
"reasoningQuality": {
|
||||
"requiredConcepts": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"unsupportedInferencesAbsent": {
|
||||
"pass": true,
|
||||
"details": []
|
||||
},
|
||||
"pass": false
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"latestRun": "2026-08-01T06-11-41",
|
||||
"caseCount": 10
|
||||
}
|
||||
Reference in New Issue
Block a user