fix: resolve 500 errors from model returning trivial status objects (root cause + v0.2 prompt fix)
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
This commit is contained in:
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"id": "diag-01",
|
||||
"input": "We've seen a spike in complaints from our warehouse team this month compared to last month.",
|
||||
"expectedPrimaryTypes": ["unexplained_change"],
|
||||
"expectedReasoningModes": ["establish_baseline", "identify_difference"],
|
||||
"shouldIdentify": ["complaints", "warehouse", "baseline comparison"],
|
||||
"shouldNotInfer": ["quality issue", "staff turnover", "training gap"],
|
||||
"description": "Baseline comparison — change without context. Should NOT jump to conclusions about quality or staff issues."
|
||||
},
|
||||
{
|
||||
"id": "diag-02",
|
||||
"input": "Some customers reported that the new app crashes when uploading photos.",
|
||||
"expectedPrimaryTypes": ["observed_problem"],
|
||||
"expectedReasoningModes": ["identify_difference", "establish_baseline"],
|
||||
"shouldIdentify": ["app crashes", "photo upload", "some customers"],
|
||||
"shouldNotInfer": ["all users affected", "server-side bug", "Android only"],
|
||||
"description": "Subset modifier — 'some customers' means not universal. Should distinguish from blanket claims."
|
||||
},
|
||||
{
|
||||
"id": "diag-03",
|
||||
"input": "Sales fell by 15% last month after we increased prices, but the CFO says revenue is still up 2%.",
|
||||
"expectedPrimaryTypes": ["contradiction"],
|
||||
"expectedReasoningModes": ["investigate_contradiction", "establish_baseline"],
|
||||
"shouldIdentify": ["sales decline", "price increase", "revenue increase", "CFO report"],
|
||||
"shouldNotInfer": ["price was set too high", "competitors gained market share", "revenue data is wrong"],
|
||||
"description": "Apparent contradiction — sales down but revenue up after price change. Distinguishes volume vs value."
|
||||
},
|
||||
{
|
||||
"id": "diag-04",
|
||||
"input": "We need to launch a marketplace app in Southeast Asia to capture the gap our competitors are exploiting.",
|
||||
"expectedPrimaryTypes": ["decision_request"],
|
||||
"expectedReasoningModes": ["decision_support", "identify_missing_information"],
|
||||
"shouldIdentify": ["marketplace app", "Southeast Asia", "competitor gap"],
|
||||
"shouldNotInfer": ["this will definitely succeed", "we have the resources", "competitors are struggling"],
|
||||
"description": "Decision request — forward-looking, needs missing info identification."
|
||||
},
|
||||
{
|
||||
"id": "diag-05",
|
||||
"input": "Our production line changed suppliers three months ago but still delivers the same defect rate as before.",
|
||||
"expectedPrimaryTypes": ["unexplained_change"],
|
||||
"expectedReasoningModes": ["establish_baseline", "identify_difference"],
|
||||
"shouldIdentify": ["supplier change", "three months ago", "same defect rate"],
|
||||
"shouldNotInfer": ["new supplier is worse", "old supplier was better", "quality process is broken"],
|
||||
"description": "Unexpected continuity — changed context but no outcome change."
|
||||
},
|
||||
{
|
||||
"id": "diag-06",
|
||||
"input": "From 45% to 62%, the completion rate for our onboarding flow improved significantly.",
|
||||
"expectedPrimaryTypes": ["unexplained_change"],
|
||||
"expectedReasoningModes": ["establish_baseline", "validate_measurement"],
|
||||
"shouldIdentify": ["completion rate", "45%", "62%", "onboarding"],
|
||||
"shouldNotInfer": ["all improvements are due to the redesign", "the old flow was bad", "users prefer the new design"],
|
||||
"description": "Quantified improvement — needs context about measurement period and baseline conditions."
|
||||
},
|
||||
{
|
||||
"id": "diag-07",
|
||||
"input": "A user claimed that our pricing model is too complex for small businesses.",
|
||||
"expectedPrimaryTypes": ["reported_claim"],
|
||||
"expectedReasoningModes": ["validate_claim", "identify_difference"],
|
||||
"shouldIdentify": ["pricing complexity", "small business", "user claim"],
|
||||
"shouldNotInfer": ["the pricing is actually complex", "other small businesses agree", "we should simplify pricing"],
|
||||
"description": "Single reported claim — needs validation, not acceptance as fact."
|
||||
},
|
||||
{
|
||||
"id": "diag-08",
|
||||
"input": "I used the phrase 'philosophical difference' in a meeting and my colleague said it meant nothing. Is that fair?",
|
||||
"expectedPrimaryTypes": ["ambiguous_statement"],
|
||||
"expectedReasoningModes": ["clarify_meaning"],
|
||||
"shouldIdentify": ["philosophical", "ambiguous", "meaning clarification"],
|
||||
"shouldNotInfer": ["the phrase was wrong", "the colleague is hostile", "we should avoid philosophical language"],
|
||||
"description": "Meta-test — self-referential ambiguous statement. Should trigger clarification mode."
|
||||
},
|
||||
{
|
||||
"id": "diag-09",
|
||||
"input": "After the deployment last week, our complaint volume tripled to 47 cases per day.",
|
||||
"expectedPrimaryTypes": ["causal_claim"],
|
||||
"expectedReasoningModes": ["investigate_contradiction", "establish_baseline"],
|
||||
"shouldIdentify": ["deployment", "complaint volume increase", "tripled", "47 cases"],
|
||||
"shouldNotInfer": ["the deployment caused the complaints", "the bug report was insufficient", "rollback is needed"],
|
||||
"description": "Post-event spike — presents correlation as potential causation. Must resist jumping to causal conclusion."
|
||||
},
|
||||
{
|
||||
"id": "diag-10",
|
||||
"input": "Some complaints involve production issues, but others say the delivery team is slow.",
|
||||
"expectedPrimaryTypes": ["observed_problem"],
|
||||
"expectedReasoningModes": ["identify_difference", "decompose_aggregate"],
|
||||
"shouldIdentify": ["production issues", "delivery speed", "complaint types"],
|
||||
"shouldNotInfer": ["production is worse than delivery", "the delivery team needs training", "both teams are underperforming equally"],
|
||||
"description": "Paired with diag-01 — distinguishes subset complaints from aggregate claims."
|
||||
}
|
||||
]
|
||||
+565
-121
@@ -1,8 +1,23 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { reconstructionSchema } from "@/lib/reconstruction/schema";
|
||||
import { parseReconstruction } from "@/lib/reconstruction/schema";
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
reconstructionSchema,
|
||||
confidenceEnum,
|
||||
importanceEnum,
|
||||
inputTypes,
|
||||
reasoningModes,
|
||||
evidenceRecordSchema,
|
||||
reconstructionV2Schema,
|
||||
analyseResponseSchema,
|
||||
parseReconstruction,
|
||||
parseReconstructionV2,
|
||||
} from "@/lib/reconstruction/schema";
|
||||
import { CONFIDENCE_VALUES } from "@/lib/llm/types.js";
|
||||
|
||||
describe("reconstruction schema", () => {
|
||||
// ──────────────────────────────────────────────
|
||||
// v0.1 — backward compatibility tests
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("v0.1 reconstruction schema", () => {
|
||||
it("validates a complete valid reconstruction", () => {
|
||||
const input = {
|
||||
observations: [{ id: "o1", description: "Saw smoke", confidence: "high" }],
|
||||
@@ -23,34 +38,19 @@ describe("reconstruction schema", () => {
|
||||
it("rejects invalid confidence values", () => {
|
||||
const input = {
|
||||
observations: [{ id: "o1", description: "test", confidence: "extreme" }],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
reportedClaims: [], assumptions: [], entities: [], transitions: [],
|
||||
expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = reconstructionSchema.safeParse(input);
|
||||
expect(result.success).toBe(false);
|
||||
if (!result.success) {
|
||||
expect(result.error.issues[0].message).toContain("Expected");
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects missing required fields", () => {
|
||||
const input = {
|
||||
observations: [{ id: "o1" }],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
reportedClaims: [], assumptions: [], entities: [], transitions: [],
|
||||
expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = reconstructionSchema.safeParse(input);
|
||||
@@ -61,13 +61,7 @@ describe("reconstruction schema", () => {
|
||||
const input = {
|
||||
observations: [],
|
||||
reportedClaims: [{ id: "rc1", description: "test", confidence: "very_high", attributedTo: null }],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = reconstructionSchema.safeParse(input);
|
||||
@@ -76,15 +70,9 @@ describe("reconstruction schema", () => {
|
||||
|
||||
it("rejects empty transitions", () => {
|
||||
const input = {
|
||||
observations: [],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
observations: [], reportedClaims: [], assumptions: [], entities: [],
|
||||
transitions: [{ id: "t1", description: "", confidence: "high", entity: "", previousState: "", currentState: "", explanationStatus: "" }],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = reconstructionSchema.safeParse(input);
|
||||
@@ -95,13 +83,7 @@ describe("reconstruction schema", () => {
|
||||
const input = {
|
||||
observations: [],
|
||||
reportedClaims: [{ id: "rc1", description: "Someone called it in", confidence: "medium", attributedTo: null }],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = reconstructionSchema.safeParse(input);
|
||||
@@ -109,18 +91,255 @@ describe("reconstruction schema", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseReconstruction", () => {
|
||||
// ──────────────────────────────────────────────
|
||||
// v0.2 — schema validation tests
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("v0.2 input classification", () => {
|
||||
it.each([
|
||||
"observed_problem", "unexplained_change", "contradiction", "decision_request",
|
||||
"causal_claim", "reported_claim", "fault_report", "ambiguous_statement",
|
||||
"question", "desired_outcome", "insufficient_context", "other",
|
||||
])("validates input type '%s'", (type) => {
|
||||
const result = inputTypes.safeParse(type);
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects invalid input types", () => {
|
||||
expect(inputTypes.safeParse("invalid_type").success).toBe(false);
|
||||
expect(inputTypes.safeParse("").success).toBe(false);
|
||||
expect(inputTypes.safeParse(null).success).toBe(false);
|
||||
});
|
||||
|
||||
it("validates reasoning modes", () => {
|
||||
const modes = [
|
||||
"establish_baseline", "identify_difference", "reconstruct_transition",
|
||||
"decompose_aggregate", "validate_measurement", "validate_claim",
|
||||
"investigate_contradiction", "clarify_meaning", "decision_support",
|
||||
"fault_investigation", "identify_missing_information", "test_possible_explanations", "other",
|
||||
];
|
||||
for (const m of modes) {
|
||||
const result = reasoningModes.safeParse(m);
|
||||
expect(result.success).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects invalid reasoning mode", () => {
|
||||
expect(reasoningModes.safeParse("no_op").success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.2 multiple secondary types and reasoning modes", () => {
|
||||
it("validates classification with multiple secondary types", () => {
|
||||
const classification = {
|
||||
primaryType: "observed_problem",
|
||||
secondaryTypes: ["fault_report", "decision_request"],
|
||||
reasoningModes: ["validate_claim", "identify_missing_information"],
|
||||
classificationReason: "Test scenario with multiple classifications",
|
||||
confidence: "high",
|
||||
};
|
||||
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: classification,
|
||||
reconstruction: {
|
||||
summary: "test", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [],
|
||||
differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [],
|
||||
importantUnknowns: [], plausibleInterpretations: [],
|
||||
},
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "high", importance: "supporting" }],
|
||||
nextQuestion: {
|
||||
id: "q1", question: "Test?", targets: ["x"], reason: "r",
|
||||
expectedInformationValue: "medium", reasoningMode: "other",
|
||||
},
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("validates with single secondary type", () => {
|
||||
const classification = {
|
||||
primaryType: "unexplained_change",
|
||||
secondaryTypes: ["observed_problem"],
|
||||
reasoningModes: ["establish_baseline"],
|
||||
classificationReason: "Single secondary",
|
||||
confidence: "medium",
|
||||
};
|
||||
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: classification,
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "medium", importance: "supporting" }],
|
||||
nextQuestion: { id: "q1", question: "Test?", targets: ["x"], reason: "r", expectedInformationValue: "low", reasoningMode: "other" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("validates with multiple reasoning modes", () => {
|
||||
const classification = {
|
||||
primaryType: "contradiction",
|
||||
secondaryTypes: [],
|
||||
reasoningModes: ["investigate_contradiction", "identify_difference", "validate_claim"],
|
||||
classificationReason: "Multiple reasoning modes applicable",
|
||||
confidence: "high",
|
||||
};
|
||||
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: classification,
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "high", importance: "important" }],
|
||||
nextQuestion: { id: "q1", question: "Test?", targets: ["x"], reason: "r", expectedInformationValue: "high", reasoningMode: "investigate_contradiction" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.2 evidence records", () => {
|
||||
it.each([
|
||||
"direct_observation", "reported_statement", "interpretation", "assumption", "inferred_relationship",
|
||||
])("validates evidence type '%s'", (eType) => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: eType, confidence: "high", importance: "supporting",
|
||||
});
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects invalid evidence type", () => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: "unknown_type", confidence: "high", importance: "supporting",
|
||||
});
|
||||
expect(result.success).toBe(false);
|
||||
});
|
||||
|
||||
it("allows null attribution", () => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: "reported_statement",
|
||||
attribution: null, confidence: "medium", importance: "incidental",
|
||||
});
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("requires source or attribution optional but not both mandatory", () => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: "direct_observation",
|
||||
confidence: "high", importance: "critical",
|
||||
});
|
||||
expect(result.success).toBe(true); // source and attribution are optional
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.2 invalid confidence and importance values", () => {
|
||||
it.each(["very_high", "extreme", "low_medium", "", "null"])(
|
||||
"invalid confidence '%s' rejected", (val) => {
|
||||
const result = confidenceEnum.safeParse(val);
|
||||
expect(result.success).toBe(false);
|
||||
}
|
||||
);
|
||||
|
||||
it("valid confidence values accepted", () => {
|
||||
for (const v of ["low", "medium", "high"]) {
|
||||
const result = confidenceEnum.safeParse(v);
|
||||
expect(result.success).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["very_high", "extreme", "low_medium", "", "critical_plus"])(
|
||||
"invalid importance '%s' rejected", (val) => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: "direct_observation",
|
||||
confidence: "high", importance: val,
|
||||
});
|
||||
expect(result.success).toBe(false);
|
||||
}
|
||||
);
|
||||
|
||||
it.each(["incidental", "supporting", "important", "critical"])(
|
||||
"valid importance '%s' accepted", (val) => {
|
||||
const result = evidenceRecordSchema.safeParse({
|
||||
id: "e1", description: "test", evidenceType: "direct_observation",
|
||||
confidence: "high", importance: val,
|
||||
});
|
||||
expect(result.success).toBe(true);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
describe("v0.2 plausible interpretations", () => {
|
||||
it("validates reconstruction with multiple plausible interpretations", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: {
|
||||
primaryType: "decision_support", secondaryTypes: [], reasoningModes: [],
|
||||
classificationReason: "Multiple interpretations possible.", confidence: "medium",
|
||||
},
|
||||
reconstruction: {
|
||||
summary: "The situation has two competing explanations.",
|
||||
actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [],
|
||||
knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [],
|
||||
plausibleInterpretations: [
|
||||
{
|
||||
id: "pi1", description: "The issue is caused by configuration drift",
|
||||
supportingEvidenceIds: ["e1", "e3"], assumptionsRequired: ["config_history_is_incomplete"], confidence: "medium",
|
||||
},
|
||||
{
|
||||
id: "pi2", description: "The issue stems from upstream dependency failure",
|
||||
supportingEvidenceIds: ["e2"], assumptionsRequired: ["dependency_outage_at_same_time"], confidence: "low",
|
||||
},
|
||||
],
|
||||
},
|
||||
evidence: [{ id: "e1", description: "Config changed on Tuesday", evidenceType: "direct_observation", confidence: "high", importance: "supporting" }],
|
||||
nextQuestion: { id: "q1", question: "What changed between Monday and Tuesday?", targets: ["timeline"], reason: "To distinguish between drift and dependency failure.", expectedInformationValue: "high", reasoningMode: "reconstruct_transition" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("allows interpretation with empty assumptionsRequired", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "other", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "low" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [{ id: "pi1", description: "Plain interpretation", supportingEvidenceIds: ["e1"], confidence: "low" }] },
|
||||
evidence: [], nextQuestion: { id: "q1", question: "?", targets: [], reason: "r", expectedInformationValue: "low", reasoningMode: "other" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.2 exactly one next question", () => {
|
||||
it("validates when exactly one next question is present", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "observed_problem", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "high" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [], nextQuestion: { id: "q1", question: "What is the baseline?", targets: ["baseline"], reason: "r", expectedInformationValue: "high", reasoningMode: "establish_baseline" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("validates when nextQuestion is absent (schema allows optional)", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "ambiguous_statement", secondaryTypes: [], reasoningModes: [], classificationReason: "No question possible.", confidence: "low" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [], nextQuestion: undefined,
|
||||
});
|
||||
|
||||
// The schema allows missing nextQuestion (optional), so this should pass validation.
|
||||
// We validate exactly-one at the evaluator level, not in the schema.
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects reconstructionV2 when required fields are missing", () => {
|
||||
const result = reconstructionV2Schema.safeParse({});
|
||||
expect(result.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseReconstruction (v0.1)", () => {
|
||||
it("parses a raw JSON string", () => {
|
||||
const raw = JSON.stringify({
|
||||
observations: [{ id: "o1", description: "test", confidence: "high" }],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
reportedClaims: [], assumptions: [], entities: [], transitions: [],
|
||||
expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
});
|
||||
|
||||
const result = parseReconstruction(raw);
|
||||
@@ -134,18 +353,119 @@ describe("parseReconstruction", () => {
|
||||
it("rejects valid JSON that fails schema validation", () => {
|
||||
const raw = JSON.stringify({
|
||||
observations: [{ id: "o1", description: "test", confidence: "extreme" }],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
reportedClaims: [], assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
});
|
||||
|
||||
expect(() => parseReconstruction(raw)).toThrow();
|
||||
});
|
||||
|
||||
it("accepts an already-parsed object", () => {
|
||||
const obj = {
|
||||
observations: [{ id: "o1", description: "test", confidence: "high" }],
|
||||
reportedClaims: [], assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
};
|
||||
|
||||
const result = parseReconstruction(obj);
|
||||
expect(result.observations[0].id).toBe("o1");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseReconstructionV2", () => {
|
||||
it("parses a raw JSON v0.2 string", () => {
|
||||
const raw = JSON.stringify({
|
||||
inputClassification: { primaryType: "observed_problem", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "high" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "high", importance: "supporting" }],
|
||||
nextQuestion: { id: "q1", question: "Test?", targets: ["x"], reason: "r", expectedInformationValue: "medium", reasoningMode: "other" },
|
||||
});
|
||||
|
||||
const result = parseReconstructionV2(raw);
|
||||
expect(result.inputClassification.primaryType).toBe("observed_problem");
|
||||
});
|
||||
|
||||
it("rejects malformed JSON string", () => {
|
||||
expect(() => parseReconstructionV2("{invalid json")).toThrow(SyntaxError);
|
||||
});
|
||||
|
||||
it("rejects valid JSON that fails schema validation", () => {
|
||||
const raw = JSON.stringify({ not: "the right structure" });
|
||||
expect(() => parseReconstructionV2(raw)).toThrow();
|
||||
});
|
||||
|
||||
it("accepts an already-parsed v0.2 object", () => {
|
||||
const obj = {
|
||||
inputClassification: { primaryType: "observed_problem", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "high" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [], nextQuestion: undefined,
|
||||
};
|
||||
|
||||
const result = parseReconstructionV2(obj);
|
||||
expect(result.inputClassification.primaryType).toBe("observed_problem");
|
||||
});
|
||||
});
|
||||
|
||||
describe("malformed model output", () => {
|
||||
it("throws on non-JSON string", () => {
|
||||
expect(() => parseReconstruction("hello world")).toThrow(SyntaxError);
|
||||
});
|
||||
|
||||
it("throws on JSON without required fields", () => {
|
||||
const raw = JSON.stringify({ notTheRightStructure: true });
|
||||
expect(() => parseReconstruction(raw)).toThrow();
|
||||
});
|
||||
|
||||
it("handles empty arrays for all v0.1 categories", () => {
|
||||
const result = parseReconstruction({
|
||||
observations: [], reportedClaims: [], assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
});
|
||||
|
||||
expect(result.observations.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// v0.2 full reconstruction validation
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("v0.2 complete valid reconstruction", () => {
|
||||
it("validates a full v0.2 output with all sections", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "observed_problem", secondaryTypes: ["fault_report"], reasoningModes: ["validate_claim", "identify_difference"], classificationReason: "Clear operational issue identified.", confidence: "high" },
|
||||
reconstruction: {
|
||||
summary: "A fault report with subset scope affecting specific users.",
|
||||
actors: [{ id: "a1", description: "Affected user group", confidence: "medium" }],
|
||||
systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [{ id: "d1", description: "Subset vs universal access", confidence: "high" }],
|
||||
knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [{ id: "u1", description: "Root cause of access failure", confidence: "medium" }],
|
||||
plausibleInterpretations: [],
|
||||
},
|
||||
evidence: [{ id: "e1", description: "User reports confirm the issue.", evidenceType: "reported_statement", source: "support tickets", confidence: "high", importance: "important" }],
|
||||
nextQuestion: { id: "q1", question: "Which specific users are affected?", targets: ["user_segment"], reason: "Narrow scope to identify pattern.", expectedInformationValue: "high", reasoningMode: "validate_claim" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("allows null source in evidence", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "other", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "low" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", attribution: null, confidence: "low", importance: "incidental" }],
|
||||
nextQuestion: undefined,
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("requires all critical importance values for evidence", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "other", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "low" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "high", importance: "critical" }],
|
||||
nextQuestion: { id: "q1", question: "?", targets: ["x"], reason: "r", expectedInformationValue: "low", reasoningMode: "other" },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true); // critical importance is valid
|
||||
});
|
||||
});
|
||||
|
||||
describe("empty scenario rejection", () => {
|
||||
@@ -160,74 +480,198 @@ describe("empty scenario rejection", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Provider parsing tests
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("provider response parsing", () => {
|
||||
it("handles Ollama generate response shape", async () => {
|
||||
vi.stubGlobal("process", { env: { OLLAMA_BASE_URL: "http://localhost:11434" } });
|
||||
|
||||
const mockResponse = JSON.stringify({
|
||||
observations: [{ id: "o1", description: "test", confidence: "high" }],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
});
|
||||
|
||||
global.fetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ response: mockResponse }),
|
||||
});
|
||||
|
||||
const { getProvider } = await import("@/lib/llm/provider");
|
||||
const provider = new getProvider().constructor ? null : getProvider();
|
||||
|
||||
// The provider is instantiated in getProvider
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
|
||||
it("handles raw JSON object response", () => {
|
||||
const parsed = parseReconstruction({
|
||||
observations: [],
|
||||
reportedClaims: [{ id: "rc1", description: "he said", confidence: "medium", attributedTo: "Alice" }],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
observations: [], reportedClaims: [{ id: "rc1", description: "he said", confidence: "medium", attributedTo: "Alice" }], assumptions: [], entities: [], transitions: [], expectedButMissing: [], presentButUnexpected: [], contradictions: [], openUncertainties: [],
|
||||
});
|
||||
|
||||
expect(parsed.reportedClaims[0].attributedTo).toBe("Alice");
|
||||
});
|
||||
});
|
||||
|
||||
describe("malformed model output", () => {
|
||||
it("throws on non-JSON string", () => {
|
||||
expect(() => parseReconstruction("hello world")).toThrow(SyntaxError);
|
||||
});
|
||||
|
||||
it("throws on JSON without required fields", () => {
|
||||
const raw = JSON.stringify({ notTheRightStructure: true });
|
||||
expect(() => parseReconstruction(raw)).toThrow();
|
||||
});
|
||||
|
||||
it("handles empty arrays for all categories", () => {
|
||||
const result = parseReconstruction({
|
||||
observations: [],
|
||||
reportedClaims: [],
|
||||
assumptions: [],
|
||||
entities: [],
|
||||
transitions: [],
|
||||
expectedButMissing: [],
|
||||
presentButUnexpected: [],
|
||||
contradictions: [],
|
||||
openUncertainties: [],
|
||||
it("handles v0.2 parsed reconstruction", () => {
|
||||
const parsed = parseReconstructionV2({
|
||||
inputClassification: { primaryType: "observed_problem", secondaryTypes: [], reasoningModes: [], classificationReason: "test", confidence: "high" },
|
||||
reconstruction: { summary: "x", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [{ id: "d1", description: "delta", confidence: "high" }], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [{ id: "e1", description: "test", evidenceType: "direct_observation", confidence: "high", importance: "supporting" }],
|
||||
nextQuestion: { id: "q1", question: "?", targets: ["x"], reason: "r", expectedInformationValue: "medium", reasoningMode: "other" },
|
||||
});
|
||||
|
||||
expect(result.observations.length).toBe(0);
|
||||
expect(parsed.inputClassification.primaryType).toBe("observed_problem");
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Deterministic evaluator scoring tests
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("deterministic evaluator scoring", () => {
|
||||
function normalise(text) {
|
||||
return String(text).toLowerCase().replace(/[^\w\s_]/g, " ").replace(/\s+/g, " ").trim();
|
||||
}
|
||||
|
||||
function checkPrimaryTypeMatch(actualPrimary, expectedTypes) {
|
||||
if (!actualPrimary || !expectedTypes?.length) return false;
|
||||
const actual = String(actualPrimary).toLowerCase().replace(/\s+/g, "_");
|
||||
return expectedTypes.some((t) => t.toLowerCase().replace(/\s+/g, "_") === actual);
|
||||
}
|
||||
|
||||
function checkReasoningModeMatch(actualModes, expectedModes) {
|
||||
if (!actualModes?.length || !expectedModes?.length) return false;
|
||||
const actual = actualModes.map((m) => String(m).toLowerCase().replace(/\s+/g, "_"));
|
||||
const expected = expectedModes.map((m) => String(m).toLowerCase().replace(/\s+/g, "_"));
|
||||
return expected.some((e) => actual.includes(e));
|
||||
}
|
||||
|
||||
it("matches primary type when exact", () => {
|
||||
expect(checkPrimaryTypeMatch("observed_problem", ["observed_problem"])).toBe(true);
|
||||
});
|
||||
|
||||
it("does not match when primary type differs", () => {
|
||||
expect(checkPrimaryTypeMatch("unexplained_change", ["observed_problem"])).toBe(false);
|
||||
});
|
||||
|
||||
it("matches when primary type is in list of expected types", () => {
|
||||
expect(checkPrimaryTypeMatch("observed_problem", ["observed_problem", "fault_report"])).toBe(true);
|
||||
expect(checkPrimaryTypeMatch("unexplained_change", ["observed_problem", "fault_report"])).toBe(false);
|
||||
});
|
||||
|
||||
it("matches reasoning mode when present in list", () => {
|
||||
expect(checkReasoningModeMatch(["establish_baseline", "identify_difference"], ["establish_baseline"])).toBe(true);
|
||||
});
|
||||
|
||||
it("does not match reasoning mode when absent", () => {
|
||||
expect(checkReasoningModeMatch(["validate_claim"], ["establish_baseline"])).toBe(false);
|
||||
});
|
||||
|
||||
it("handles empty lists gracefully", () => {
|
||||
expect(checkPrimaryTypeMatch(null, [])).toBe(false);
|
||||
expect(checkPrimaryTypeMatch("observed_problem", [])).toBe(false);
|
||||
expect(checkReasoningModeMatch([], ["establish_baseline"])).toBe(false);
|
||||
});
|
||||
|
||||
it("normalises whitespace in comparison", () => {
|
||||
expect(normalise("hello world")).toBe("hello world");
|
||||
expect(normalise("Test_With-Symbols!")).toBe("test_with_symbols");
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Paired test case loading
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("paired test cases", () => {
|
||||
const pairedTests = [
|
||||
{ id: "p1a", input: "All customers cannot download invoices.", expectedPrimaryTypes: ["observed_problem"], notes: "Universal scope" },
|
||||
{ id: "p1b", input: "Some customers cannot download invoices.", expectedPrimaryTypes: ["observed_problem"], notes: "Subset scope — key difference from p1a" },
|
||||
{ id: "p2a", input: "Complaints increased by 35%.", expectedPrimaryTypes: ["unexplained_change"], notes: "Isolated metric change" },
|
||||
{ id: "p2b", input: "Complaints increased by 35% while production increased by 40%.", expectedPrimaryTypes: ["unexplained_change"], notes: "Context changes significance" },
|
||||
{ id: "p3a", input: "Sales are falling.", expectedPrimaryTypes: ["observed_problem"], notes: "Vague claim" },
|
||||
{ id: "p3b", input: "Sales fell sharply immediately after the price increase.", expectedPrimaryTypes: ["causal_claim"], notes: "Adds temporal anchor and cause" },
|
||||
{ id: "p4a", input: "I think therefore I am.", expectedPrimaryTypes: ["ambiguous_statement"], notes: "Philosophical statement" },
|
||||
{ id: "p4b", input: "I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.", expectedPrimaryTypes: ["question"], notes: "Meta-context changes classification" },
|
||||
];
|
||||
|
||||
it.each(pairedTests)("paired test '%s' loads correctly", (tc) => {
|
||||
expect(tc.id).toBeDefined();
|
||||
expect(tc.input.length).toBeGreaterThan(0);
|
||||
expect(Array.isArray(tc.expectedPrimaryTypes)).toBe(true);
|
||||
expect(tc.notes.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("has meaningful differences between paired test A and B inputs", () => {
|
||||
const p1a = pairedTests.find((t) => t.id === "p1a");
|
||||
const p1b = pairedTests.find((t) => t.id === "p1b");
|
||||
expect(p1a.input).toContain("All customers");
|
||||
expect(p1b.input).toContain("Some customers");
|
||||
});
|
||||
|
||||
it("has at least 8 test cases covering different classification types", () => {
|
||||
const coveredTypes = new Set(pairedTests.map((tc) => tc.expectedPrimaryTypes[0]));
|
||||
expect(coveredTypes.size).toBeGreaterThanOrEqual(4); // at least 4 different types
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Confidence and importance value validation
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("confidence and importance enums", () => {
|
||||
it("has exactly three confidence values: low, medium, high", () => {
|
||||
const validConfidences = ["low", "medium", "high"];
|
||||
for (const c of validConfidences) {
|
||||
expect(confidenceEnum.safeParse(c).success).toBe(true);
|
||||
}
|
||||
// CONFIDENCE_VALUES should match
|
||||
expect(CONFIDENCE_VALUES).toEqual(["low", "medium", "high"]);
|
||||
});
|
||||
|
||||
it("has exactly four importance values", () => {
|
||||
const validImportances = ["incidental", "supporting", "important", "critical"];
|
||||
for (const imp of validImportances) {
|
||||
expect(evidenceRecordSchema.safeParse({ id: "x", description: "y", evidenceType: "direct_observation", confidence: "high", importance: imp }).success).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects values outside the defined enums", () => {
|
||||
expect(confidenceEnum.safeParse("very_high").success).toBe(false);
|
||||
expect(evidenceRecordSchema.safeParse({ id: "x", description: "y", evidenceType: "direct_observation", confidence: "high", importance: "critical_plus" }).success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Missing next question test
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("missing next question handling", () => {
|
||||
it("schema allows optional nextQuestion for ambiguous inputs", () => {
|
||||
const result = reconstructionV2Schema.safeParse({
|
||||
inputClassification: { primaryType: "ambiguous_statement", secondaryTypes: [], reasoningModes: [], classificationReason: "Cannot ask meaningful question.", confidence: "low" },
|
||||
reconstruction: { summary: "Ambiguous philosophical statement detected.", actors: [], systemsOrObjects: [], expectedStates: [], observedStates: [], differences: [], knownTransitions: [], unexplainedTransitions: [], contradictions: [], importantUnknowns: [], plausibleInterpretations: [] },
|
||||
evidence: [], nextQuestion: undefined,
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
});
|
||||
|
||||
it("schema rejects missing required fields", () => {
|
||||
const result = reconstructionV2Schema.safeParse({});
|
||||
expect(result.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────
|
||||
// Mock evaluation run test
|
||||
// ──────────────────────────────────────────────
|
||||
|
||||
describe("mock evaluation", () => {
|
||||
function normalise(text) {
|
||||
return String(text).toLowerCase().replace(/[^\w\s_]/g, " ").replace(/\s+/g, " ").trim();
|
||||
}
|
||||
|
||||
it("mock provider can generate deterministic v0.2 output", async () => {
|
||||
// Test that the evaluator's mock provider produces valid schema output
|
||||
const mockInput = "All customers cannot download invoices.";
|
||||
|
||||
// The normaliser should work correctly
|
||||
const normed = normalise(mockInput);
|
||||
expect(normed).toContain("customers");
|
||||
expect(normed).toContain("invoices");
|
||||
});
|
||||
|
||||
it("mock evaluation logic produces expected classification for 'all' vs 'some'", () => {
|
||||
// Verify the evaluator's mock logic handles the key distinction
|
||||
const allInput = "All customers cannot download invoices.";
|
||||
const someInput = "Some customers cannot download invoices.";
|
||||
|
||||
const hasAllWord = /\ball\b|\bno one\b|\bevery\b/i.test(allInput);
|
||||
const hasSomeWord = /some\b/i.test(someInput);
|
||||
|
||||
expect(hasAllWord).toBe(true);
|
||||
expect(hasSomeWord).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
{"id":"tc-001","input":"All customers cannot download their invoices.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["customers","invoices","download","access_issue"],"shouldNotInfer":[],"notes":"Full scope problem — every customer is affected. Should not infer root cause."}
|
||||
{"id":"tc-002","input":"Some customers cannot download their invoices.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["some_customers","invoices","download"],"shouldNotInfer":["root_cause","payment_system_failure"],"notes":"Partial scope — subset of users affected. The word 'some' is the key distinction from tc-001."}
|
||||
{"id":"tc-003","input":"Complaints increased by 35%.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["complaints","increase","35_percent"],"shouldNotInfer":["cause_of_complaints","customer_dissatisfaction_is_worse"],"notes":"Change in isolation — need baseline to understand significance."}
|
||||
{"id":"tc-004","input":"Complaints increased by 35% while production increased by 40%.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["identify_difference","validate_measurement"],"shouldIdentify":["complaints_increase","production_increase","relative_rates"],"shouldNotInfer":["production_quality_declined"],"notes":"Paired with tc-003 — the production context changes meaning significantly."}
|
||||
{"id":"tc-005","input":"Sales are falling.","expectedPrimaryTypes":["observed_problem","unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["sales_decline","direction_negative"],"shouldNotInfer":["cause_of_fall","competitor_action"],"notes":"Vague claim — need baseline, timeline, and definition of 'falling'."}
|
||||
{"id":"tc-006","input":"Sales fell sharply immediately after the price increase.","expectedPrimaryTypes":["observed_problem","causal_claim"],"expectedReasoningModes":["investigate_contradiction","test_possible_explanations"],"shouldIdentify":["sales_decline","price_increase","temporal_correlation"],"shouldNotInfer":["price_increase_caused_the_fall"],"notes":"Paired with tc-005 — adds temporal anchor and proposed cause."}
|
||||
{"id":"tc-007","input":"The quarterly revenue exceeded targets but net profit declined by 12%.","expectedPrimaryTypes":["contradiction","unexplained_change"],"expectedReasoningModes":["investigate_contradiction","identify_missing_information"],"shouldIdentify":["revenue_above_target","profit_decline","divergence"],"shouldNotInfer":["cost_overrun_is_the_cause"],"notes":"Apparent contradiction — revenue up but profit down. Missing cost breakdown."}
|
||||
{"id":"tc-008","input":"Revenue from the premium tier dropped while total revenue grew.","expectedPrimaryTypes":["observed_problem","unexplained_change"],"expectedReasoningModes":["decompose_aggregate","identify_difference"],"shouldIdentify":["premium_tier_decline","total_revenue_growth","segment_cannibalization_risk"],"shouldNotInfer":["pricing_change_occurred"],"notes":"Aggregate masking — total growth hides segment decline."}
|
||||
{"id":"tc-009","input":"We need to improve our customer retention rate.","expectedPrimaryTypes":["decision_request","desired_outcome"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["retention_improvement_desired","current_state_unknown"],"shouldNotInfer":["retention_rate_is_low","churn_has_increased"],"notes":"Desired outcome without stating the problem. Need to know if retention is actually bad."}
|
||||
{"id":"tc-010","input":"The system latency went from 200ms to 5 seconds on Tuesday.","expectedPrimaryTypes":["unexplained_change","observed_problem"],"expectedReasoningModes":["reconstruct_transition","identify_missing_information"],"shouldIdentify":["latency_baseline_200ms","latency_spike_5s","timestamp_tuesday"],"shouldNotInfer":["database_cause","release_cause"],"notes":"Specific measurement with timing anchor. Should identify transition but not infer cause."}
|
||||
{"id":"tc-011","input":"The new release should fix the login issue.","expectedPrimaryTypes":["decision_request","causal_claim"],"expectedReasoningModes":["validate_claim","investigate_contradiction"],"shouldIdentify":["proposed_solution","login_issue","solution_claim"],"shouldNotInfer":["login_issue_is_real","release_will_work"],"notes":"Proposed solution before problem is fully understood. Assumes the issue and fix are connected."}
|
||||
{"id":"tc-012","input":"I think therefore I am.","expectedPrimaryTypes":["ambiguous_statement","question"],"expectedReasoningModes":["clarify_meaning","identify_missing_information"],"shouldIdentify":["philosophical_statement","insufficient_operational_context"],"shouldNotInfer":["business_problem_exists","actionable_insight_possible"],"notes":"Ambiguous philosophical statement. Should not try to find operational meaning."}
|
||||
{"id":"tc-013","input":"I used the phrase 'I think therefore I am' to test whether this system understands ambiguous statements.","expectedPrimaryTypes":["question","ambiguous_statement"],"expectedReasoningModes":["clarify_meaning"],"shouldIdentify":["meta_context","testing_hypothesis","self_reference"],"shouldNotInfer":[],"notes":"Paired with tc-12 — the meta-context changes classification entirely."}
|
||||
{"id":"tc-014","input":"The warehouse manager reported that inventory counts don't match the system.","expectedPrimaryTypes":["reported_claim","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction"],"shouldIdentify":["warehouse_manager_report","inventory_mismatch","system_discrepancy","source_attribution"],"shouldNotInfer":["theft_occurred","software_bug"],"notes":"Reported claim — must distinguish what was said from what it means."}
|
||||
{"id":"tc-015","input":"We've seen a 35% increase in customer complaints.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["establish_baseline","validate_measurement"],"shouldIdentify":["complaints_increase","percentage_metric"],"shouldNotInfer":["product_quality_declined","customer_satisfaction_drop"],"notes":"Needs baseline — is this absolute or relative? Over what period?"}
|
||||
{"id":"tc-016","input":"The number of active users increased by 500%, from 4 to 2,001.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_measurement","decompose_aggregate"],"shouldIdentify":["active_users_metric","absolute_vs_relative_growth","small_base_problem"],"shouldNotInfer":["product_success"],"notes":"Misleading absolute count where rate matters. Small base inflates percentage."}
|
||||
{"id":"tc-017","input":"User engagement metrics improved but the support ticket backlog grew by 200%.","expectedPrimaryTypes":["contradiction"],"expectedReasoningModes":["investigate_contradiction","identify_difference"],"shouldIdentify":["engagement_improvement","support_backlog_growth","divergent_metrics"],"shouldNotInfer":["users_are_angry","product_quality_is_worse"],"notes":"Two metrics telling opposite stories. Could mean engagement is superficial."}
|
||||
{"id":"tc-018","input":"The manufacturing team needs better quality control.","expectedPrimaryTypes":["decision_request","fault_report"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["manufacturing_team","quality_control_desired"],"shouldNotInfer":["quality_is_bad","defect_rate_is_high"],"notes":"Solution proposed without problem specification. What specific quality issue?"}
|
||||
{"id":"tc-019","input":"All users in the EU region are getting a 403 error when trying to access the dashboard.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["eu_region","403_error","access_denied","geographic_scope"],"shouldNotInfer":["gdpr_cause","regulatory_change"],"notes":"Geographic subset fault. Should not infer GDPR as cause without evidence."}
|
||||
{"id":"tc-020","input":"Some users in the EU region are getting a 403 error when trying to access the dashboard.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference","decompose_aggregate"],"shouldIdentify":["eu_region_subset","403_error","partial_reachability"],"shouldNotInfer":["all_eu_users_affected"],"notes":"Paired with tc-19 — 'some' vs 'all' is the material difference."}
|
||||
{"id":"tc-021","input":"Production output was 1,200 units last month and 1,180 units this month.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_measurement","establish_baseline"],"shouldIdentify":["production_output","month_over_month_decline","absolute_difference"],"shouldNotInfer":["efficiency_loss_occurred","equipment_failure"],"notes":"Small absolute change needs context — 1.7% drop might be normal variation."}
|
||||
{"id":"tc-022","input":"The CFO reported that the company's cash position is healthy.","expectedPrimaryTypes":["reported_claim"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["cfo_statement","cash_position_claim","source_attribution_cfo"],"shouldNotInfer":["cash_is_healthy","financial_stability_is_real"],"notes":"Reported opinion — must distinguish what was said from reality."}
|
||||
{"id":"tc-023","input":"We have enough funding to operate for 18 months.","expectedPrimaryTypes":["decision_request","observed_problem"],"expectedReasoningModes":["validate_claim","identify_missing_information"],"shouldIdentify":["funding_period","operational_sustainability","burn_rate_unknown"],"shouldNotInfer":["no_risk_exists"],"notes":"Claim about sustainability without burn rate context."}
|
||||
{"id":"tc-024","input":"The new feature was deployed at 3am and user complaints tripled the next day.","expectedPrimaryTypes":["causal_claim","unexplained_change"],"expectedReasoningModes":["test_possible_explanations","reconstruct_transition"],"shouldIdentify":["feature_deployment","timing_3am","complaint_tripling","temporal_relationship"],"shouldNotInfer":["deployment_caused_complaints"],"notes":"Temporal proximity ≠ causation. Should identify both events but not claim cause."}
|
||||
{"id":"tc-025","input":"We need to launch a mobile app to capture market share.","expectedPrimaryTypes":["decision_request","desired_outcome"],"expectedReasoningModes":["decision_support","identify_missing_information"],"shouldIdentify":["mobile_app_proposed","market_share_desired"],"shouldNotInfer":["no_mobile_app_exists","competitors_have_apps"],"notes":"Desired outcome without problem statement. What evidence supports this decision?"}
|
||||
{"id":"tc-026","input":"The system has been running for 90 days without failure since the migration.","expectedPrimaryTypes":["observed_problem"],"expectedReasoningModes":["validate_claim","establish_baseline"],"shouldIdentify":["uptime_90_days","post_migration_context","baseline_established"],"shouldNotInfer":["system_is_stable_forever"],"notes":"Positive claim about system stability with temporal anchor."}
|
||||
{"id":"tc-027","input":"No one has submitted the required compliance report despite multiple reminders.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_missing_information","investigate_contradiction"],"shouldIdentify":["compliance_report","multiple_reminders","non_submission","absent_action"],"shouldNotInfer":["deliberate_refusal","negligence"],"notes":"Expected-but-missing information. Action was required but absent."}
|
||||
{"id":"tc-028","input":"The audit revealed that 3 of the last 10 monthly reports were submitted with incorrect data.","expectedPrimaryTypes":["observed_problem","contradiction"],"expectedReasoningModes":["validate_measurement","decompose_aggregate"],"shouldIdentify":["audit_findings","incorrect_reports_rate_3_of_10","data_accuracy_issue"],"shouldNotInfer":["intentional_falsification","systemic_failure"],"notes":"Aggregate data — 30% error rate requires context about severity."}
|
||||
{"id":"tc-029","input":"We should implement the new CRM because our competitors have one.","expectedPrimaryTypes":["decision_request","causal_claim"],"expectedReasoningModes":["test_possible_explanations","validate_claim"],"shouldIdentify":["crm_proposal","competitor_comparison","competitive_pressure"],"shouldNotInfer":["crm_will_help","we_lack_crm","competitors_success_is_from_crm"],"notes":"FOMO-driven decision request without problem analysis."}
|
||||
{"id":"tc-030","input":"The server response time was acceptable last quarter but degraded this month.","expectedPrimaryTypes":["unexplained_change","observed_problem"],"expectedReasoningModes":["reconstruct_transition","identify_missing_information"],"shouldIdentify":["response_time_baseline_acceptable","degradation_timeline","quarter_to_month_comparison"],"shouldNotInfer":["load_increase_occurred"],"notes":"Baseline comparison with transition over time. Need specifics."}
|
||||
{"id":"tc-031","input":"The regulatory requirement says all data must be stored within national borders, but our backup server is in another country.","expectedPrimaryTypes":["contradiction","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction","identify_missing_information"],"shouldIdentify":["regulatory_requirement","data_location_violation","cross_border_backup"],"shouldNotInfer":["compliance_failure_is_certain"],"notes":"Regulatory conflict — requires verification of both claim and current state."}
|
||||
{"id":"tc-032","input":"External analysts expect our industry to decline by 15% next year due to regulatory changes.","expectedPrimaryTypes":["causal_claim","reported_claim"],"expectedReasoningModes":["validate_claim","test_possible_explanations"],"shouldIdentify":["industry_decline_prediction","external_source","regulatory_cause","15_percent_forecast"],"shouldNotInfer":["decline_will_occur"],"notes":"External prediction — must treat as claim, not fact."}
|
||||
{"id":"tc-033","input":"The database schema was changed on Friday but the reports are still working.","expectedPrimaryTypes":["unexplained_change"],"expectedReasoningModes":["validate_claim","test_possible_explanations"],"shouldIdentify":["schema_change","reports_working_post_change","unexpected_continuity"],"shouldNotInfer":["change_was_harmless"],"notes":"Expected impact did not occur — should flag as unexplained."}
|
||||
{"id":"tc-034","input":"Some team members say the new process is better while others say it's slower.","expectedPrimaryTypes":["contradiction","observed_problem"],"expectedReasoningModes":["validate_claim","investigate_contradiction","identify_missing_information"],"shouldIdentify":["subjective_split","new_process_evaluation","conflicting_opinions","measurement_gap"],"shouldNotInfer":["process_is_better_or_worse"],"notes":"Conflicting subjective claims — need measurable criteria."}
|
||||
{"id":"tc-035","input":"The application works fine on Chrome but not on Safari.","expectedPrimaryTypes":["observed_problem","fault_report"],"expectedReasoningModes":["validate_claim","identify_difference"],"shouldIdentify":["chrome_compatibility","safari_incompatibility","browser_specific_issue"],"shouldNotInfer":["webkit_bug"],"notes":"Browser-specific fault. Should identify the difference but not the technical cause."}
|
||||
Reference in New Issue
Block a user