Feature/product platform foundation v0.62 #1
@@ -22,7 +22,9 @@
|
||||
|
||||
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
|
||||
|
||||
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
|
||||
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
|
||||
|
||||
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
|
||||
|
||||
@@ -144,8 +146,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54U. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54V. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54U)
|
||||
### Return-to-Work Note (Experiment 54V)
|
||||
|
||||
54T found stable false/null and true/target behaviour across the tested repeats; broader stability beyond those three repetitions remains unproven. 54U tested the next user-facing step: turning a fixed clarification target into one concise neutral question. Results: preference/constraint case asked only about the boundary-vs-tradeoff distinction, affordability definition case stayed within "affordable" meaning, private factual capacity case correctly asked for self-reported availability. No question broadened the target or introduced assumptions. When-to-ask, answer handling, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
|
||||
54U showed a fixed clarification target can become one clean question in three tested cases; the correction clarifies the conclusion covers only those targets, not broader wording quality. 54V tested whether the user's answer resolves only that target without rewriting everything else: hard constraint resolved cleanly (true/null), affordability definition resolved correctly (true/null), and an incomplete answer ("depends on the opportunity") remained unresolved rather than being forced into a stronger category (false/uncertainty). All three cases preserved boundaries. Graph updates, next-question choice, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
|
||||
|
||||
@@ -8432,6 +8432,8 @@ No explanation, score, confidence, answer options, recommendation, preferred int
|
||||
|
||||
All three cases returned one neutral question addressing only the supplied clarification target, with no introduced assumptions, evidence requests, multi-part structure, or implied preferred interpretations.
|
||||
|
||||
**Corrected conclusion:** The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Word-overlap metrics are diagnostic only; manual semantic review is the stronger evidence.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three cases tested; limited domain coverage.
|
||||
@@ -8444,3 +8446,187 @@ All three cases returned one neutral question addressing only the supplied clari
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
|
||||
|
||||
## Experiment 54V — Can a Clarification Answer Resolve Only the Target Without Rewriting Everything Else? (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Given the original source, a fixed clarification target, the clarification question, and the user's answer, can the model identify what has now been resolved without adding new meaning or rewriting unrelated reasoning?
|
||||
|
||||
This is test-only.
|
||||
Do not integrate with the graph, Behaviour Selection, or UI.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A clarification answer should be able to resolve one narrow user-owned ambiguity without causing the model to:
|
||||
- reinterpret the whole source;
|
||||
- add unsupported consequences;
|
||||
- reopen unrelated uncertainty.
|
||||
|
||||
If the model cannot preserve that boundary, clarification answers may create as much ambiguity as they remove.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as 54S / 54T / 54U)
|
||||
Model: `qwen-claude:latest` (same as 54S / 54T / 54U)
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives:
|
||||
```json
|
||||
{ "source": "...", "clarificationTarget": "...", "clarificationQuestion": "...", "userAnswer": "..." }
|
||||
```
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly:
|
||||
```json
|
||||
{ "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
|
||||
```
|
||||
|
||||
No next question, recommendation, confidence score, graph update, extra interpretation, or explanation.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Hard Constraint Resolved
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||||
**User answer:** It's a hard constraint. I don't want any increase in risk.
|
||||
|
||||
**Human reference:**
|
||||
- resolvedMeaning: "avoiding additional risk is a hard constraint"
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
Must not infer: growth is impossible; which growth option should be chosen; how much risk currently exists.
|
||||
|
||||
### Case 2 — Definition Resolved
|
||||
|
||||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||||
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
|
||||
**Clarification question:** Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?
|
||||
**User answer:** I care about the total cost over five years, not the upfront price.
|
||||
|
||||
**Human reference:**
|
||||
- resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost"
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
Do not invent a budget or preferred system.
|
||||
|
||||
### Case 3 — Answer Does Not Fully Resolve Target
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||||
**User answer:** It depends on the opportunity.
|
||||
|
||||
**Human reference:**
|
||||
- The target is **not fully resolved**.
|
||||
- targetResolved: false
|
||||
- remainingUncertainty should preserve that the acceptable trade-off still depends on circumstances.
|
||||
- resolvedMeaning may state only that the user's risk position is conditional on the opportunity.
|
||||
|
||||
Do not force the answer into either preference or hard-constraint category.
|
||||
|
||||
### Results
|
||||
|
||||
#### Case 1 — Hard Constraint Resolved
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is established as a hard constraint, meaning no increase in risk is acceptable."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.**
|
||||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||||
- Avoided wider consequences? **Yes** — did not state growth is impossible, did not choose a growth option, did not estimate current risk levels. The phrase "meaning no increase in risk is acceptable" is a direct restatement of the user answer ("I don't want any increase in risk"), not an addition from the source.
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
#### Case 2 — Definition Resolved
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Affordable refers to a low overall or long-term cost rather than a low upfront price."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.**
|
||||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||||
- Avoided wider consequences? **Yes** — did not invent a budget amount, did not recommend a system.
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
#### Case 3 — Answer Does Not Fully Resolve Target
|
||||
|
||||
**Returned result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is conditional rather than fixed, varying based on the specific opportunity."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "It remains unclear whether avoiding additional risk is strictly a preference/trade-off or a hard constraint, as the stance depends on the opportunity."
|
||||
|
||||
**Manual semantic review:**
|
||||
- Used only the user answer to resolve the target? **Yes.** The model preserved the conditionality present in "It depends on the opportunity" without forcing into either category.
|
||||
- Preserved unresolved ambiguity where present? **Yes.** Correctly kept the target unresolved and described the remaining uncertainty specifically about that target.
|
||||
- Avoided wider consequences? **Yes.**
|
||||
- Avoided adding unsupported facts? **Yes.**
|
||||
- Avoided generating another question? **Yes.**
|
||||
|
||||
**Classification:** resolution_correct
|
||||
|
||||
### Timing
|
||||
|
||||
- Total time: 38,052ms (38.1s)
|
||||
- Average: 12,684.0ms per call
|
||||
- Fastest: 8,479ms (Case 1)
|
||||
- Slowest: 15,208ms (Case 3)
|
||||
|
||||
### Required Questions — Answers
|
||||
|
||||
1. Did Case 1 resolve the target to a hard constraint without adding wider consequences? **Yes.** The resolved meaning stays within the user answer's scope. No inference about growth feasibility, option selection, or current risk levels.
|
||||
2. Did Case 2 resolve "affordable" to long-term cost without inventing a budget? **Yes.** The model correctly captured the five-year perspective without adding any budget figure or system recommendation.
|
||||
3. Did Case 3 correctly keep the target unresolved? **Yes.** The model returned targetResolved=false, preserved conditionality in resolvedMeaning, and provided meaningful remainingUncertainty.
|
||||
4. Did any case force an ambiguous answer into a stronger meaning? **No.** Case 3's conditional answer was kept at its actual strength — neither strengthened to preference nor to hard constraint.
|
||||
5. Did any case rewrite unrelated parts of the source? **No.** In Cases 1 and 3 (same source), the model treated the "grow" portion identically to the original source meaning without reinterpreting it.
|
||||
6. Did any case generate another question? **No.** All resolvedMeaning fields are statements, not questions.
|
||||
7. How many cases were resolution_correct / resolution_failed? **3 correct, 0 failed.**
|
||||
8. Does this establish how graph state should be updated? **No.** This only tests semantic recognition of what a clarification answer resolves; it does not test any graph update mechanism.
|
||||
9. Does this establish what question should come next? **No.** The experiment tested one directional step (answer → resolved meaning) and did not test the next question generation cycle.
|
||||
10. Does this establish how Behaviour Selection should react? **No.** No behaviour selection logic was tested or involved.
|
||||
|
||||
### Evaluation Conclusion
|
||||
|
||||
**Clarification answers resolved only the intended target across all tested cases.**
|
||||
|
||||
All three cases returned correct resolution boundaries: Cases 1 and 2 settled the target cleanly; Case 3 preserved incomplete information at its actual strength without forcing it into a stronger category. The model did not widen beyond the clarification target, invent consequences, or generate new questions in any case.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three cases tested; limited domain coverage (risk constraint, affordability definition, conditional constraint).
|
||||
- Same host/model used throughout — results may vary with different configurations.
|
||||
- Does not establish how graph state should update from resolved meanings.
|
||||
- Does not establish what question should come next after resolution.
|
||||
- Does not establish how Behaviour Selection should react to resolved vs unresolved targets.
|
||||
- Semantic quality assessed through structural checks and manual review; broader generalisation untested.
|
||||
- Case 3's remainingUncertainty output is longer than the human reference — acceptable because it describes the uncertainty rather than adding meaning, but worth noting.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { config } from "dotenv";
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
config({ path: path.resolve(__dirname, "../../.env.local") });
|
||||
|
||||
const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
|
||||
const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
|
||||
|
||||
if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
|
||||
throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
|
||||
}
|
||||
|
||||
/**
|
||||
* Make one live Ollama chat call: take a clarification answer
|
||||
* and return what was resolved, whether the target is resolved,
|
||||
* and any remaining uncertainty about that specific target.
|
||||
*/
|
||||
async function callClarificationAnswerResolution(source, clarificationTarget, clarificationQuestion, userAnswer) {
|
||||
const instruction = `Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
Return valid JSON only in this shape:
|
||||
{
|
||||
"resolvedMeaning": "short statement",
|
||||
"targetResolved": true,
|
||||
"remainingUncertainty": null
|
||||
}`;
|
||||
|
||||
const messages = [
|
||||
{ role: "system", content: instruction.trim() },
|
||||
{
|
||||
role: "user",
|
||||
content: `Source: ${JSON.stringify(source)}
|
||||
|
||||
Clarification target: ${clarificationTarget}
|
||||
|
||||
Clarification question: ${clarificationQuestion}
|
||||
|
||||
User's answer: ${userAnswer}`,
|
||||
},
|
||||
];
|
||||
|
||||
const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: OLLAMA_MODEL,
|
||||
messages,
|
||||
format: "json",
|
||||
stream: false,
|
||||
}),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
|
||||
}
|
||||
|
||||
const data = await res.json();
|
||||
const rawContent = data.message?.content ?? "";
|
||||
const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
|
||||
|
||||
return JSON.parse(cleaned.trim());
|
||||
}
|
||||
|
||||
// Three fixed cases -- human-reviewed inputs and expected outputs
|
||||
const CASES = [
|
||||
{
|
||||
id: "Case 1 - Hard Constraint Resolved",
|
||||
source: "I want the business to grow, but I don't want to take on more risk.",
|
||||
clarificationTarget: "whether avoiding additional risk is a preference/trade-off or a hard constraint",
|
||||
clarificationQuestion: "Do you view avoiding additional risk as a preference/trade-off or a hard constraint?",
|
||||
userAnswer: "It's a hard constraint. I don't want any increase in risk.",
|
||||
humanReference: {
|
||||
resolvedMeaning: "avoiding additional risk is a hard constraint",
|
||||
targetResolved: true,
|
||||
remainingUncertainty: null,
|
||||
},
|
||||
mustNotInfer: [
|
||||
"that growth is impossible",
|
||||
"which growth option should be chosen",
|
||||
"how much risk currently exists",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "Case 2 - Definition Resolved",
|
||||
source: "I want to replace the system, but the new option needs to be affordable.",
|
||||
clarificationTarget: "whether affordable means low upfront cost or low overall/long-term cost",
|
||||
clarificationQuestion: 'Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?',
|
||||
userAnswer: "I care about the total cost over five years, not the upfront price.",
|
||||
humanReference: {
|
||||
resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost",
|
||||
targetResolved: true,
|
||||
remainingUncertainty: null,
|
||||
},
|
||||
mustNotInfer: [
|
||||
"a specific budget amount",
|
||||
"which system to choose",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "Case 3 - Answer Does Not Fully Resolve Target",
|
||||
source: "I want the business to grow, but I don't want to take on more risk.",
|
||||
clarificationTarget: "whether avoiding additional risk is a preference/trade-off or a hard constraint",
|
||||
clarificationQuestion: "Do you view avoiding additional risk as a preference/trade-off or a hard constraint?",
|
||||
userAnswer: "It depends on the opportunity.",
|
||||
humanReference: {
|
||||
resolvedMeaning: "the user's risk position is conditional on the opportunity",
|
||||
targetResolved: false,
|
||||
remainingUncertainty: "acceptable trade-off still depends on circumstances",
|
||||
},
|
||||
mustNotInfer: [
|
||||
"forcing into hard constraint",
|
||||
"forcing into preference/trade-off",
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
function classifyResolution(modelResult, caseRef) {
|
||||
const resolvedMeaning = modelResult.resolvedMeaning;
|
||||
const targetResolved = modelResult.targetResolved;
|
||||
const remainingUncertainty = modelResult.remainingUncertainty;
|
||||
|
||||
// Structural checks
|
||||
if (typeof resolvedMeaning !== "string" || !resolvedMeaning.trim()) {
|
||||
return { classification: "resolution_failed", reason: "missing or empty resolvedMeaning" };
|
||||
}
|
||||
|
||||
if (typeof targetResolved !== "boolean") {
|
||||
return { classification: "resolution_failed", reason: "targetResolved is not a boolean" };
|
||||
}
|
||||
|
||||
// Check targetResolved matches expected type
|
||||
const targetResolvedCorrect =
|
||||
(caseRef.humanReference.targetResolved === true && targetResolved === true) ||
|
||||
(caseRef.humanReference.targetResolved === false && targetResolved === false);
|
||||
|
||||
if (!targetResolvedCorrect) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: `targetResolved is ${targetResolved} but expected ${caseRef.humanReference.targetResolved}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Check for unsupported inferences (must-not-infer patterns)
|
||||
const meaningLower = resolvedMeaning.toLowerCase();
|
||||
let unsupportedAdditions = [];
|
||||
for (const pattern of caseRef.mustNotInfer) {
|
||||
if (meaningLower.includes(pattern.toLowerCase())) {
|
||||
unsupportedAdditions.push(pattern);
|
||||
}
|
||||
}
|
||||
|
||||
if (unsupportedAdditions.length > 0) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: `introduces unsupported meaning: "${unsupportedAdditions.join(", ")}"`,
|
||||
};
|
||||
}
|
||||
|
||||
// Check that resolvedMeaning is concise
|
||||
const wordCount = resolvedMeaning.trim().split(/\s+/).length;
|
||||
if (wordCount > 30) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: `resolvedMeaning too long (${wordCount} words) -- may be generating extra interpretation`,
|
||||
};
|
||||
}
|
||||
|
||||
// Check that model did not generate another question
|
||||
if (resolvedMeaning.includes("?")) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: "resolvedMeaning contains a question mark -- model may have generated a new question",
|
||||
};
|
||||
}
|
||||
|
||||
// Case-specific checks for remainingUncertainty
|
||||
if (caseRef.humanReference.targetResolved === true) {
|
||||
if (remainingUncertainty !== null && remainingUncertainty !== undefined) {
|
||||
return {
|
||||
classification: "resolution_correct",
|
||||
reason: `targetResolved=${targetResolved}, resolvedMeaning="${resolvedMeaning}", but remainingUncertainty was "${remainingUncertainty}" when expected null`,
|
||||
warning: "remainingUncertainty present in a case where it should be null",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (caseRef.humanReference.targetResolved === false) {
|
||||
if (remainingUncertainty === null || remainingUncertainty === undefined) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: "remainingUncertainty is null but target was not fully resolved -- should describe the uncertainty",
|
||||
};
|
||||
}
|
||||
if (typeof remainingUncertainty !== "string" || !remainingUncertainty.trim()) {
|
||||
return {
|
||||
classification: "resolution_failed",
|
||||
reason: "remainingUncertainty present but empty or not a string",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
classification: "resolution_correct",
|
||||
reason: `targetResolved=${targetResolved}, resolvedMeaning="${resolvedMeaning}"`,
|
||||
};
|
||||
}
|
||||
|
||||
function checkOutputSchema(modelResult) {
|
||||
const violations = [];
|
||||
const requiredKeys = ["resolvedMeaning", "targetResolved"];
|
||||
for (const key of requiredKeys) {
|
||||
if (!(key in modelResult)) violations.push(`missing field: ${key}`);
|
||||
}
|
||||
if (typeof modelResult.resolvedMeaning !== "string") violations.push("resolvedMeaning is not a string");
|
||||
if (typeof modelResult.targetResolved !== "boolean") violations.push("targetResolved is not a boolean");
|
||||
if ("remainingUncertainty" in modelResult &&
|
||||
modelResult.remainingUncertainty !== null &&
|
||||
typeof modelResult.remainingUncertainty !== "string") {
|
||||
violations.push("remainingUncertainty must be null or a string");
|
||||
}
|
||||
const forbiddenKeys = ["nextQuestion", "recommendation", "confidenceScore", "graphUpdate"];
|
||||
for (const key of forbiddenKeys) {
|
||||
if (key in modelResult) violations.push(`unexpected field: ${key}`);
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
describe("Experiment 54V - Clarification Answer Resolution", () => {
|
||||
const results = [];
|
||||
const timings = [];
|
||||
|
||||
for (const c of CASES) {
|
||||
it(c.id, async () => {
|
||||
const start = Date.now();
|
||||
const result = await callClarificationAnswerResolution(
|
||||
c.source,
|
||||
c.clarificationTarget,
|
||||
c.clarificationQuestion,
|
||||
c.userAnswer
|
||||
);
|
||||
const elapsed = Date.now() - start;
|
||||
timings.push({ caseId: c.id, ms: elapsed });
|
||||
|
||||
const ev = classifyResolution(result, c);
|
||||
const schemaOk = checkOutputSchema(result);
|
||||
|
||||
results.push({
|
||||
case: c,
|
||||
modelResult: result,
|
||||
classification: ev,
|
||||
schemaViolations: schemaOk,
|
||||
timingMs: elapsed,
|
||||
});
|
||||
|
||||
// Structural assertions
|
||||
expect(result.resolvedMeaning).toBeDefined();
|
||||
expect(typeof result.resolvedMeaning).toBe("string");
|
||||
expect(result.targetResolved).toBeDefined();
|
||||
expect(typeof result.targetResolved).toBe("boolean");
|
||||
|
||||
// Semantic assertions based on case reference
|
||||
if (c.id === "Case 3 - Answer Does Not Fully Resolve Target") {
|
||||
expect(result.targetResolved).toBe(false);
|
||||
} else {
|
||||
expect(result.targetResolved).toBe(true);
|
||||
}
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
it("Experiment 54V: aggregate results", () => {
|
||||
const correct = results.filter((r) => r.classification.classification === "resolution_correct").length;
|
||||
const failed = results.filter((r) => r.classification.classification === "resolution_failed").length;
|
||||
|
||||
console.log("\n=== Experiment 54V Results ===");
|
||||
for (const r of results) {
|
||||
console.log(`\n--- ${r.case.id} ---`);
|
||||
console.log("Source:", r.case.source);
|
||||
console.log("Target:", r.case.clarificationTarget);
|
||||
console.log("Question:", r.case.clarificationQuestion);
|
||||
console.log("Answer:", r.case.userAnswer);
|
||||
console.log("Resolved meaning:", r.modelResult.resolvedMeaning);
|
||||
console.log("targetResolved:", r.modelResult.targetResolved);
|
||||
console.log("remainingUncertainty:", r.modelResult.remainingUncertainty);
|
||||
console.log("Classification:", r.classification.classification, r.classification.reason);
|
||||
if (r.schemaViolations.length > 0) console.log("Schema violations:", r.schemaViolations);
|
||||
}
|
||||
console.log(`\nresolution_correct: ${correct}/${results.length}`);
|
||||
console.log(`resolution_failed: ${failed}/${results.length}`);
|
||||
|
||||
const totalMs = timings.reduce((s, t) => s + t.ms, 0);
|
||||
console.log(`Total time: ${totalMs}ms`);
|
||||
console.log(`Average: ${(totalMs / timings.length).toFixed(1)}ms per call`);
|
||||
console.log(`Fastest: ${Math.min(...timings.map((t) => t.ms))}ms`);
|
||||
console.log(`Slowest: ${Math.max(...timings.map((t) => t.ms))}ms`);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user