diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 7b6e5f8..fdfc148 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -38,7 +38,9 @@ Experiment 37 corrected the routing defect from Experiment 36 and tested a cross Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review. -Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. This confirms broadening did not alter downstream resolution in this scenario; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review. +Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning. The broader target changed the clarification question but not the resolved meaning for the tested explicit answer; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review. + +Experiment 54Z tested whether convergence between precise and broadened targets holds with weaker answers. Source same as 54Y. Two weak answers tested against both fixed variants: (1) "Risk matters more to me" — both variants produced materially equivalent meaning (risk not a hard constraint, but stronger than growth). (2) "I'd normally avoid more risk, but for the right opportunity I might accept some" — variants diverged: Variant A collapsed conditionality into flat preference; Variant B preserved conditional structure and remaining uncertainty. Unexpectedly, the broader target preserved more nuance for the conditional answer. Target broadening has material consequences with weaker answers, but direction is unpredictable. 4 live calls completed. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review. Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed. @@ -150,8 +152,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Y. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 54Y) +### Return-to-Work Note (Experiment 54Z) -Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review. +54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening has material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 02f4a09..71d1887 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -9022,7 +9022,7 @@ The precise target asked whether avoiding extra risk is a trade-off/preference o **`resolutions_materially_equivalent`** -Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. The loss of specificity in the target did not change what was learned from the user's answer. +Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning. ### Questions Answered @@ -9050,9 +9050,9 @@ Despite different questions, both resolved meanings from the same fixed answer e ### Experiment Conclusion -**The broader target produced materially equivalent downstream clarification in the tested scenario.** +**The broader target changed the clarification question but not the resolved meaning for the tested explicit answer.** -The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. The loss of specificity did not change what was learned from the user in this scenario. +The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning. The key finding is: **Does the distinction we lost actually matter? — In this tested scenario, it did not.** Specificity loss is not automatically a failure; it depends on whether it changes downstream meaning. Whether this holds across other scenarios and ambiguity patterns remains untested. @@ -9098,3 +9098,203 @@ This experiment created one new test file only. No specificity-consequence logic ### Return-to-Work Note (Experiment 54Y) Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review. + +## Experiment 54Z — Does Target Broadening Change Resolution When the Answer Is Less Explicit? (2026-08-08) + +### Objective + +Experiment 54Y showed that with a very explicit hard-constraint answer ("It's a hard constraint. I don't want any increase in risk."), both precise and broadened clarification targets converged on materially equivalent resolved meaning — even though the generated questions were materially different. + +This leaves one unresolved consequence: + +> If the user's answer is less explicit, do those two different questions lead to materially different resolved meaning? + +54Z tests that only. Passive and test-only. No redesign of target generation. No integration with Behaviour Selection, graph, or UI. No production code changes. + +### Hypothesis + +The precise and broadened questions may behave differently when the answer does not explicitly name the missing distinction. A weaker answer could: + +- remain correctly unresolved under the precise question; +- but be interpreted as a resolved priority decision under the broader question. + +If that happens, target broadening has a real downstream consequence. If both variants preserve equivalent uncertainty, the broadening may be less consequential than expected. + +### Configuration + +Host: `http://192.168.1.111:11434` (same as 54R–54Y) +Model: `qwen-claude:latest` (same as 54R–54Y) + +### Number of Live Inference Calls + +Exactly **4** live Ollama calls — one answer-resolution per variant × two weaker answers. No question-generation calls (questions are fixed from Experiment 54Y). + +### Context Used + +- `docs/current-handoff.md` +- Experiment 54Y in `docs/design-evolution-log.md` (as basis for the unresolved consequence) +- `tests/reconstruction/semantic-clarification-specificity-consequence.test.js` (structural reference: question generation and answer resolution helpers) +- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract) + +### Fixed Source + +> I want the business to grow, but I don't want to take on more risk. + +### Variant A — Precise Target + +**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint. +**Fixed question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off? + +### Variant B — Broadened Target + +**Clarification target:** priority between business growth and risk avoidance when they conflict. +**Fixed question:** When business growth and risk avoidance conflict, which do you prioritize? + +--- + +### Answer 1 — Priority Without Constraint Meaning + +**User answer:** "Risk matters more to me." + +**Expected behavior (human-reviewed):** +This answer does not clearly establish whether risk avoidance is a hard constraint or merely a stronger preference. For Variant A, the precise target should therefore remain unresolved. For Variant B, the answer may legitimately resolve the priority target as "risk avoidance has higher priority than growth." + +### Answer 1 Results + +| Variant | resolvedMeaning | targetResolved | remainingUncertainty | +|---|---|---|---| +| A (Precise) | The user treats avoiding additional risk as a strong priority or preference rather than an absolute, non-negotiable constraint. | true | null | +| B (Broadened) | Risk avoidance is prioritized over business growth when they conflict. | true | null | + +### Answer 1 Analysis + +Variant A interprets "Risk matters more to me" as meaning risk avoidance is a **strong preference/priority rather than an absolute constraint** — this maps correctly onto the precise target (preference/trade-off vs hard constraint). The model marked targetResolved=true because it interpreted the answer as settling the distinction toward "preference/trade-off." + +Variant B interprets the same answer as meaning **risk avoidance has higher priority over growth when they conflict** — this maps correctly onto the broadened target (priority ordering). + +**Classification: resolutions_materially_equivalent** + +Both variants map the weak answer to a preference/priority-over-constraint interpretation. Neither resolves to "hard constraint." The resolved meanings use different framing but preserve the same downstream reasoning state: risk is not an absolute boundary, it is a prioritized consideration. For downstream use (what can/cannot be done), both produce equivalent uncertainty about whether risk could ever be accepted. + +Both variants were flagged as `potential_erasal_of_uncertainty` because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases. + +--- + +### Answer 2 — Conditional Trade-Off + +**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some." + +**Expected behavior (human-reviewed):** +This answer indicates risk avoidance is not an absolute hard constraint; some trade-off may be acceptable depending on the opportunity. For Variant A, this should resolve away from "hard constraint" while retaining conditionality. For Variant B, it may establish that risk is normally prioritized but can yield to growth in some cases. + +### Answer 2 Results + +| Variant | resolvedMeaning | targetResolved | remainingUncertainty | +|---|---|---|---| +| A (Precise) | Avoiding additional risk is a preference or trade-off rather than a hard constraint. | true | null | +| B (Broadened) | Default priority is risk avoidance, with a conditional willingness to accept some risk for highly suitable opportunities. | false | It remains unclear how "the right opportunity" is defined and which factor strictly takes precedence when a specific growth opportunity carries significant risk. | + +### Answer 2 Analysis + +**This is the critical divergence.** Variant A collapses the conditional nature of the answer into a simple preference-vs-constraint resolution. The model says "preference or trade-off rather than a hard constraint" — but loses the key information that there are conditions (the right opportunity) under which even this preference could shift. This was flagged as `forced_certainty_detected` and `potential_erasal_of_uncertainty`. + +Variant B preserves the conditionality ("conditional willingness") and correctly marks targetResolved=false because the answer does not establish a stable priority — the priority shifts depending on context. It also identifies remaining uncertainty about what constitutes "the right opportunity." + +**Classification: resolutions_materially_different** + +This is a material divergence. Variant A erases the conditional nature of the user's stated position and produces a flat preference-versus-constraint resolution. Variant B preserves both the default-priority-and-conditional structure AND the remaining uncertainty about when conditions change. For downstream reasoning, this matters because: +- Under Variant A's meaning: risk avoidance = preference/trade-off → may be willing to accept risk in some cases (inferred) +- Under Variant B's meaning: default priority risk, conditionally willing → conditional willingness is preserved explicitly + +**However**, the divergence exists primarily in remainingUncertainty content, not in the resolvedMeaning itself. Both agree that risk avoidance is not a hard constraint. The difference is in whether the model preserves "there are conditions we don't yet understand" versus collapsing everything to "not a hard constraint." + +--- + +### Questions Answered + +1. **For Answer 1, did Variant A correctly preserve uncertainty about preference versus hard constraint?** Partially. Variant A mapped the answer toward "preference/trade-off rather than absolute constraint" but marked it as fully resolved (targetResolved=true) with no remainingUncertainty — erasing the ambiguity that "matters more" doesn't define a boundary. + +2. **For Answer 1, did Variant B resolve the broader priority target?** Yes. The broadened target ("priority between growth and risk") was resolved as "risk avoidance is prioritized over business growth when they conflict." This is a correct mapping from the weak answer to the priority frame. + +3. **Did Answer 1 therefore create materially different resolution states between A and B?** No. Both map to the same downstream state: risk avoidance is not an absolute boundary but a stronger consideration than growth. The resolutions are materially equivalent for downstream reasoning about what can/cannot be done. + +4. **For Answer 2, did Variant A correctly identify that risk avoidance is not an absolute hard constraint?** Partially correct on the outcome (not a hard constraint) but failed to preserve conditionality — the "might accept some" conditional was collapsed into a flat preference resolution with no remaining uncertainty. + +5. **For Answer 2, did Variant B preserve the conditional nature of the priority?** Yes. Variant B preserved both the default-priority-and-conditional structure and identified remaining uncertainty about when conditions shift. + +6. **Were the Answer 2 resolution states materially equivalent or different?** Different. Variant A erased conditionality; Variant B preserved it plus remainingUncertainty. This is a material divergence for downstream reasoning state. + +7. **Did either variant force a weak answer into stronger meaning than the user supplied?** Yes — Variant A for Answer 2 collapsed conditional willingness ("might accept some") into a flat preference classification, erasing the conditionality layer. + +8. **Did either variant erase uncertainty that should remain?** Yes — Variant A for both answers showed `potential_erasal_of_uncertainty`. For Answer 1, "Risk matters more to me" became a fully resolved preference (no remainingUncertainty). For Answer 2, conditionality was erased. + +9. **Does target broadening have a material downstream consequence when answers are less explicit in these tested cases?** Yes — specifically for Answer 2 (conditional trade-off). The precise target question led the model to map to a flat preference-vs-constraint resolution and erase conditionality. The broadened target preserved conditional structure. This means target broadening has a real, asymmetrical consequence: the broadened question can actually preserve nuance that the precise question erases in this case. + +10. **Does this establish that precise targets are always required?** No — Answer 1 showed no material divergence, and for Answer 2 the broader target preserved more nuance than the precise one. Neither is universally better. + +11. **Does this establish how target-generation logic should be changed?** No — only two answers tested; neither variant was consistently better; no fix designed from these results. + +12. **Does this establish UI behaviour?** No — this is a clarification-target test only. + +### Limitations + +- Only one source scenario and two user answers were tested. Different sources may behave differently. +- Only one ambiguity pattern (preference/constraint) was tested with weak answers. Other patterns not assessed. +- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently. +- The asymmetric finding (broadened target preserving more nuance for Answer 2) is surprising and warrants further testing with additional answers that include explicit conditionality. +- Two tested cases; broader generalisation remains untested. + +### Experiment Conclusion + +**Target broadening changed wording but not material resolution under weaker answers — except when the answer contained explicit conditionality.** + +For Answer 1 ("Risk matters more to me."), both variants produced materially equivalent downstream meaning: risk avoidance is stronger than growth but not an absolute constraint. The broader target did not create a material divergence here. + +For Answer 2 ("I'd normally avoid more risk, but for the right opportunity I might accept some."), the variants diverged. Variant A (precise) collapsed conditionality into a flat preference resolution and erased uncertainty. Variant B (broadened) preserved conditional structure and remaining uncertainty about what constitutes "the right opportunity." + +**Unexpected finding:** The broadened target preserved more nuance than the precise target for the conditional answer. This contradicts the intuition that precision always preserves information — sometimes broader framing lets the model retain conditional language it would otherwise collapse under a narrower frame. + +The key finding: target broadening's consequence depends on the answer's content. For pure priority statements, broadening is inconsequential. For conditional trade-off statements, broadening can preserve (or erase) meaning differently than precise targeting — sometimes better, potentially worse in other cases. + +### Focused Test Result + +All 4 live inference calls completed successfully (all tests passed). Answer 1: materially equivalent resolutions from both variants. Answer 2: materially different resolutions — Variant A erased conditionality; Variant B preserved it. One forced certainty detection (Variant A on Answer 2). + +### Historical Comparison Result + +Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. The explicitness of the user's answer matters more than the question framing for determining downstream meaning equivalence. This confirms the hypothesis that weak answers expose differences between precise and broadened targets — but the direction of difference is not predictable in advance (sometimes broader is better, sometimes it may be worse). + +### Documentation Updated + +- `docs/design-evolution-log.md` — added full Experiment 54Z entry; applied 54Y wording corrections +- `docs/current-handoff.md` — updated with Experiment 54Z summary and new Return-to-Work note + +### Confirmation Host and Model Remained Unchanged + +Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54Y. + +### Confirmation Semantic Instructions and Output Contracts Remained Unchanged + +Answer-resolution instruction identical to Experiment 54V. Output contract unchanged from 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`). + +### Confirmation Production Prompts and Schemas Remained Unchanged + +No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file. + +### Confirmation Behaviour Selection Remained Unchanged + +Behaviour Selection was not called or referenced. No integration with the selector occurred. + +### Confirmation Graph and UI Remained Unchanged + +No graph files read or modified. No UI code touched. The experiment is test-only. + +### Confirmation No Weak-Answer Consequence Logic Entered Active Runtime + +This experiment created one new test file only. No weak-answer consequence logic entered any active runtime path, production module, or behaviour selection output. + +--- + +### Return-to-Work Note (Experiment 54Z) + +54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer, the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening does have material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review. diff --git a/tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js b/tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js new file mode 100644 index 0000000..b70ab25 --- /dev/null +++ b/tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js @@ -0,0 +1,392 @@ +import { describe, it, expect } from "vitest"; +import { config } from "dotenv"; +import path from "path"; +import { fileURLToPath } from "url"; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); +config({ path: path.resolve(__dirname, "../../.env.local") }); + +const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL; +const OLLAMA_MODEL = process.env.OLLAMA_MODEL; + +if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) { + throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local"); +} + +/** + * Make one live Ollama chat call: take a clarification answer + * and return what was resolved, whether the target is resolved, + * and any remaining uncertainty about that specific target. + */ +async function callClarificationAnswerResolution(source, clarificationTarget, clarificationQuestion, userAnswer) { + const instruction = `Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question. + +Return valid JSON only in this shape: +{ + "resolvedMeaning": "short statement", + "targetResolved": true, + "remainingUncertainty": null +}`; + + const messages = [ + { role: "system", content: instruction.trim() }, + { + role: "user", + content: `Source: ${JSON.stringify(source)} + +Clarification target: ${clarificationTarget} + +Clarification question: ${clarificationQuestion} + +User's answer: ${userAnswer}`, + }, + ]; + + const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + model: OLLAMA_MODEL, + messages, + format: "json", + stream: false, + }), + }); + + if (!res.ok) { + throw new Error(`Ollama API error: ${res.status} ${res.statusText}`); + } + + const data = await res.json(); + const rawContent = data.message?.content ?? ""; + const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, ""); + + return JSON.parse(cleaned.trim()); +} + +// ────────────────────────────────────────────── +// Fixed source (Experiment 54Z) +// ────────────────────────────────────────────── + +const SOURCE = "I want the business to grow, but I don't want to take on more risk."; + +// Variant A - Precise target and fixed question (from Experiment 54Y) +const VARIANT_A_TARGET = "whether avoiding additional risk is a preference/trade-off or a hard constraint"; +const VARIANT_A_QUESTION = "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?"; + +// Variant B - Broadened target and fixed question (from Experiment 54Y) +const VARIANT_B_TARGET = "priority between business growth and risk avoidance when they conflict"; +const VARIANT_B_QUESTION = "When business growth and risk avoidance conflict, which do you prioritize?"; + +// Answer 1 - Priority Without Constraint Meaning +const ANSWER_1 = "Risk matters more to me."; + +// Answer 2 - Conditional Trade-Off +const ANSWER_2 = "I'd normally avoid more risk, but for the right opportunity I might accept some."; + +// ────────────────────────────────────────────── +// Semantic evaluation helpers +// ────────────────────────────────────────────── + +/** + * Classify whether two resolution results are materially equivalent + * or materially different for downstream reasoning. + */ +function classifyResolutionEquivalence(resA, resB) { + const a = resA.resolvedMeaning.toLowerCase(); + const b = resB.resolvedMeaning.toLowerCase(); + + // Check hard constraint semantics in each + const hasHardConstraint = (r) => + /hard.*(constraint|boundary|limit)|no.*(increase|more|additional|take on).*risk|absolut.*no.*risk|must.*(not|avoid)/i.test(r); + + // Check conditional/trade-off semantics + const hasConditionality = (r) => + /\bconditionally\b|depends.*on|might?.*accept|under.*condition|depending|unless|except/i.test(r) || + /normally.*avoid.*but|for.*(the |some )?right.*(opportunity|case|situation)/i.test(r); + + // Check "matters more" priority semantics (no hard constraint implied) + const hasPriorityMeaning = (r) => + /\bmatters.?more\b|higher.?priority|priorit.*risk|risk.*takes\s*(precedence|priority)|more\s*important\s*than/i.test(r); + + const aHasHard = hasHardConstraint(a); + const bHasHard = hasHardConstraint(b); + const aHasCond = hasConditionality(a); + const bHasCond = hasConditionality(b); + const aHasPriority = hasPriorityMeaning(a); + const bHasPriority = hasPriorityMeaning(b); + + // If one is clearly hard constraint and the other is not - materially different + if (aHasHard !== bHasHard) { + return "resolutions_materially_different"; + } + + // Check conditionality divergence + if ((aHasCond && !bHasCond) || (!aHasCond && bHasCond)) { + return "resolutions_materially_different"; + } + + // If both resolve differently on targetResolution + const aResolved = resA.targetResolved === true; + const bResolved = resB.targetResolved === true; + if (aResolved !== bResolved) { + return "resolutions_materially_different"; + } + + // Check remaining uncertainty state divergence + const aHasUncertainty = resA.remainingUncertainty !== null && resA.remainingUncertainty !== undefined; + const bHasUncertainty = resB.remainingUncertainty !== null && resB.remainingUncertainty !== undefined; + + if (aHasUncertainty !== bHasUncertainty) { + return "resolutions_materially_different"; + } + + // Both have constraint semantics + if (aHasHard && bHasHard) { + return "resolutions_materially_equivalent"; + } + + // Neither has hard constraint - compare priority framing consistency + if ((aHasUncertainty && bHasUncertainty) || (!aHasUncertainty && !bHasUncertainty)) { + const bothAboutRiskBoundary = /risk/i.test(a) && /risk/i.test(b); + if (bothAboutRiskBoundary && aHasPriority === bHasPriority) { + return "resolutions_materially_equivalent"; + } + } + + // Default to manual review needed + return "manual_review_required"; +} + +/** + * Check whether the model forced stronger meaning than the answer supplies. + */ +function detectsForcedCertainty(res, userAnswer) { + const m = res.resolvedMeaning.toLowerCase(); + const a = userAnswer.toLowerCase(); + + // If the answer is vague but the resolution claims absolute/hard constraint + if (!a.includes("constraint") && !a.includes("absolute") && !a.includes("must not")) { + if (/must.*not.*take|no.*risk|hard.*constraint|absolute.*limit|will.*never/i.test(m)) { + return "forced_certainty_detected"; + } + } + + // If the answer is conditional but the resolution claims unconditional + if (a.includes("might") || a.includes("but") || a.includes("unless") || a.includes("conditionally")) { + if (/will.*never|must.*not|absolute|never.*accept/i.test(m) && !/normally.*but|conditional|depends/i.test(m)) { + return "forced_certainty_detected"; + } + } + + return null; +} + +/** + * Check whether the model erased uncertainty that should remain. + */ +function detectsErasedUncertainty(res, userAnswer) { + const hasRemaining = res.remainingUncertainty !== null && res.remainingUncertainty !== undefined; + + // For Answer 1 (weak): if targetResolved is true and no remaining uncertainty, + // the model may have erased uncertainty that should remain + const answerIsWeak = !userAnswer.includes("constraint") && !userAnswer.includes("absolute"); + if (answerIsWeak && res.targetResolved === true && !hasRemaining) { + return "potential_erasal_of_uncertainty"; + } + + return null; +} + +// ────────────────────────────────────────────── +// Test suite - Experiment 54Z +// ────────────────────────────────────────────── + +describe("Experiment 54Z - Clarification Broadening with Weak Answers", () => { + const results = { + answer1: { variantA: null, variantB: null }, + answer2: { variantA: null, variantB: null }, + timings: [], + }; + + // -- Answer 1, Variant A -- + + it("Answer 1 -> Variant A (precise target)", async () => { + const start = Date.now(); + const result = await callClarificationAnswerResolution( + SOURCE, VARIANT_A_TARGET, VARIANT_A_QUESTION, ANSWER_1 + ); + const elapsed = Date.now() - start; + results.timings.push({ stage: "A1-V-A", ms: elapsed }); + results.answer1.variantA = result; + }, 120000); + + // -- Answer 1, Variant B -- + + it("Answer 1 -> Variant B (broadened target)", async () => { + const start = Date.now(); + const result = await callClarificationAnswerResolution( + SOURCE, VARIANT_B_TARGET, VARIANT_B_QUESTION, ANSWER_1 + ); + const elapsed = Date.now() - start; + results.timings.push({ stage: "A1-V-B", ms: elapsed }); + results.answer1.variantB = result; + }, 120000); + + // -- Answer 2, Variant A -- + + it("Answer 2 -> Variant A (precise target)", async () => { + const start = Date.now(); + const result = await callClarificationAnswerResolution( + SOURCE, VARIANT_A_TARGET, VARIANT_A_QUESTION, ANSWER_2 + ); + const elapsed = Date.now() - start; + results.timings.push({ stage: "A2-V-A", ms: elapsed }); + results.answer2.variantA = result; + }, 120000); + + // -- Answer 2, Variant B -- + + it("Answer 2 -> Variant B (broadened target)", async () => { + const start = Date.now(); + const result = await callClarificationAnswerResolution( + SOURCE, VARIANT_B_TARGET, VARIANT_B_QUESTION, ANSWER_2 + ); + const elapsed = Date.now() - start; + results.timings.push({ stage: "A2-V-B", ms: elapsed }); + results.answer2.variantB = result; + }, 120000); + + // -- Aggregate Evaluation -- + + it("54Z: evaluate Answer 1 pairwise comparison", () => { + const resA = results.answer1.variantA; + const resB = results.answer1.variantB; + + console.log("\n========== Experiment 54Z Results =========="); + console.log("\n--- Source ---"); + console.log(SOURCE); + console.log("\n--- Answer 1: " + ANSWER_1 + " ---"); + console.log("\nVariant A (precise):"); + console.log(" resolvedMeaning:", resA.resolvedMeaning); + console.log(" targetResolved:", resA.targetResolved); + console.log(" remainingUncertainty:", resA.remainingUncertainty ?? null); + + console.log("\nVariant B (broadened):"); + console.log(" resolvedMeaning:", resB.resolvedMeaning); + console.log(" targetResolved:", resB.targetResolved); + console.log(" remainingUncertainty:", resB.remainingUncertainty ?? null); + + const equiv = classifyResolutionEquivalence(resA, resB); + results.answer1.classification = equiv; + + console.log("\nAnswer 1 comparison: " + equiv); + + // Forced certainty check + const forcedA1 = detectsForcedCertainty(resA, ANSWER_1); + const forcedB1 = detectsForcedCertainty(resB, ANSWER_1); + if (forcedA1) console.log("Answer 1 Variant A: " + forcedA1); + if (forcedB1) console.log("Answer 1 Variant B: " + forcedB1); + + // Erased uncertainty check + const erasedA1 = detectsErasedUncertainty(resA, ANSWER_1); + const erasedB1 = detectsErasedUncertainty(resB, ANSWER_1); + if (erasedA1) console.log("Answer 1 Variant A: " + erasedA1); + if (erasedB1) console.log("Answer 1 Variant B: " + erasedB1); + + // Answer 1 human review questions + console.log("\n--- Manual Review Questions ---"); + console.log("Q1: Did Variant A preserve uncertainty about preference vs hard constraint?", + !resA.targetResolved || resA.remainingUncertainty !== null ? "Yes" : "No - need review"); + console.log("Q2: Did Variant B resolve the broader priority target?", + resB.targetResolved ? "Yes" : "No"); + console.log("Q3: Materially different resolution states for Answer 1?", + equiv === "resolutions_materially_different" || equiv === "manual_review_required" ? "Possibly - manual review needed" : "No"); + + results.answer1 = { ...results.answer1, forcedA1, forcedB1, erasedA1, erasedB1 }; + }); + + it("54Z: evaluate Answer 2 pairwise comparison", () => { + const resA = results.answer2.variantA; + const resB = results.answer2.variantB; + + console.log("\n--- Answer 2: " + ANSWER_2 + " ---"); + console.log("\nVariant A (precise):"); + console.log(" resolvedMeaning:", resA.resolvedMeaning); + console.log(" targetResolved:", resA.targetResolved); + console.log(" remainingUncertainty:", resA.remainingUncertainty ?? null); + + console.log("\nVariant B (broadened):"); + console.log(" resolvedMeaning:", resB.resolvedMeaning); + console.log(" targetResolved:", resB.targetResolved); + console.log(" remainingUncertainty:", resB.remainingUncertainty ?? null); + + const equiv = classifyResolutionEquivalence(resA, resB); + results.answer2.classification = equiv; + + console.log("\nAnswer 2 comparison: " + equiv); + + // Forced certainty check + const forcedA2 = detectsForcedCertainty(resA, ANSWER_2); + const forcedB2 = detectsForcedCertainty(resB, ANSWER_2); + if (forcedA2) console.log("Answer 2 Variant A: " + forcedA2); + if (forcedB2) console.log("Answer 2 Variant B: " + forcedB2); + + // Erased uncertainty check + const erasedA2 = detectsErasedUncertainty(resA, ANSWER_2); + const erasedB2 = detectsErasedUncertainty(resB, ANSWER_2); + if (erasedA2) console.log("Answer 2 Variant A: " + erasedA2); + if (erasedB2) console.log("Answer 2 Variant B: " + erasedB2); + + // Answer 2 human review questions + console.log("\n--- Manual Review Questions ---"); + console.log("Q4: Did Variant A correctly identify risk avoidance is not absolute hard constraint?", + !resA.targetResolved || (resA.targetResolved && resA.remainingUncertainty !== null) ? "Yes/Partially" : "No - need review"); + console.log("Q5: Did Variant B preserve conditionality?", + /normally.*but|conditional|depends|might/i.test(resB.resolvedMeaning) ? "Yes" : "Need review"); + console.log("Q6: Materially equivalent resolution states for Answer 2?", + equiv === "resolutions_materially_equivalent" ? "Yes" : + equiv === "resolutions_materially_different" ? "No - they differ" : + "Manual review needed"); + + results.answer2 = { ...results.answer2, forcedA2, forcedB2, erasedA2, erasedB2 }; + }); + + it("54Z: aggregate timing and summary", () => { + const totalMs = results.timings.reduce((s, t) => s + t.ms, 0); + const msArr = results.timings.map((t) => t.ms); + + console.log("\n--- Timing ---"); + console.log("Calls:", results.timings.length); + console.log("Total:", totalMs + "ms"); + console.log("Average:", (totalMs / results.timings.length).toFixed(1) + "ms per call"); + console.log("Fastest:", Math.min(...msArr) + "ms"); + console.log("Slowest:", Math.max(...msArr) + "ms"); + + // Summary + const a1Diff = results.answer1.classification === "resolutions_materially_different"; + const a2Diff = results.answer2.classification === "resolutions_materially_different"; + const anyForced = [results.answer1.forcedA1, results.answer1.forcedB1, results.answer2.forcedA2, results.answer2.forcedB2].some(Boolean); + const anyErased = [results.answer1.erasedA1, results.answer1.erasedB1, results.answer2.erasedA2, results.answer2.erasedB2].some(Boolean); + + console.log("\n--- Summary ---"); + console.log("Answer 1 materially different:", a1Diff); + console.log("Answer 2 materially different:", a2Diff); + console.log("Any forced certainty detected:", anyForced); + console.log("Any uncertainty erased:", anyErased); + + results.summary = { + totalTimeMs: totalMs, + averageMs: (totalMs / results.timings.length).toFixed(1), + fastestMs: Math.min(...msArr), + slowestMs: Math.max(...msArr), + answer1MateriallyDifferent: a1Diff, + answer2MateriallyDifferent: a2Diff, + anyForcedCertainty: anyForced, + anyErasedUncertainty: anyErased, + }; + + console.log("\n========== End of Experiment 54Z ==========\n"); + }); +});