experiment: test consequence of clarification target broadening

2026-08-08 07:29:36 +01:00
parent 4302e3c435
commit 1d1dceefa3
3 changed files with 538 additions and 7 deletions
@@ -36,7 +36,9 @@ Experiment 37 corrected the routing defect from Experiment 36 and tested a cross

 **Commit:** pending (experiment: validate cold-start project recovery) — to be committed this session.

-Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. Two of three targets stayed fully specific; one reproduced the 54W-style broadening on preference-versus-constraint distinctions. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
+Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
+
+Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. This confirms broadening did not alter downstream resolution in this scenario; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.

 Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.

@@ -148,8 +150,8 @@ Answer before continuing:

 ---

-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54W. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Y. Branch: `feature/user-workspace-ux-v0.7`.*

-### Return-to-Work Note (Experiment 54W)
+### Return-to-Work Note (Experiment 54Y)

-54R–54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No drift was detected across the full chain on this single pair of scenarios, though A2's output lost the "preference/trade-off vs hard constraint" granularity from earlier experiments — the chain still succeeded because the coarser representation remained workable. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-chain.test.js` for the full experiment and results. Status pending Rob's review.
+Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
@@ -8901,7 +8901,7 @@ A clarification target can be broadly relevant without being precise enough. Cas

 ### Experiment Conclusion

-Clarification targets remained usable but broadened in one of three tested cases (Case 1). The broadening reproduced the same pattern observed in 54W: preference-versus-constraint distinctions tend to become priority-ordering framings. This is not a failure — the target remains actionable — but it confirms that specificity is lost for at least one class of user-owned ambiguity, and this must be accounted for in downstream question design.
+Clarification targets remained usable but broadened in one of three tested cases (Case 1). The broadening reproduced the same pattern observed in 54W: preference-versus-constraint distinctions tend to become priority-ordering framings. This is not a failure — the target remains actionable — but Specificity loss occurred in one of the three tested ambiguity patterns and was absent in the other two.

 ### Focused Test Result

@@ -8909,7 +8909,7 @@ Clarification targets remained usable but broadened in one of three tested cases

 ### Historical Comparison Result

-The Case 1 result reproduces the A2 output from Experiment 54W ("Priority between business growth and risk avoidance when they conflict" → "preferred priority between business growth and risk avoidance"). The broadening pattern is consistent across both experiments using the same instruction, model, and host. This confirms the issue is not incidental to one particular chain execution but appears inherent to how this model interprets preference-versus-constraint ambiguity under the current instruction.
+The Case 1 result reproduces the A2 output from Experiment 54W ("Priority between business growth and risk avoidance when they conflict" → "preferred priority between business growth and risk avoidance"). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation.

 ### Documentation Updated

@@ -8942,4 +8942,159 @@ This experiment created one new test file only. No clarification-target logic en

 ### Return-to-Work Note (Experiment 54X)

-54W showed the full clarification chain worked in the tested pair but Stage A2 broadened one target; 54X isolated target specificity using three clarification cases under the same 54S instruction — preference/constraint distinction was lost to priority framing (broadened), affordability definition stayed precise (specific), and private factual capacity stayed precise (specific). One of three targets reproduced the 54W-style broadening pattern. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target-specificity.test.js` for the full experiment and results. Status pending Rob's review.
+54W showed the full clarification chain worked in the tested pair but Stage A2 broadened one target; 54X isolated target specificity using three clarification cases under the same 54S instruction — preference/constraint distinction was lost to priority framing (broadened), affordability definition stayed precise (specific), and private factual capacity stayed precise (specific). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target-specificity.test.js` for the full experiment and results. Status pending Rob's review.
+
+## Experiment 54Y — Does a Broader Clarification Target Actually Change the User Question or Resolution? (2026-08-08)
+
+### Objective
+
+Test whether the specificity loss observed in Experiments 54W/54X actually matters downstream. If the clarification target shifts from the precise distinction "preference/trade-off versus hard constraint" to the broader "priority between growth and risk," does that materially change the question asked or the meaning resolved from the user's answer? Passive and test-only. No redesign of target generation, no integration with Behaviour Selection, graph, or UI.
+
+### Hypothesis
+
+The broader target may remain workable but could alter the user-facing distinction. Specifically:
+- the precise target may ask whether risk avoidance is a hard boundary or a trade-off;
+- the broader target may instead ask which objective has priority.
+
+Those questions are related, but the user's answers need not mean exactly the same thing.
+
+### Configuration
+
+Host: `http://192.168.1.111:11434` (same as 54R–54X)
+Model: `qwen-claude:latest` (same as 54R–54X)
+
+### Number of Live Inference Calls
+
+Exactly **4** live Ollama calls — one question per variant + one answer-resolution per variant.
+
+### Context Used
+
+- `docs/current-handoff.md`
+- Experiment 54X only in `docs/design-evolution-log.md` (as historical context for the broadening observation)
+- `tests/reconstruction/semantic-clarification-question.test.js` (for structural reference: instruction and output contract)
+- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (for structural reference: instruction and output contract)
+
+### Fixed Scenario
+
+**Source:** "I want the business to grow, but I don't want to take on more risk."
+
+**Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
+
+### Variant A — Precise Target
+
+**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
+
+This is the human-reviewed specific target.
+
+### Variant B — Broadened Target
+
+**Clarification target:** priority between business growth and risk avoidance when they conflict.
+
+This mirrors the broader target observed in Experiments 54W and 54X.
+
+---
+
+### Stage 1 Results — Question Generation
+
+| Variant | Generated Question |
+|---|---|
+| A (Precise) | Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off? |
+| B (Broadened) | When business growth and risk avoidance conflict, which do you prioritize? |
+
+**Question analysis:** Variant A frames the question around whether avoiding risk is a hard constraint or a preference/trade-off — directly addressing the boundary distinction. Variant B frames it around priority ordering between growth and risk when they conflict — reframing the decision as relative importance rather than a boundary question. The model did not impose a reference wording for Variant B; it produced its natural framing from the broadened target.
+
+### Stage 2 Results — Answer Resolution
+
+| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
+|---|---|---|---|
+| A (Precise) | Avoiding additional risk is established as a hard constraint. | true | null |
+| B (Broadened) | Risk avoidance takes absolute priority over business growth when they conflict. | true | null |
+
+**Resolution analysis:** Both variants produced `targetResolved: true` with `remainingUncertainty: null`. The resolved meanings differ in wording but convey materially equivalent meaning for downstream reasoning: "avoiding additional risk is a hard constraint" and "risk avoidance takes absolute priority over business growth when they conflict" establish the same boundary — no more risk will be accepted. Neither resolution introduced unsupported wider consequences.
+
+### Question Equivalence Classification
+
+**`questions_materially_different`**
+
+The precise target asked whether avoiding extra risk is a trade-off/preference or a hard constraint (a boundary question). The broadened target asked which objective has priority when they conflict (an ordering question). These ask the user to resolve different conceptual distinctions. The distinction was lost as predicted.
+
+### Resolution Equivalence Classification
+
+**`resolutions_materially_equivalent`**
+
+Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. The loss of specificity in the target did not change what was learned from the user's answer.
+
+### Questions Answered
+
+1. **What question did the precise target generate?** "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?"
+2. **What question did the broadened target generate?** "When business growth and risk avoidance conflict, which do you prioritize?"
+3. **Did both questions ask the user to resolve the same underlying distinction?** No — one asked about boundary (constraint vs trade-off); the other asked about priority ordering.
+4. **Did the broader target turn preference-versus-constraint into simple priority ordering?** Yes — it reframed the distinction as relative importance rather than an absolute boundary.
+5. **What resolved meaning did Variant A produce from the fixed answer?** "Avoiding additional risk is established as a hard constraint." (targetResolved: true)
+6. **What resolved meaning did Variant B produce from the same answer?** "Risk avoidance takes absolute priority over business growth when they conflict." (targetResolved: true)
+7. **Were the two resolved meanings materially equivalent?** Yes — both establish that no additional risk will be accepted for growth.
+8. **Did either variant leave remaining uncertainty?** No — both returned null, indicating full resolution of the target from the fixed answer.
+9. **Did either variant introduce unsupported wider consequences?** No — neither inference extended beyond the source meaning and the user's explicit answer.
+10. **Does the broadening materially affect downstream clarification in this tested scenario?** No — despite different questions, the same answer produced the same downstream meaning.
+11. **Does this establish that broad targets are generally safe or unsafe?** No — only one scenario tested.
+12. **Does this establish how target generation should be changed?** No — no fix designed from these results.
+13. **Does this establish UI behaviour?** No — this is a clarification-target test only.
+
+### Limitations
+
+- Only one source scenario and one fixed answer were tested. Different sources may behave differently.
+- Only one ambiguity pattern (preference/constraint) was tested for downstream consequence. Other patterns not assessed.
+- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
+- The semantic equivalence classification is based on structured heuristic checks supplemented by the test output — for definitive judgment, human review of the actual resolved meanings is required.
+- One tested ambiguity pattern; broader safety/generalisation remains untested.
+
+### Experiment Conclusion
+
+**The broader target produced materially equivalent downstream clarification in the tested scenario.**
+
+The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. The loss of specificity did not change what was learned from the user in this scenario.
+
+The key finding is: **Does the distinction we lost actually matter? — In this tested scenario, it did not.** Specificity loss is not automatically a failure; it depends on whether it changes downstream meaning. Whether this holds across other scenarios and ambiguity patterns remains untested.
+
+### Focused Test Result
+
+Both variants produced `targetResolved: true` with zero remaining uncertainty and zero unsupported inferences. Resolutions were materially equivalent despite questions being materially different. 4/4 live inference calls completed successfully (all tests passed).
+
+### Historical Comparison Result
+
+Variant A's resolution ("Avoiding additional risk is established as a hard constraint") matches the expected outcome from Experiment 54V Case 1 and the handoff summary. Variant B's resolution ("Risk avoidance takes absolute priority over business growth when they conflict") represents a coarser framing — but not an incorrect one — for downstream use. The result confirms that the coarser representation remains workable even when it loses the preference-versus-constraint granularity.
+
+### Documentation Updated
+
+- `docs/design-evolution-log.md` — added full Experiment 54Y entry; applied 54X wording corrections
+- `docs/current-handoff.md` — updated with Experiment 54Y summary and new Return-to-Work note
+
+### Confirmation Host and Model Remained Unchanged
+
+Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54X.
+
+### Confirmation Semantic Instructions and Output Contracts Remained Unchanged
+
+Both question and resolution instructions were identical to those defined in Experiments 54U and 54V. Output contracts unchanged from 54U (`{ "question": "..." }`) and 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`).
+
+### Confirmation Production Prompts and Schemas Remained Unchanged
+
+No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
+
+### Confirmation Behaviour Selection Remained Unchanged
+
+Behaviour Selection was not called or referenced. No integration with the selector occurred.
+
+### Confirmation Graph and UI Remained Unchanged
+
+No graph files read or modified. No UI code touched. The experiment is test-only.
+
+### Confirmation No Specificity-Consequence Logic Entered Active Runtime
+
+This experiment created one new test file only. No specificity-consequence logic entered any active runtime path, production module, or behaviour selection output.
+
+---
+
+### Return-to-Work Note (Experiment 54Y)
+
+Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
@@ -0,0 +1,374 @@
+import { describe, it, expect } from "vitest";
+import { config } from "dotenv";
+import path from "path";
+import { fileURLToPath } from "url";
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+config({ path: path.resolve(__dirname, "../../.env.local") });
+
+const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
+const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
+
+if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
+  throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
+}
+
+/**
+ * Make one live Ollama chat call: turn a fixed clarification target
+ * into one concise, neutral user-facing question.
+ */
+async function callClarificationQuestion(source, clarificationTarget) {
+  const instruction = `Write one concise clarification question that asks only about the supplied clarification target. Keep it neutral between the possible meanings. Do not introduce new facts, assumptions, evidence requests, recommendations, or additional questions. Do not explain why you are asking.
+
+Return valid JSON only in this shape:
+{
+  "question": "one clarification question"
+}`;
+
+  const messages = [
+    { role: "system", content: instruction.trim() },
+    {
+      role: "user",
+      content: `Source: ${JSON.stringify(source)}
+
+Clarification target: ${clarificationTarget}`,
+    },
+  ];
+
+  const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      model: OLLAMA_MODEL,
+      messages,
+      format: "json",
+      stream: false,
+    }),
+  });
+
+  if (!res.ok) {
+    throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
+  }
+
+  const data = await res.json();
+  const rawContent = data.message?.content ?? "";
+  const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
+
+  return JSON.parse(cleaned.trim());
+}
+
+/**
+ * Make one live Ollama chat call: take a clarification answer
+ * and return what was resolved, whether the target is resolved,
+ * and any remaining uncertainty about that specific target.
+ */
+async function callClarificationAnswerResolution(source, clarificationTarget, clarificationQuestion, userAnswer) {
+  const instruction = `Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
+
+Return valid JSON only in this shape:
+{
+  "resolvedMeaning": "short statement",
+  "targetResolved": true,
+  "remainingUncertainty": null
+}`;
+
+  const messages = [
+    { role: "system", content: instruction.trim() },
+    {
+      role: "user",
+      content: `Source: ${JSON.stringify(source)}
+
+Clarification target: ${clarificationTarget}
+
+Clarification question: ${clarificationQuestion}
+
+User's answer: ${userAnswer}`,
+    },
+  ];
+
+  const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      model: OLLAMA_MODEL,
+      messages,
+      format: "json",
+      stream: false,
+    }),
+  });
+
+  if (!res.ok) {
+    throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
+  }
+
+  const data = await res.json();
+  const rawContent = data.message?.content ?? "";
+  const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
+
+  return JSON.parse(cleaned.trim());
+}
+
+// ──────────────────────────────────────────────
+// Shared scenario and fixed inputs
+// ──────────────────────────────────────────────
+
+const SOURCE = "I want the business to grow, but I don't want to take on more risk.";
+const FIXED_USER_ANSWER = "It's a hard constraint. I don't want any increase in risk.";
+
+// Variant A -- Precise target
+const VARIANT_A_TARGET = "whether avoiding additional risk is a preference/trade-off or a hard constraint";
+
+// Variant B -- Broadened target
+const VARIANT_B_TARGET = "priority between business growth and risk avoidance when they conflict";
+
+// ──────────────────────────────────────────────
+// Semantic helpers
+// ──────────────────────────────────────────────
+
+/**
+ * Determine whether two questions ask the user to resolve the same
+ * underlying conceptual distinction (not just different wording).
+ */
+function classifyQuestionEquivalence(qA, qB) {
+  const a = qA.toLowerCase();
+  const b = qB.toLowerCase();
+
+  // Check if both frame around preference/trade-off vs hard constraint boundary
+  const isPreferenceConstraintFrame = (q) =>
+    /preference|trade.?off|constraint|boundary|absolute.*risk|no.*increase.*(risk|loss)/i.test(q);
+
+  // Check if both frame around priority ordering between growth and risk
+  const isPriorityOrderingFrame = (q) =>
+    /priority|which.*more.*important|priorit.*growth|growth.*versus.*risk|trade.*growth/i.test(q);
+
+  const aIsPC = isPreferenceConstraintFrame(a);
+  const bIsPC = isPreferenceConstraintFrame(b);
+  const aIsPriority = isPriorityOrderingFrame(a);
+  const bIsPriority = isPriorityOrderingFrame(b);
+
+  // If one is preference/constraint and the other is priority ordering, they are materially different
+  if (aIsPC !== bIsPC && !(aIsPC === bIsPriority)) {
+    return "questions_materially_different";
+  }
+
+  // Check: does one turn a preference/constraint distinction into simple priority?
+  if ((aIsPC && bIsPriority) || (bIsPC && aIsPriority)) {
+    return "questions_materially_different";
+  }
+
+  // If both are same frame type, check if they address the same decision point
+  if (aIsPC && bIsPC) {
+    const bothAboutRiskBoundary = /risk|constraint|boundary/i.test(a) && /risk|constraint|boundary/i.test(b);
+    if (bothAboutRiskBoundary) return "questions_materially_equivalent";
+  }
+
+  // If neither matches the key frames, do manual-style overlap check
+  // Check for shared core concepts: growth, risk, decision/choose/pick
+  const sharedCore = [/growth/i.test(a) && /growth/i.test(b) && /risk/i.test(a) && /risk/i.test(b)];
+  if (sharedCore) return "questions_materially_equivalent";
+
+  return "questions_materially_different";
+}
+
+/**
+ * Determine whether two resolved meanings are materially equivalent.
+ */
+function classifyResolutionEquivalence(rA, rB) {
+  const a = rA.toLowerCase();
+  const b = rB.toLowerCase();
+
+  // Both agree risk avoidance is a hard boundary/constraint/absolute limit?
+  const isHardConstraintMeaning = (r) =>
+    /hard.*(constraint|boundary|limit)|no.*(increase|addition).*risk|absolut.*no.*risk|must.*(not|avoid)/i.test(r);
+
+  // Both say risk avoidance is a trade-off/preference?
+  const isPreferenceMeaning = (r) =>
+    /preference|trade.?off|willing.*trade|condition|depends/i.test(r);
+
+  const aIsHard = isHardConstraintMeaning(a);
+  const bIsHard = isHardConstraintMeaning(b);
+  const aIsPref = isPreferenceMeaning(a);
+  const bIsPref = isPreferenceMeaning(b);
+
+  if (aIsHard === bIsHard && aIsPref === bIsPref) {
+    return "resolutions_materially_equivalent";
+  }
+
+  // If one resolved to hard constraint and the other to preference -- materially different
+  if ((aIsHard && bIsPref) || (bIsHard && aIsPref)) {
+    return "resolutions_materially_different";
+  }
+
+  // Check broader semantic overlap on risk-avoidance meaning
+  const sharedRiskWords = /risk/i.test(a) && /risk/i.test(b);
+  const sharedConstraintWords = /(constraint|boundary|limit|absolute)/i.test(a) && /(constraint|boundary|limit|absolute)/i.test(b);
+
+  if (sharedConstraintWords && sharedRiskWords) return "resolutions_materially_equivalent";
+
+  // Manual-style: check if the core meaning (risk is not acceptable) is present in both
+  const bothSayRiskNotAcceptable = /no.*(increase|accept|more).*risk/i.test(a) && /no.*(increase|accept|more).*risk/i.test(b);
+  if (bothSayRiskNotAcceptable) return "resolutions_materially_equivalent";
+
+  // If one says risk is constraint and the other just mentions priority -- might still be equivalent
+  // in terms of downstream consequence (user does not want more risk either way)
+  const aHasCore = /no|hard|constraint|limit|absolute/i.test(a);
+  const bHasCore = /no|hard|constraint|limit|absolute/i.test(b);
+  if (aHasCore && bHasCore) return "resolutions_materially_equivalent";
+
+  return "resolutions_materially_different";
+}
+
+function hasUnsupportedInference(resolvedMeaning, source) {
+  const m = resolvedMeaning.toLowerCase();
+  // Check for meanings that go beyond the fixed answer and source
+  const unsupportedPatterns = [
+    "growth is impossible",
+    "cannot grow",
+    "should not grow",
+    "specific risk type",
+    "which risk",
+    "budget of",
+    "financial loss",
+    "revenue",
+    "profit",
+    "cost of",
+    "which option",
+  ];
+  const found = unsupportedPatterns.filter((p) => m.includes(p));
+  return found.length > 0 ? found : null;
+}
+
+// ──────────────────────────────────────────────
+// Test suite
+// ──────────────────────────────────────────────
+
+describe("Experiment 54Y - Clarification Target Specificity Consequence", () => {
+  const results = {
+    variantA: { question: null, resolution: null },
+    variantB: { question: null, resolution: null },
+    timings: [],
+  };
+
+  // -- Stage 1: Generate one question per variant --
+
+  it("Stage 1A: generate clarification question from precise target", async () => {
+    const start = Date.now();
+    const result = await callClarificationQuestion(SOURCE, VARIANT_A_TARGET);
+    const elapsed = Date.now() - start;
+    results.timings.push({ stage: "1A-question", ms: elapsed });
+
+    expect(result).toHaveProperty("question");
+    expect(typeof result.question).toBe("string");
+    expect(result.question.trim().endsWith("?")).toBe(true);
+
+    results.variantA.question = result.question;
+  }, 120000);
+
+  it("Stage 1B: generate clarification question from broadened target", async () => {
+    const start = Date.now();
+    const result = await callClarificationQuestion(SOURCE, VARIANT_B_TARGET);
+    const elapsed = Date.now() - start;
+    results.timings.push({ stage: "1B-question", ms: elapsed });
+
+    expect(result).toHaveProperty("question");
+    expect(typeof result.question).toBe("string");
+    expect(result.question.trim().endsWith("?")).toBe(true);
+
+    results.variantB.question = result.question;
+  }, 120000);
+
+  // -- Stage 2: Resolve same answer per variant --
+
+  it("Stage 2A: resolve fixed answer against Variant A question", async () => {
+    const start = Date.now();
+    const result = await callClarificationAnswerResolution(
+      SOURCE,
+      VARIANT_A_TARGET,
+      results.variantA.question,
+      FIXED_USER_ANSWER
+    );
+    const elapsed = Date.now() - start;
+    results.timings.push({ stage: "2A-resolution", ms: elapsed });
+
+    expect(result).toHaveProperty("resolvedMeaning");
+    expect(typeof result.resolvedMeaning).toBe("string");
+    expect(typeof result.targetResolved).toBe("boolean");
+
+    results.variantA.resolution = result;
+  }, 120000);
+
+  it("Stage 2B: resolve fixed answer against Variant B question", async () => {
+    const start = Date.now();
+    const result = await callClarificationAnswerResolution(
+      SOURCE,
+      VARIANT_B_TARGET,
+      results.variantB.question,
+      FIXED_USER_ANSWER
+    );
+    const elapsed = Date.now() - start;
+    results.timings.push({ stage: "2B-resolution", ms: elapsed });
+
+    expect(result).toHaveProperty("resolvedMeaning");
+    expect(typeof result.resolvedMeaning).toBe("string");
+    expect(typeof result.targetResolved).toBe("boolean");
+
+    results.variantB.resolution = result;
+  }, 120000);
+
+  // -- Aggregate evaluation --
+
+  it("54Y: evaluate question equivalence", () => {
+    const eq = classifyQuestionEquivalence(results.variantA.question, results.variantB.question);
+    results.questionEquivalence = eq;
+    console.log("\n=== Experiment 54Y Results ===");
+    console.log("\n--- Source ---");
+    console.log(SOURCE);
+    console.log("\n--- Fixed User Answer ---");
+    console.log(FIXED_USER_ANSWER);
+    console.log("\n--- Variant A (Precise Target) ---");
+    console.log("Target:", VARIANT_A_TARGET);
+    console.log("Generated question:", results.variantA.question);
+    console.log("--- Variant B (Broadened Target) ---");
+    console.log("Target:", VARIANT_B_TARGET);
+    console.log("Generated question:", results.variantB.question);
+    console.log("\n--- Question Equivalence: " + eq + " ---");
+  });
+
+  it("54Y: evaluate resolution equivalence", () => {
+    const eq = classifyResolutionEquivalence(results.variantA.resolution.resolvedMeaning, results.variantB.resolution.resolvedMeaning);
+    results.resolutionEquivalence = eq;
+
+    console.log("\n--- Variant A Resolution ---");
+    console.log("resolvedMeaning:", results.variantA.resolution.resolvedMeaning);
+    console.log("targetResolved:", results.variantA.resolution.targetResolved);
+    console.log("remainingUncertainty:", results.variantA.resolution.remainingUncertainty ?? null);
+
+    console.log("\n--- Variant B Resolution ---");
+    console.log("resolvedMeaning:", results.variantB.resolution.resolvedMeaning);
+    console.log("targetResolved:", results.variantB.resolution.targetResolved);
+    console.log("remainingUncertainty:", results.variantB.resolution.remainingUncertainty ?? null);
+
+    console.log("\n--- Resolution Equivalence: " + eq + " ---");
+
+    // Check for unsupported inferences
+    const aUnsupported = hasUnsupportedInference(results.variantA.resolution.resolvedMeaning, SOURCE);
+    const bUnsupported = hasUnsupportedInference(results.variantB.resolution.resolvedMeaning, SOURCE);
+    if (aUnsupported) console.log("Variant A unsupported inferences:", aUnsupported);
+    if (bUnsupported) console.log("Variant B unsupported inferences:", bUnsupported);
+
+    // Timing summary
+    const totalMs = results.timings.reduce((s, t) => s + t.ms, 0);
+    const msArr = results.timings.map((t) => t.ms);
+    console.log("\n--- Timing ---");
+    console.log("Calls:", results.timings.length);
+    console.log("Total:", totalMs + "ms");
+    console.log("Average:", (totalMs / results.timings.length).toFixed(1) + "ms per call");
+    console.log("Fastest:", Math.min(...msArr) + "ms");
+    console.log("Slowest:", Math.max(...msArr) + "ms");
+
+    // Store for later review
+    results.totalCalls = results.timings.length;
+    results.totalTimeMs = totalMs;
+  });
+});