experiment: test clarification null stability

This commit is contained in:
2026-08-08 06:13:43 +01:00
parent 6944f358a9
commit c8ead0f690
3 changed files with 419 additions and 6 deletions
+4 -4
View File
@@ -22,7 +22,7 @@
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). 3/3 correctly classified. The model can identify the specific unresolved user-owned distinction without generating a question, choosing a winner, or changing Behaviour Selection. One behavioral note: when requiresUserClarification is false, the model sometimes defaults to producing a target rather than respecting null — but this did not affect any of the three tested cases in this run. Same host/model; no production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). Same host/model; no production code changed. Status pending Rob's review.
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
@@ -144,8 +144,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 3853, 54A54S. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 3853, 54A54T. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54S)
### Return-to-Work Note (Experiment 54T)
54R showed that evidence uncertainty and user-owned ambiguity can be distinguished across three tested patterns. 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — growth-versus-risk priority correctly identified preference/trade-off versus hard constraint; delivery causes correctly returned null (evidence-resolvable); affordable ambiguity correctly identified upfront versus long-term cost meaning. 3/3 clarified-target classification correct. No clarification question was generated. Behaviour Selection remained untouched. Same host/model (qwen-claude:latest on 192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
54S correctly identified clarification targets in its final run but earlier false/null behaviour was unstable across unrecorded repetitions. 54T repeated one `requiresUserClarification: false` case and one true control three times each to test stability. False consistently returned null (3/3). True consistently returned the intended semantic target (3/3). No asymmetric instability was observed in this experiment. No prompt correction or production integration was attempted. Behaviour Selection remained untouched. Same host/model (qwen-claude:latest on 192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.
+143 -2
View File
@@ -8146,10 +8146,151 @@ The model correctly identified the definition ambiguity — upfront cost versus
### Conclusion
**The model identified the correct clarification target in all tested cases.**
**The final three-case run was correct, but earlier repetitions showed instability when clarification was explicitly not required. Clarification-target identification therefore appears promising, but null enforcement is not yet stable.**
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour (the model occasionally produced a target despite `requiresUserClarification: false` in earlier runs), not merely a test warning. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
## Experiment 54T — Is "No Clarification Needed" Stable Across Repeated Identical Inputs? (2026-08-08)
### Objective
First, tighten Experiment 54S so its conclusion reflects the instability observed during earlier runs.
Then test one narrow question:
> **When `requiresUserClarification` is explicitly false, does the model consistently return no clarification target across repeated identical calls?**
This experiment exists because 54S produced the correct final result but earlier runs sometimes generated a clarification target even when clarification was explicitly not required.
This is passive and test-only.
Do not change the clarification-target instruction yet.
Do not generate questions.
Do not change production behaviour.
### Hypothesis
Two possibilities are plausible.
**Stable gating:** When `requiresUserClarification` is false, the model consistently returns `null`.
**Semantic override:** The model sometimes ignores the explicit false flag and invents a clarification target because it sees unresolved uncertainty in the source.
Either result is useful.
Do not try to correct the behaviour in this experiment.
### Configuration
Host: `http://192.168.1.111:11434` (same as 54S)
Model: `qwen-claude:latest` (same as 54S)
No localhost fallback. No experiment-specific model variable.
### Number of Live Inference Calls
Exactly **6** live Ollama calls — three per case, repeated identical input each time.
### Semantic Instruction Used
Exact Experiment 54S instruction unchanged:
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
No examples. No expected answers. Same instruction for all six cases.
### Output Contract
Unchanged from 54S:
```json
{
"clarificationTarget": "short statement" | null
}
```
### Case A — Evidence-Resolvable / False (the unstable case from 54S)
**Source:** "Orders are arriving late and customers have started complaining."
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
**Fixed clarification status:** `requiresUserClarification: false`
**Expected result:** `clarificationTarget: null`
Run this exact case **3 times** without changing wording. This is the unstable Case 2 from 54S.
### Case B — User-Owned Ambiguity / True Control
**Source:** "I want the business to grow, but I don't want to take on more risk."
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
**Fixed clarification status:** `requiresUserClarification: true`
**Expected semantic target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
Run this exact case **3 times**. Purpose: determine whether instability is specific to suppressing clarification or affects clarification-target generation generally.
### Results
#### Case A — Evidence-Resolvable / False
| Run | Result | Classification |
|-----|--------|----------------|
| 1 | `null` (15,566ms) | null ✓ |
| 2 | `null` (13,800ms) | null ✓ |
| 3 | `null` (19,087ms) | null ✓ |
**Null count: 3/3**
**Non-null count: 0/3**
No clarification targets were invented. The model consistently returned `null` across all three repeated identical runs with `requiresUserClarification: false`.
#### Case B — User-Owned Ambiguity / True Control
| Run | Target | Classification |
|-----|--------|----------------|
| 1 | "The relative priority between business growth and strict risk avoidance when they conflict" (18,595ms) | target_correct |
| 2 | "Preferred priority between accelerating business growth and strictly avoiding additional risk" (19,458ms) | target_correct |
| 3 | "Your maximum acceptable level of additional risk relative to desired business growth." (18,976ms) | target_correct |
**Correct-target count: 3/3**
**Incorrect-target count: 0/3**
**Null count: 0/3**
All three runs produced semantically correct targets aligned with the human reference. No null responses observed when clarification was required.
### Timing
- Total time: 105,470ms (105.5s)
- Average: 17,578.3ms per call
- Fastest: 13,798ms (Case A run 2)
- Slowest: 19,457ms (Case B run 3)
### Required Questions — Answers
1. How many Case A runs returned `null`? **3**
2. How many Case A runs returned a non-null clarification target? **0**
3. If Case A produced a target, what distinction did it invent? **N/A — none produced**
4. How many Case B runs produced the correct clarification target? **3**
5. Did Case B ever incorrectly return `null`? **No**
6. Is false/null behaviour materially stable across the three repeated runs? **Yes** — all 3 returned null with zero variance
7. Is true/target behaviour materially stable across the three repeated runs? **Yes** — all 3 produced semantically correct targets with zero variance
8. Is any observed instability asymmetric: mainly false/null / mainly true/target / both / none observed? **None observed in this experiment.** Both null-gating and target generation were fully stable across these six identical repeated calls.
9. Does this experiment establish why instability occurs? **No** — this is an observation experiment, not a diagnostic one.
10. Does this establish how to enforce null behaviour? **No** — it shows the current instruction+output contract *can* produce stable null when false, but does not explain earlier variability.
11. Does this establish how Behaviour Selection should work? **No.**
### Limitations
- Only two cases tested (one false, one true); limited domain coverage.
- Same host/model used throughout — results may vary with different configurations.
- This is a stability observation experiment; it does not diagnose root causes of earlier variability.
- Does not establish generalisation beyond these specific inputs.
- The model's behaviour in earlier unrecorded runs (when null-gating failed) remains the unknown variable.
### Evaluation Conclusion
**Clarification null-gating was stable across all tested repeats**
Case A returned `null` in 3 of 3 runs. Case B produced correct targets in 3 of 3 runs. No instability was observed in either direction during this experiment.
### Status
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.
@@ -0,0 +1,272 @@
import { describe, it, expect } from "vitest";
import { config } from "dotenv";
import path from "path";
import { fileURLToPath } from "url";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
config({ path: path.resolve(__dirname, "../../.env.local") });
const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
}
/**
* Make one live Ollama chat call: identify the specific user-owned
* distinction that remains unresolved when clarification is required.
* Uses the exact Experiment 54S instruction unchanged.
*/
async function callClarificationTarget(source, disagreement, requiresUserClarification) {
const instruction = `Identify the specific unresolved distinction that only the user can clarify.
If clarification is required (requiresUserClarification: true), return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact.
If clarification is not required (requiresUserClarification: false), return null.
Do not write a question. Do not add evidence needs. Do not select a preferred interpretation.
Return valid JSON only in this shape:
{
"clarificationTarget": "short statement" | null
}`;
const messages = [
{ role: "system", content: instruction.trim() },
{
role: "user",
content: `Source: ${JSON.stringify(source)}
Disagreement:
${disagreement.map((d, i) => `${i + 1}. ${d}`).join("\n")}
requiresUserClarification: ${requiresUserClarification}`,
},
];
const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: OLLAMA_MODEL,
messages,
format: "json",
stream: false,
}),
});
if (!res.ok) {
throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
}
const data = await res.json();
const rawContent = data.message?.content ?? "";
const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
return JSON.parse(cleaned.trim());
}
// ──────────────────────────────────────────────
// Human reference for Case B semantic review
// ──────────────────────────────────────────────
const HUMAN_REF_CASE_B = {
expectedDistinction:
"whether avoiding additional risk is a preference/trade-off or a hard constraint",
requiredConcepts: [
"risk",
"constraint",
"preference",
"trade-off",
"avoiding",
"additional",
"hard",
],
forbiddenConcepts: ["evidence", "investigate", "check", "look at", "data"],
};
function evaluateCaseBTarget(target) {
if (target == null) return { classification: "null", target };
if (typeof target !== "string" || !target.trim())
return { classification: "null", target };
const t = target.trim().toLowerCase();
if (t.endsWith("?")) return { classification: "null", target };
// Semantic check against human reference
const reqConcepts = HUMAN_REF_CASE_B.requiredConcepts;
const hasRequired = reqConcepts.some(
(c) => t.includes(c.toLowerCase())
);
const forbiddenConcepts = HUMAN_REF_CASE_B.forbiddenConcepts;
const hasForbidden = forbiddenConcepts.some(
(c) => t.includes(c.toLowerCase())
);
if (!hasRequired || hasForbidden) {
return { classification: "target_incorrect", target };
}
return { classification: "target_correct", target };
}
// ──────────────────────────────────────────────
// Fixed cases
// ──────────────────────────────────────────────
const CASE_A = {
id: "Case A — Evidence-Resolvable / False",
source: "Orders are arriving late and customers have started complaining.",
disagreement: [
"delays may be caused by insufficient staff capacity",
"delays may be caused by unreliable supplier lead times",
],
requiresUserClarification: false,
expectedNull: true,
};
const CASE_B = {
id: "Case B — User-Owned Ambiguity / True Control",
source: "I want the business to grow, but I don't want to take on more risk.",
disagreement: [
"growth should be prioritised even if some additional risk is unavoidable",
"avoiding additional risk is a hard constraint even if growth is slower",
],
requiresUserClarification: true,
expectedNull: false,
};
const RUNS_PER_CASE = 3;
// ──────────────────────────────────────────────
// Test suite — Case A (false → null stability)
// ──────────────────────────────────────────────
describe("Experiment 54T — Clarification Null Stability", () => {
const results = { a: [], b: [] };
const timings = [];
describe("Case A — Evidence-Resolvable / False (repeated " + RUNS_PER_CASE + "×)", () => {
for (let i = 0; i < RUNS_PER_CASE; i++) {
it(`run ${i + 1}`, async () => {
const start = Date.now();
const result = await callClarificationTarget(
CASE_A.source,
CASE_A.disagreement,
CASE_A.requiresUserClarification
);
const elapsed = Date.now() - start;
timings.push({ caseId: CASE_A.id, run: i + 1, ms: elapsed });
const isNull = result.clarificationTarget == null;
results.a.push({ run: i + 1, result, isNull });
// Structural check
expect(result.clarificationTarget).toBeDefined();
// Null-gating: false → null expected
if (CASE_A.expectedNull) {
console.log(
`[54T Case A run ${i + 1}] clarificationTarget: ${isNull ? "null ✓" : `"${result.clarificationTarget}" ✗`}`
);
expect(isNull).toBe(true);
} else {
expect(isNull).toBe(false);
}
}, 120000);
}
});
describe("Case B — User-Owned Ambiguity / True Control (repeated " + RUNS_PER_CASE + "×)", () => {
for (let i = 0; i < RUNS_PER_CASE; i++) {
it(`run ${i + 1}`, async () => {
const start = Date.now();
const result = await callClarificationTarget(
CASE_B.source,
CASE_B.disagreement,
CASE_B.requiresUserClarification
);
const elapsed = Date.now() - start;
timings.push({ caseId: CASE_B.id, run: i + 1, ms: elapsed });
const ev = evaluateCaseBTarget(result.clarificationTarget);
results.b.push({ run: i + 1, result, classification: ev.classification });
// Structural check
expect(result.clarificationTarget).toBeDefined();
console.log(
`[54T Case B run ${i + 1}] classification: ${ev.classification} | target: "${result.clarificationTarget ?? "null"}"`
);
// true → non-null expected
if (CASE_B.expectedNull) {
expect(ev.classification).not.toBe("null");
} else {
// This case expects a non-null correct target
expect(ev.classification).toBe("target_correct");
}
}, 120000);
}
});
// ──────────────────────────────────────────────
// Aggregate summary
// ──────────────────────────────────────────────
it("54T aggregate results", () => {
const aNull = results.a.filter((r) => r.isNull).length;
const aNonNull = results.a.filter((r) => !r.isNull).length;
const bCorrect = results.b.filter((r) => r.classification === "target_correct").length;
const bIncorrect = results.b.filter((r) => r.classification === "target_incorrect").length;
const bNull = results.b.filter((r) => r.classification === "null").length;
const totalMs = timings.reduce((s, t) => s + t.ms, 0);
const avgMs = totalMs / timings.length;
const fastMs = Math.min(...timings.map((t) => t.ms));
const slowMs = Math.max(...timings.map((t) => t.ms));
console.log("\n=== Experiment 54T Aggregate Results ===");
console.log(`\n--- Case A (false → null) ---`);
for (const r of results.a) {
console.log(
`Run ${r.run}: ${r.isNull ? "null" : `"${r.result.clarificationTarget}"`}`
);
}
console.log(`Null: ${aNull}/${results.a.length}`);
console.log(`Non-null: ${aNonNull}/${results.a.length}`);
if (aNonNull > 0) {
console.log("\nInvented clarification targets:");
results
.filter((r) => !r.isNull)
.forEach((r) => console.log(` - "${r.result.clarificationTarget}"`));
}
console.log(`\n--- Case B (true → target) ---`);
for (const r of results.b) {
console.log(
`Run ${r.run}: ${r.classification} | "${r.result ?? "null"}"`
);
}
console.log(`Correct: ${bCorrect}/${results.b.length}`);
console.log(`Incorrect: ${bIncorrect}/${results.b.length}`);
console.log(`Null: ${bNull}/${results.b.length}`);
console.log(`\n--- Timing ---`);
console.log(`Total live calls: ${timings.length}`);
console.log(`Total time: ${totalMs}ms (${(totalMs / 1000).toFixed(1)}s)`);
console.log(`Average: ${avgMs.toFixed(1)}ms per call`);
console.log(`Fastest: ${fastMs}ms`);
console.log(`Slowest: ${slowMs}ms`);
// Summary assertions
expect(timings.length).toBe(6);
expect(aNull + aNonNull).toBe(3);
expect(bCorrect + bIncorrect + bNull).toBe(3);
});
});