experiment: classify captured answer-meaning strengthening

2026-08-11 10:19:19 +01:00
parent a00f7b170d
commit bda3abf893
2 changed files with 208 additions and 0 deletions
@@ -397,3 +397,21 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
 ### Experiment 57J.30 — Proposal-Boundary Live Variance

 **Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.
+
+### Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening (Deterministic)
+
+**Objective:** Determine whether the validator correctly rejected the captured Trial 2 `userSupportedMeaning` ("The decision is conditional on evidence...") as stronger than the raw answer ("Before deciding, I need evidence..."). Fully deterministic — zero Ollama calls.
+
+**Method:** Replicated production `deriveAnswerMeaningProfile()` and `validateAnswerMeaningCompatibilityWithRawAnswer()` logic deterministically against all four captured strings (raw answer, rejected Trial 2, accepted comparisons A and B). Also ran existing apply-proposal tests (64 pass) and rejected-proposal-snapshot tests (7 pass).
+
+**Key finding:** The rejected Trial 2 string contains the word "conditional" which triggers `hasConditionalQualification()` at line 2775 of `lib/graph/apply-proposal.js`, pushing it into the `conditional_tradeoff` category. Both accepted comparisons lack trigger words and correctly remain classified as `other`. The raw answer also profiles as `other`.
+
+**Rejection mechanism:** `validateAnswerMeaningCompatibilityWithRawAnswer()` line 2982–2986: when raw answer = "other" and supported meaning ≠ "other", the validator adds the "stronger reasoning category" error. This is exactly what occurs with Trial 2 (other → conditional_tradeoff).
+
+**Classification: A — VALIDATOR CORRECT.** The captured rejected meaning introduces `conditional_tradeoff` where the raw answer only establishes `other`. This is a real strengthening: "The decision is conditional on..." prescribes a requirement on the decision itself rather than reporting a personal information need. Both accepted comparison variants correctly remain in `other`.
+
+**What this establishes:** The validator's rejection was correct for the captured pair. The mechanism is purely the keyword detector firing on "conditional" — not cold-start variance or structural elements.
+
+**What it does NOT establish:** Whether "conditional" is the ideal trigger word across all contexts, whether "Before deciding" should itself have triggered conditional semantics, generalisation to other answers/domains, or whether cold-start node variance (57J.32) separately affects proposal quality.
+
+Configured Ollama: none used. Production code changed: NO. Tests permanently changed: NO. Temporary test used and removed: YES.