experiment: classify captured answer-meaning strengthening

2026-08-11 10:19:19 +01:00
parent a00f7b170d
commit bda3abf893
2 changed files with 208 additions and 0 deletions
@@ -0,0 +1,190 @@
+# Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening
+
+## Objective
+
+Answer exactly: given the exact raw answer and exact rejected `userSupportedMeaning` captured in 57J.32 Trial 2, is the current validator correct to classify the proposal meaning as a stronger reasoning category than the user established?
+
+This task addresses only the existing semantic contract — not cold-start graph variance, addedNodes/edges, or provenance/connectivity.
+
+## Configured apparatus
+
+- **Branch:** `feature/rejected-proposal-diagnostics-v0.16`
+- **HEAD at experiment start:** `a00f7b1` — experiment: inspect rejected proposal live variance
+- **Ollama calls made:** 0 (fully deterministic)
+- **Production code changed:** NO
+- **Tests permanently changed:** NO
+
+## Fixed captured evidence
+
+### Raw user answer
+
+> "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
+
+### Rejected Trial 2 `userSupportedMeaning`
+
+> "The decision is conditional on evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers."
+
+### Accepted comparison A
+
+> "The user requires direct evidence that projected office savings are realistic and that the relocation will not materially increase the loss of key engineers before making a decision."
+
+### Accepted comparison B
+
+> "The user requires concrete evidence verifying that projected office savings are realistic and confirming that key engineer attrition will not materially increase before deciding on the relocation."
+
+## Part 1 — Classifier trace (deterministic, from production code)
+
+### Raw answer profile
+
+| Field | Value |
+|---|---|
+| `category` | `other` |
+| `resolutionGuidance` | `null` |
+
+**Reasoning:** No uncertain, conditional, constraint, or priority trigger words fire. The text passes through all detection gates and reaches the default "other" category.
+
+### Rejected Trial 2 profile
+
+| Field | Value |
+|---|---|
+| `category` | `conditional_tradeoff` |
+| `resolutionGuidance` | `may_resolve` |
+
+**Reasoning:** `hasConditionalQualification()` fires on the word "conditional" inside "decision is conditional on" (line 2775 of `lib/graph/apply-proposal.js`). This sets `conditionalPreferenceStructure = true`, which returns `conditional_tradeoff` before any other gate is reached.
+
+### Accepted comparison A profile
+
+| Field | Value |
+|---|---|
+| `category` | `other` |
+| `resolutionGuidance` | `null` |
+
+**Reasoning:** No trigger words fire. "Requires" is not in the conditional qualification list. Passes to default "other".
+
+### Accepted comparison B profile
+
+| Field | Value |
+|---|---|
+| `category` | `other` |
+| `resolutionGuidance` | `null` |
+
+**Reasoning:** Same as A — no trigger words fire. "Before deciding" does not match any conditional/uncertainty/constraint/priority gate. Reaches default "other".
+
+## Part 2 — Exact rejection mechanism
+
+### Function
+
+`validateAnswerMeaningCompatibilityWithRawAnswer()` in `lib/graph/apply-proposal.js`, line 2932.
+
+### Branch/condition
+
+Lines 2982–2986:
+```javascript
+if (rawAnswerProfile.category === "other") {
+  if (supportedMeaningProfile.category !== "other") {
+    errors.push(
+      "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes.",
+    );
+  }
+}
+```
+
+### Categories involved
+
+- **Raw answer category:** `other` — no protective category signal detected
+- **Rejected meaning category:** `conditional_tradeoff` — fired by `hasConditionalQualification()` matching "conditional" in "decision is conditional on"
+
+### Why the proposed category is considered stronger
+
+The validator's guard for unclassified ("other") answers works on a simple principle: if the raw answer establishes no specific reasoning category, and the extracted meaning lands in any protected category (uncertain, explicit_hard_constraint, relative_priority_only, conditional_tradeoff), that is treated as introducing a stronger reasoning structure than the user supplied.
+
+The `conditional_tradeoff` category signals "there is a default position qualified by an exception condition" — which implies the user has a preference/constraint stance that can be overridden under specific circumstances. This is categorically stronger than a neutral information need ("I need evidence before deciding"), which the raw answer establishes.
+
+## Part 3 — Human semantic comparison
+
+### Raw answer establishes:
+
+**A** (information needed before deciding) — YES
+The raw answer explicitly states "Before deciding, I need evidence..." — this unambiguously establishes an information need prior to decision-making.
+
+**B** (decision is conditional on satisfying that evidence) — Partially / borderline
+"Before deciding" implies a temporal/priority relationship but does not assert conditionality of the *decision itself*. It reports the speaker's personal requirement rather than prescribing a property of "the decision."
+
+**C** (hard veto/constraint) — NO
+No hard-constraint language present.
+
+**D** (explicit decision rule) — NO
+No rule structure established.
+
+**E** — Cannot distinguish A from B with full certainty; the strongest supported meaning is A.
+
+### Rejected Trial 2 meaning: "The decision is conditional on..."
+
+**Classification: SLIGHT STRENGTHENING → MATERIAL STRENGTHENING (borderline)**
+
+"Before deciding, I need..." frames the condition as the *speaker's* requirement. "The decision is conditional on..." frames it as an impersonal property of the decision itself. The shift from personal information need to prescriptive decision structure is a real change — not merely a paraphrase. However, it stays within the same broad semantic domain (evidence-before-decision).
+
+The stronger case for MATERIAL STRENGTHENING: In reasoning terms, "the decision requires X" can be operationalized as a hard gate on decision-making, whereas "I need X before deciding" is descriptive of intent. The validator's categorical treatment is therefore defensible.
+
+### Accepted comparison A: "The user requires evidence..."
+
+**Classification: SLIGHT STRENGTHENING**
+
+More explicit about who holds the requirement ("the user"), more precise ("before making a decision"). Still within the same information-need semantic domain as the raw answer. Does not introduce conditionality of the decision itself — stays in `other`.
+
+### Accepted comparison B: "The user requires concrete evidence verifying..."
+
+**Classification: SLIGHT STRENGTHENING**
+
+Uses "concrete" and "verifying/confirming" which are mild strengthening adjectives, but does not cross into any protected reasoning category. Stays in `other`.
+
+## Part 4 — Deterministic reproduction
+
+### Command
+
+```
+npx vitest run tests/graph/experiment-57j33-tmp.test.mjs --reporter=verbose
+```
+
+(8 focused tests exercising deriveAnswerMeaningProfile and validateAnswerMeaningCompatibilityWithRawAnswer against all four captured strings.)
+
+### Result
+
+All 8 tests PASS.
+
+| Test | Expected | Actual | Status |
+|---|---|---|---|
+| Raw answer profiles as 'other' | `other` | `other` | PASS |
+| Rejected Trial 2 profiles as 'conditional_tradeoff' | `conditional_tradeoff` | `conditional_tradeoff` | PASS |
+| Comparison A profiles as 'other' | `other` | `other` | PASS |
+| Comparison B profiles as 'other' | `other` | `other` | PASS |
+| Validator rejects Trial 2 | error present | error present | PASS |
+| Validator accepts comparison A | no errors | no errors | PASS |
+| Validator accepts comparison B | no errors | no errors | PASS |
+| Trigger: 'conditional' fires hasConditionalQualification | true for Trial 2, false for raw | confirmed | PASS |
+
+### Captured Trial 2 rejection reproduced: YES
+
+### Classification: **A — VALIDATOR CORRECT**
+
+### Why
+
+The validator correctly identifies that "The decision is conditional on..." introduces a `conditional_tradeoff` category where the raw answer only establishes `other`. The `conditional` keyword at line 2775 of `hasConditionalQualification()` fires because "decision is conditional on" contains the word "conditional". This pushes the meaning from a neutral information need into a protected reasoning category that implies default preference + exception qualification — which is indeed stronger than what the raw answer establishes.
+
+The key insight: this is not a subtle wording issue. The rejected Trial 2 string literally contains the word "conditional" which triggers a category detector in production code. The accepted comparisons A and B do not contain any trigger words and correctly remain classified as `other`.
+
+### What this establishes
+
+1. The validator's rejection of the captured Trial 2 meaning is **correct** — the meaning introduces a stronger reasoning category (`conditional_tradeoff`) where the raw answer only supports `other`.
+2. The mechanism is the `hasConditionalQualification()` keyword detector (line 2775) firing on "conditional" in "decision is conditional on".
+3. Both accepted comparison variants (A and B) remain correctly classified as `other` by the same detector.
+4. The rejection does not involve cold-start graph variance or structural elements — it is purely a meaning-category mismatch at the validator gate.
+
+### What it does NOT establish
+
+1. Whether "conditional" is the ideal trigger word for `hasConditionalQualification()` in all contexts (this is about the existing boundary only).
+2. Whether the raw answer's "Before deciding" should itself have triggered conditional semantics — that would require changing the detector, which is outside scope.
+3. Generalisation to other answers or domains beyond this specific captured pair.
+4. Whether the cold-start node variance (6→8 nodes) observed in 57J.32 affects proposal quality downstream — that is a separate investigation.
+
+### Temporary test removed: YES