From bda3abf8934eaa6c9a4cada5153081093bacf685 Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 10:19:19 +0100 Subject: [PATCH] experiment: classify captured answer-meaning strengthening --- docs/current-handoff.md | 18 ++++ docs/experiment-57j33.md | 190 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 208 insertions(+) create mode 100644 docs/experiment-57j33.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 87e9603..bc5a9e9 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -397,3 +397,21 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti ### Experiment 57J.30 — Proposal-Boundary Live Variance **Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed. + +### Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening (Deterministic) + +**Objective:** Determine whether the validator correctly rejected the captured Trial 2 `userSupportedMeaning` ("The decision is conditional on evidence...") as stronger than the raw answer ("Before deciding, I need evidence..."). Fully deterministic — zero Ollama calls. + +**Method:** Replicated production `deriveAnswerMeaningProfile()` and `validateAnswerMeaningCompatibilityWithRawAnswer()` logic deterministically against all four captured strings (raw answer, rejected Trial 2, accepted comparisons A and B). Also ran existing apply-proposal tests (64 pass) and rejected-proposal-snapshot tests (7 pass). + +**Key finding:** The rejected Trial 2 string contains the word "conditional" which triggers `hasConditionalQualification()` at line 2775 of `lib/graph/apply-proposal.js`, pushing it into the `conditional_tradeoff` category. Both accepted comparisons lack trigger words and correctly remain classified as `other`. The raw answer also profiles as `other`. + +**Rejection mechanism:** `validateAnswerMeaningCompatibilityWithRawAnswer()` line 2982–2986: when raw answer = "other" and supported meaning ≠ "other", the validator adds the "stronger reasoning category" error. This is exactly what occurs with Trial 2 (other → conditional_tradeoff). + +**Classification: A — VALIDATOR CORRECT.** The captured rejected meaning introduces `conditional_tradeoff` where the raw answer only establishes `other`. This is a real strengthening: "The decision is conditional on..." prescribes a requirement on the decision itself rather than reporting a personal information need. Both accepted comparison variants correctly remain in `other`. + +**What this establishes:** The validator's rejection was correct for the captured pair. The mechanism is purely the keyword detector firing on "conditional" — not cold-start variance or structural elements. + +**What it does NOT establish:** Whether "conditional" is the ideal trigger word across all contexts, whether "Before deciding" should itself have triggered conditional semantics, generalisation to other answers/domains, or whether cold-start node variance (57J.32) separately affects proposal quality. + +Configured Ollama: none used. Production code changed: NO. Tests permanently changed: NO. Temporary test used and removed: YES. diff --git a/docs/experiment-57j33.md b/docs/experiment-57j33.md new file mode 100644 index 0000000..948a4cb --- /dev/null +++ b/docs/experiment-57j33.md @@ -0,0 +1,190 @@ +# Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening + +## Objective + +Answer exactly: given the exact raw answer and exact rejected `userSupportedMeaning` captured in 57J.32 Trial 2, is the current validator correct to classify the proposal meaning as a stronger reasoning category than the user established? + +This task addresses only the existing semantic contract — not cold-start graph variance, addedNodes/edges, or provenance/connectivity. + +## Configured apparatus + +- **Branch:** `feature/rejected-proposal-diagnostics-v0.16` +- **HEAD at experiment start:** `a00f7b1` — experiment: inspect rejected proposal live variance +- **Ollama calls made:** 0 (fully deterministic) +- **Production code changed:** NO +- **Tests permanently changed:** NO + +## Fixed captured evidence + +### Raw user answer + +> "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +### Rejected Trial 2 `userSupportedMeaning` + +> "The decision is conditional on evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers." + +### Accepted comparison A + +> "The user requires direct evidence that projected office savings are realistic and that the relocation will not materially increase the loss of key engineers before making a decision." + +### Accepted comparison B + +> "The user requires concrete evidence verifying that projected office savings are realistic and confirming that key engineer attrition will not materially increase before deciding on the relocation." + +## Part 1 — Classifier trace (deterministic, from production code) + +### Raw answer profile + +| Field | Value | +|---|---| +| `category` | `other` | +| `resolutionGuidance` | `null` | + +**Reasoning:** No uncertain, conditional, constraint, or priority trigger words fire. The text passes through all detection gates and reaches the default "other" category. + +### Rejected Trial 2 profile + +| Field | Value | +|---|---| +| `category` | `conditional_tradeoff` | +| `resolutionGuidance` | `may_resolve` | + +**Reasoning:** `hasConditionalQualification()` fires on the word "conditional" inside "decision is conditional on" (line 2775 of `lib/graph/apply-proposal.js`). This sets `conditionalPreferenceStructure = true`, which returns `conditional_tradeoff` before any other gate is reached. + +### Accepted comparison A profile + +| Field | Value | +|---|---| +| `category` | `other` | +| `resolutionGuidance` | `null` | + +**Reasoning:** No trigger words fire. "Requires" is not in the conditional qualification list. Passes to default "other". + +### Accepted comparison B profile + +| Field | Value | +|---|---| +| `category` | `other` | +| `resolutionGuidance` | `null` | + +**Reasoning:** Same as A — no trigger words fire. "Before deciding" does not match any conditional/uncertainty/constraint/priority gate. Reaches default "other". + +## Part 2 — Exact rejection mechanism + +### Function + +`validateAnswerMeaningCompatibilityWithRawAnswer()` in `lib/graph/apply-proposal.js`, line 2932. + +### Branch/condition + +Lines 2982–2986: +```javascript +if (rawAnswerProfile.category === "other") { + if (supportedMeaningProfile.category !== "other") { + errors.push( + "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes.", + ); + } +} +``` + +### Categories involved + +- **Raw answer category:** `other` — no protective category signal detected +- **Rejected meaning category:** `conditional_tradeoff` — fired by `hasConditionalQualification()` matching "conditional" in "decision is conditional on" + +### Why the proposed category is considered stronger + +The validator's guard for unclassified ("other") answers works on a simple principle: if the raw answer establishes no specific reasoning category, and the extracted meaning lands in any protected category (uncertain, explicit_hard_constraint, relative_priority_only, conditional_tradeoff), that is treated as introducing a stronger reasoning structure than the user supplied. + +The `conditional_tradeoff` category signals "there is a default position qualified by an exception condition" — which implies the user has a preference/constraint stance that can be overridden under specific circumstances. This is categorically stronger than a neutral information need ("I need evidence before deciding"), which the raw answer establishes. + +## Part 3 — Human semantic comparison + +### Raw answer establishes: + +**A** (information needed before deciding) — YES +The raw answer explicitly states "Before deciding, I need evidence..." — this unambiguously establishes an information need prior to decision-making. + +**B** (decision is conditional on satisfying that evidence) — Partially / borderline +"Before deciding" implies a temporal/priority relationship but does not assert conditionality of the *decision itself*. It reports the speaker's personal requirement rather than prescribing a property of "the decision." + +**C** (hard veto/constraint) — NO +No hard-constraint language present. + +**D** (explicit decision rule) — NO +No rule structure established. + +**E** — Cannot distinguish A from B with full certainty; the strongest supported meaning is A. + +### Rejected Trial 2 meaning: "The decision is conditional on..." + +**Classification: SLIGHT STRENGTHENING → MATERIAL STRENGTHENING (borderline)** + +"Before deciding, I need..." frames the condition as the *speaker's* requirement. "The decision is conditional on..." frames it as an impersonal property of the decision itself. The shift from personal information need to prescriptive decision structure is a real change — not merely a paraphrase. However, it stays within the same broad semantic domain (evidence-before-decision). + +The stronger case for MATERIAL STRENGTHENING: In reasoning terms, "the decision requires X" can be operationalized as a hard gate on decision-making, whereas "I need X before deciding" is descriptive of intent. The validator's categorical treatment is therefore defensible. + +### Accepted comparison A: "The user requires evidence..." + +**Classification: SLIGHT STRENGTHENING** + +More explicit about who holds the requirement ("the user"), more precise ("before making a decision"). Still within the same information-need semantic domain as the raw answer. Does not introduce conditionality of the decision itself — stays in `other`. + +### Accepted comparison B: "The user requires concrete evidence verifying..." + +**Classification: SLIGHT STRENGTHENING** + +Uses "concrete" and "verifying/confirming" which are mild strengthening adjectives, but does not cross into any protected reasoning category. Stays in `other`. + +## Part 4 — Deterministic reproduction + +### Command + +``` +npx vitest run tests/graph/experiment-57j33-tmp.test.mjs --reporter=verbose +``` + +(8 focused tests exercising deriveAnswerMeaningProfile and validateAnswerMeaningCompatibilityWithRawAnswer against all four captured strings.) + +### Result + +All 8 tests PASS. + +| Test | Expected | Actual | Status | +|---|---|---|---| +| Raw answer profiles as 'other' | `other` | `other` | PASS | +| Rejected Trial 2 profiles as 'conditional_tradeoff' | `conditional_tradeoff` | `conditional_tradeoff` | PASS | +| Comparison A profiles as 'other' | `other` | `other` | PASS | +| Comparison B profiles as 'other' | `other` | `other` | PASS | +| Validator rejects Trial 2 | error present | error present | PASS | +| Validator accepts comparison A | no errors | no errors | PASS | +| Validator accepts comparison B | no errors | no errors | PASS | +| Trigger: 'conditional' fires hasConditionalQualification | true for Trial 2, false for raw | confirmed | PASS | + +### Captured Trial 2 rejection reproduced: YES + +### Classification: **A — VALIDATOR CORRECT** + +### Why + +The validator correctly identifies that "The decision is conditional on..." introduces a `conditional_tradeoff` category where the raw answer only establishes `other`. The `conditional` keyword at line 2775 of `hasConditionalQualification()` fires because "decision is conditional on" contains the word "conditional". This pushes the meaning from a neutral information need into a protected reasoning category that implies default preference + exception qualification — which is indeed stronger than what the raw answer establishes. + +The key insight: this is not a subtle wording issue. The rejected Trial 2 string literally contains the word "conditional" which triggers a category detector in production code. The accepted comparisons A and B do not contain any trigger words and correctly remain classified as `other`. + +### What this establishes + +1. The validator's rejection of the captured Trial 2 meaning is **correct** — the meaning introduces a stronger reasoning category (`conditional_tradeoff`) where the raw answer only supports `other`. +2. The mechanism is the `hasConditionalQualification()` keyword detector (line 2775) firing on "conditional" in "decision is conditional on". +3. Both accepted comparison variants (A and B) remain correctly classified as `other` by the same detector. +4. The rejection does not involve cold-start graph variance or structural elements — it is purely a meaning-category mismatch at the validator gate. + +### What it does NOT establish + +1. Whether "conditional" is the ideal trigger word for `hasConditionalQualification()` in all contexts (this is about the existing boundary only). +2. Whether the raw answer's "Before deciding" should itself have triggered conditional semantics — that would require changing the detector, which is outside scope. +3. Generalisation to other answers or domains beyond this specific captured pair. +4. Whether the cold-start node variance (6→8 nodes) observed in 57J.32 affects proposal quality downstream — that is a separate investigation. + +### Temporary test removed: YES