191 lines
9.7 KiB
Markdown
191 lines
9.7 KiB
Markdown
# Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening
|
||
|
||
## Objective
|
||
|
||
Answer exactly: given the exact raw answer and exact rejected `userSupportedMeaning` captured in 57J.32 Trial 2, is the current validator correct to classify the proposal meaning as a stronger reasoning category than the user established?
|
||
|
||
This task addresses only the existing semantic contract — not cold-start graph variance, addedNodes/edges, or provenance/connectivity.
|
||
|
||
## Configured apparatus
|
||
|
||
- **Branch:** `feature/rejected-proposal-diagnostics-v0.16`
|
||
- **HEAD at experiment start:** `a00f7b1` — experiment: inspect rejected proposal live variance
|
||
- **Ollama calls made:** 0 (fully deterministic)
|
||
- **Production code changed:** NO
|
||
- **Tests permanently changed:** NO
|
||
|
||
## Fixed captured evidence
|
||
|
||
### Raw user answer
|
||
|
||
> "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||
|
||
### Rejected Trial 2 `userSupportedMeaning`
|
||
|
||
> "The decision is conditional on evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers."
|
||
|
||
### Accepted comparison A
|
||
|
||
> "The user requires direct evidence that projected office savings are realistic and that the relocation will not materially increase the loss of key engineers before making a decision."
|
||
|
||
### Accepted comparison B
|
||
|
||
> "The user requires concrete evidence verifying that projected office savings are realistic and confirming that key engineer attrition will not materially increase before deciding on the relocation."
|
||
|
||
## Part 1 — Classifier trace (deterministic, from production code)
|
||
|
||
### Raw answer profile
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| `category` | `other` |
|
||
| `resolutionGuidance` | `null` |
|
||
|
||
**Reasoning:** No uncertain, conditional, constraint, or priority trigger words fire. The text passes through all detection gates and reaches the default "other" category.
|
||
|
||
### Rejected Trial 2 profile
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| `category` | `conditional_tradeoff` |
|
||
| `resolutionGuidance` | `may_resolve` |
|
||
|
||
**Reasoning:** `hasConditionalQualification()` fires on the word "conditional" inside "decision is conditional on" (line 2775 of `lib/graph/apply-proposal.js`). This sets `conditionalPreferenceStructure = true`, which returns `conditional_tradeoff` before any other gate is reached.
|
||
|
||
### Accepted comparison A profile
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| `category` | `other` |
|
||
| `resolutionGuidance` | `null` |
|
||
|
||
**Reasoning:** No trigger words fire. "Requires" is not in the conditional qualification list. Passes to default "other".
|
||
|
||
### Accepted comparison B profile
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| `category` | `other` |
|
||
| `resolutionGuidance` | `null` |
|
||
|
||
**Reasoning:** Same as A — no trigger words fire. "Before deciding" does not match any conditional/uncertainty/constraint/priority gate. Reaches default "other".
|
||
|
||
## Part 2 — Exact rejection mechanism
|
||
|
||
### Function
|
||
|
||
`validateAnswerMeaningCompatibilityWithRawAnswer()` in `lib/graph/apply-proposal.js`, line 2932.
|
||
|
||
### Branch/condition
|
||
|
||
Lines 2982–2986:
|
||
```javascript
|
||
if (rawAnswerProfile.category === "other") {
|
||
if (supportedMeaningProfile.category !== "other") {
|
||
errors.push(
|
||
"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes.",
|
||
);
|
||
}
|
||
}
|
||
```
|
||
|
||
### Categories involved
|
||
|
||
- **Raw answer category:** `other` — no protective category signal detected
|
||
- **Rejected meaning category:** `conditional_tradeoff` — fired by `hasConditionalQualification()` matching "conditional" in "decision is conditional on"
|
||
|
||
### Why the proposed category is considered stronger
|
||
|
||
The validator's guard for unclassified ("other") answers works on a simple principle: if the raw answer establishes no specific reasoning category, and the extracted meaning lands in any protected category (uncertain, explicit_hard_constraint, relative_priority_only, conditional_tradeoff), that is treated as introducing a stronger reasoning structure than the user supplied.
|
||
|
||
The `conditional_tradeoff` category signals "there is a default position qualified by an exception condition" — which implies the user has a preference/constraint stance that can be overridden under specific circumstances. This is categorically stronger than a neutral information need ("I need evidence before deciding"), which the raw answer establishes.
|
||
|
||
## Part 3 — Human semantic comparison
|
||
|
||
### Raw answer establishes:
|
||
|
||
**A** (information needed before deciding) — YES
|
||
The raw answer explicitly states "Before deciding, I need evidence..." — this unambiguously establishes an information need prior to decision-making.
|
||
|
||
**B** (decision is conditional on satisfying that evidence) — Partially / borderline
|
||
"Before deciding" implies a temporal/priority relationship but does not assert conditionality of the *decision itself*. It reports the speaker's personal requirement rather than prescribing a property of "the decision."
|
||
|
||
**C** (hard veto/constraint) — NO
|
||
No hard-constraint language present.
|
||
|
||
**D** (explicit decision rule) — NO
|
||
No rule structure established.
|
||
|
||
**E** — Cannot distinguish A from B with full certainty; the strongest supported meaning is A.
|
||
|
||
### Rejected Trial 2 meaning: "The decision is conditional on..."
|
||
|
||
**Classification: SLIGHT STRENGTHENING → MATERIAL STRENGTHENING (borderline)**
|
||
|
||
"Before deciding, I need..." frames the condition as the *speaker's* requirement. "The decision is conditional on..." frames it as an impersonal property of the decision itself. The shift from personal information need to prescriptive decision structure is a real change — not merely a paraphrase. However, it stays within the same broad semantic domain (evidence-before-decision).
|
||
|
||
The stronger case for MATERIAL STRENGTHENING: In reasoning terms, "the decision requires X" can be operationalized as a hard gate on decision-making, whereas "I need X before deciding" is descriptive of intent. The validator's categorical treatment is therefore defensible.
|
||
|
||
### Accepted comparison A: "The user requires evidence..."
|
||
|
||
**Classification: SLIGHT STRENGTHENING**
|
||
|
||
More explicit about who holds the requirement ("the user"), more precise ("before making a decision"). Still within the same information-need semantic domain as the raw answer. Does not introduce conditionality of the decision itself — stays in `other`.
|
||
|
||
### Accepted comparison B: "The user requires concrete evidence verifying..."
|
||
|
||
**Classification: SLIGHT STRENGTHENING**
|
||
|
||
Uses "concrete" and "verifying/confirming" which are mild strengthening adjectives, but does not cross into any protected reasoning category. Stays in `other`.
|
||
|
||
## Part 4 — Deterministic reproduction
|
||
|
||
### Command
|
||
|
||
```
|
||
npx vitest run tests/graph/experiment-57j33-tmp.test.mjs --reporter=verbose
|
||
```
|
||
|
||
(8 focused tests exercising deriveAnswerMeaningProfile and validateAnswerMeaningCompatibilityWithRawAnswer against all four captured strings.)
|
||
|
||
### Result
|
||
|
||
All 8 tests PASS.
|
||
|
||
| Test | Expected | Actual | Status |
|
||
|---|---|---|---|
|
||
| Raw answer profiles as 'other' | `other` | `other` | PASS |
|
||
| Rejected Trial 2 profiles as 'conditional_tradeoff' | `conditional_tradeoff` | `conditional_tradeoff` | PASS |
|
||
| Comparison A profiles as 'other' | `other` | `other` | PASS |
|
||
| Comparison B profiles as 'other' | `other` | `other` | PASS |
|
||
| Validator rejects Trial 2 | error present | error present | PASS |
|
||
| Validator accepts comparison A | no errors | no errors | PASS |
|
||
| Validator accepts comparison B | no errors | no errors | PASS |
|
||
| Trigger: 'conditional' fires hasConditionalQualification | true for Trial 2, false for raw | confirmed | PASS |
|
||
|
||
### Captured Trial 2 rejection reproduced: YES
|
||
|
||
### Classification: **A — VALIDATOR CORRECT**
|
||
|
||
### Why
|
||
|
||
The validator correctly identifies that "The decision is conditional on..." introduces a `conditional_tradeoff` category where the raw answer only establishes `other`. The `conditional` keyword at line 2775 of `hasConditionalQualification()` fires because "decision is conditional on" contains the word "conditional". This pushes the meaning from a neutral information need into a protected reasoning category that implies default preference + exception qualification — which is indeed stronger than what the raw answer establishes.
|
||
|
||
The key insight: this is not a subtle wording issue. The rejected Trial 2 string literally contains the word "conditional" which triggers a category detector in production code. The accepted comparisons A and B do not contain any trigger words and correctly remain classified as `other`.
|
||
|
||
### What this establishes
|
||
|
||
1. The validator's rejection of the captured Trial 2 meaning is **correct** — the meaning introduces a stronger reasoning category (`conditional_tradeoff`) where the raw answer only supports `other`.
|
||
2. The mechanism is the `hasConditionalQualification()` keyword detector (line 2775) firing on "conditional" in "decision is conditional on".
|
||
3. Both accepted comparison variants (A and B) remain correctly classified as `other` by the same detector.
|
||
4. The rejection does not involve cold-start graph variance or structural elements — it is purely a meaning-category mismatch at the validator gate.
|
||
|
||
### What it does NOT establish
|
||
|
||
1. Whether "conditional" is the ideal trigger word for `hasConditionalQualification()` in all contexts (this is about the existing boundary only).
|
||
2. Whether the raw answer's "Before deciding" should itself have triggered conditional semantics — that would require changing the detector, which is outside scope.
|
||
3. Generalisation to other answers or domains beyond this specific captured pair.
|
||
4. Whether the cold-start node variance (6→8 nodes) observed in 57J.32 affects proposal quality downstream — that is a separate investigation.
|
||
|
||
### Temporary test removed: YES
|