robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: classify captured answer-meaning strengthening

Browse Source
This commit is contained in:
robbond
2026-08-11 10:19:19 +01:00
parent a00f7b170d
commit bda3abf893
2 changed files with 208 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+18
View File
@@ -397,3 +397,21 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
### Experiment 57J.30 — Proposal-Boundary Live Variance
**Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.
### Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening (Deterministic)
**Objective:** Determine whether the validator correctly rejected the captured Trial 2 `userSupportedMeaning` ("The decision is conditional on evidence...") as stronger than the raw answer ("Before deciding, I need evidence..."). Fully deterministic — zero Ollama calls.
**Method:** Replicated production `deriveAnswerMeaningProfile()` and `validateAnswerMeaningCompatibilityWithRawAnswer()` logic deterministically against all four captured strings (raw answer, rejected Trial 2, accepted comparisons A and B). Also ran existing apply-proposal tests (64 pass) and rejected-proposal-snapshot tests (7 pass).
**Key finding:** The rejected Trial 2 string contains the word "conditional" which triggers `hasConditionalQualification()` at line 2775 of `lib/graph/apply-proposal.js`, pushing it into the `conditional_tradeoff` category. Both accepted comparisons lack trigger words and correctly remain classified as `other`. The raw answer also profiles as `other`.
**Rejection mechanism:** `validateAnswerMeaningCompatibilityWithRawAnswer()` line 2982–2986: when raw answer = "other" and supported meaning ≠ "other", the validator adds the "stronger reasoning category" error. This is exactly what occurs with Trial 2 (other → conditional_tradeoff).
**Classification: A — VALIDATOR CORRECT.** The captured rejected meaning introduces `conditional_tradeoff` where the raw answer only establishes `other`. This is a real strengthening: "The decision is conditional on..." prescribes a requirement on the decision itself rather than reporting a personal information need. Both accepted comparison variants correctly remain in `other`.
**What this establishes:** The validator's rejection was correct for the captured pair. The mechanism is purely the keyword detector firing on "conditional" — not cold-start variance or structural elements.
**What it does NOT establish:** Whether "conditional" is the ideal trigger word across all contexts, whether "Before deciding" should itself have triggered conditional semantics, generalisation to other answers/domains, or whether cold-start node variance (57J.32) separately affects proposal quality.
Configured Ollama: none used. Production code changed: NO. Tests permanently changed: NO. Temporary test used and removed: YES.
docs/experiment-57j33.md
+190
View File
@@ -0,0 +1,190 @@
# Experiment 57J.33 — Classify Captured Answer-Meaning Strengthening
## Objective
Answer exactly: given the exact raw answer and exact rejected `userSupportedMeaning` captured in 57J.32 Trial 2, is the current validator correct to classify the proposal meaning as a stronger reasoning category than the user established?
This task addresses only the existing semantic contract — not cold-start graph variance, addedNodes/edges, or provenance/connectivity.
## Configured apparatus
- **Branch:** `feature/rejected-proposal-diagnostics-v0.16`
- **HEAD at experiment start:** `a00f7b1` — experiment: inspect rejected proposal live variance
- **Ollama calls made:** 0 (fully deterministic)
- **Production code changed:** NO
- **Tests permanently changed:** NO
## Fixed captured evidence
### Raw user answer
> "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
### Rejected Trial 2 `userSupportedMeaning`
> "The decision is conditional on evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers."
### Accepted comparison A
> "The user requires direct evidence that projected office savings are realistic and that the relocation will not materially increase the loss of key engineers before making a decision."
### Accepted comparison B
> "The user requires concrete evidence verifying that projected office savings are realistic and confirming that key engineer attrition will not materially increase before deciding on the relocation."
## Part 1 — Classifier trace (deterministic, from production code)
### Raw answer profile
| Field | Value |
|---|---|
| `category` | `other` |
| `resolutionGuidance` | `null` |
**Reasoning:** No uncertain, conditional, constraint, or priority trigger words fire. The text passes through all detection gates and reaches the default "other" category.
### Rejected Trial 2 profile
| Field | Value |
|---|---|
| `category` | `conditional_tradeoff` |
| `resolutionGuidance` | `may_resolve` |
**Reasoning:** `hasConditionalQualification()` fires on the word "conditional" inside "decision is conditional on" (line 2775 of `lib/graph/apply-proposal.js`). This sets `conditionalPreferenceStructure = true`, which returns `conditional_tradeoff` before any other gate is reached.
### Accepted comparison A profile
| Field | Value |
|---|---|
| `category` | `other` |
| `resolutionGuidance` | `null` |
**Reasoning:** No trigger words fire. "Requires" is not in the conditional qualification list. Passes to default "other".
### Accepted comparison B profile
| Field | Value |
|---|---|
| `category` | `other` |
| `resolutionGuidance` | `null` |
**Reasoning:** Same as A — no trigger words fire. "Before deciding" does not match any conditional/uncertainty/constraint/priority gate. Reaches default "other".
## Part 2 — Exact rejection mechanism
### Function
`validateAnswerMeaningCompatibilityWithRawAnswer()` in `lib/graph/apply-proposal.js`, line 2932.
### Branch/condition
Lines 2982–2986:
```javascript
if (rawAnswerProfile.category === "other") {
if (supportedMeaningProfile.category !== "other") {
errors.push(
"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes.",
);
}
}
```
### Categories involved
- **Raw answer category:** `other` — no protective category signal detected
- **Rejected meaning category:** `conditional_tradeoff` — fired by `hasConditionalQualification()` matching "conditional" in "decision is conditional on"
### Why the proposed category is considered stronger
The validator's guard for unclassified ("other") answers works on a simple principle: if the raw answer establishes no specific reasoning category, and the extracted meaning lands in any protected category (uncertain, explicit_hard_constraint, relative_priority_only, conditional_tradeoff), that is treated as introducing a stronger reasoning structure than the user supplied.
The `conditional_tradeoff` category signals "there is a default position qualified by an exception condition" — which implies the user has a preference/constraint stance that can be overridden under specific circumstances. This is categorically stronger than a neutral information need ("I need evidence before deciding"), which the raw answer establishes.
## Part 3 — Human semantic comparison
### Raw answer establishes:
**A** (information needed before deciding) — YES
The raw answer explicitly states "Before deciding, I need evidence..." — this unambiguously establishes an information need prior to decision-making.
**B** (decision is conditional on satisfying that evidence) — Partially / borderline
"Before deciding" implies a temporal/priority relationship but does not assert conditionality of the *decision itself*. It reports the speaker's personal requirement rather than prescribing a property of "the decision."
**C** (hard veto/constraint) — NO
No hard-constraint language present.
**D** (explicit decision rule) — NO
No rule structure established.
**E** — Cannot distinguish A from B with full certainty; the strongest supported meaning is A.
### Rejected Trial 2 meaning: "The decision is conditional on..."
**Classification: SLIGHT STRENGTHENING → MATERIAL STRENGTHENING (borderline)**
"Before deciding, I need..." frames the condition as the *speaker's* requirement. "The decision is conditional on..." frames it as an impersonal property of the decision itself. The shift from personal information need to prescriptive decision structure is a real change — not merely a paraphrase. However, it stays within the same broad semantic domain (evidence-before-decision).
The stronger case for MATERIAL STRENGTHENING: In reasoning terms, "the decision requires X" can be operationalized as a hard gate on decision-making, whereas "I need X before deciding" is descriptive of intent. The validator's categorical treatment is therefore defensible.
### Accepted comparison A: "The user requires evidence..."
**Classification: SLIGHT STRENGTHENING**
More explicit about who holds the requirement ("the user"), more precise ("before making a decision"). Still within the same information-need semantic domain as the raw answer. Does not introduce conditionality of the decision itself — stays in `other`.
### Accepted comparison B: "The user requires concrete evidence verifying..."
**Classification: SLIGHT STRENGTHENING**
Uses "concrete" and "verifying/confirming" which are mild strengthening adjectives, but does not cross into any protected reasoning category. Stays in `other`.
## Part 4 — Deterministic reproduction
### Command
```
npx vitest run tests/graph/experiment-57j33-tmp.test.mjs --reporter=verbose
```
(8 focused tests exercising deriveAnswerMeaningProfile and validateAnswerMeaningCompatibilityWithRawAnswer against all four captured strings.)
### Result
All 8 tests PASS.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Raw answer profiles as 'other' | `other` | `other` | PASS |
| Rejected Trial 2 profiles as 'conditional_tradeoff' | `conditional_tradeoff` | `conditional_tradeoff` | PASS |
| Comparison A profiles as 'other' | `other` | `other` | PASS |
| Comparison B profiles as 'other' | `other` | `other` | PASS |
| Validator rejects Trial 2 | error present | error present | PASS |
| Validator accepts comparison A | no errors | no errors | PASS |
| Validator accepts comparison B | no errors | no errors | PASS |
| Trigger: 'conditional' fires hasConditionalQualification | true for Trial 2, false for raw | confirmed | PASS |
### Captured Trial 2 rejection reproduced: YES
### Classification: **A — VALIDATOR CORRECT**
### Why
The validator correctly identifies that "The decision is conditional on..." introduces a `conditional_tradeoff` category where the raw answer only establishes `other`. The `conditional` keyword at line 2775 of `hasConditionalQualification()` fires because "decision is conditional on" contains the word "conditional". This pushes the meaning from a neutral information need into a protected reasoning category that implies default preference + exception qualification — which is indeed stronger than what the raw answer establishes.
The key insight: this is not a subtle wording issue. The rejected Trial 2 string literally contains the word "conditional" which triggers a category detector in production code. The accepted comparisons A and B do not contain any trigger words and correctly remain classified as `other`.
### What this establishes
1. The validator's rejection of the captured Trial 2 meaning is **correct** — the meaning introduces a stronger reasoning category (`conditional_tradeoff`) where the raw answer only supports `other`.
2. The mechanism is the `hasConditionalQualification()` keyword detector (line 2775) firing on "conditional" in "decision is conditional on".
3. Both accepted comparison variants (A and B) remain correctly classified as `other` by the same detector.
4. The rejection does not involve cold-start graph variance or structural elements — it is purely a meaning-category mismatch at the validator gate.
### What it does NOT establish
1. Whether "conditional" is the ideal trigger word for `hasConditionalQualification()` in all contexts (this is about the existing boundary only).
2. Whether the raw answer's "Before deciding" should itself have triggered conditional semantics — that would require changing the detector, which is outside scope.
3. Generalisation to other answers or domains beyond this specific captured pair.
4. Whether the cold-start node variance (6→8 nodes) observed in 57J.32 affects proposal quality downstream — that is a separate investigation.
### Temporary test removed: YES