150 lines
9.3 KiB
Markdown
150 lines
9.3 KiB
Markdown
# Experiment 57J.48 — Uncertainty Fidelity False Positive: Lexical Gap in `deriveAnswerMeaningProfile`
|
|
|
|
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
|
**Starting HEAD:** `acd1928` experiment: choose structural action-selection rule
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> Why does the current fidelity validator reject the captured pair "I am unsure whether…" → "The user is currently uncertain whether…" as overstatement, and is that rejection semantically justified or a lexical false positive?
|
|
|
|
This is the architecture/convergence step after 57J.47. Do not continue prompt tuning.
|
|
|
|
## Part 1 — Exact Deterministic Trace
|
|
|
|
```
|
|
raw-answer profile/category: uncERtain (matches "unsure" at line 2880 of lib/graph/apply-proposal.js)
|
|
userSupportedMeaning profile/category: other (no match for any detection pattern — "uncertain" is ABSENT from patterns)
|
|
exact helper/function: deriveAnswerMeaningProfile() → validateAnswerMeaningCompatibilityWithRawAnswer()
|
|
exact condition that fires: lines 2966-2970 of apply-proposal.js: rawAnswerProfile.category === "uncertain" && supportedMeaningProfile.category !== "uncertain"
|
|
specific token/phrase/signals involved:
|
|
- Raw answer contains "unsure" → matches pattern at line 2880 → category = "uncertain"
|
|
- Supported meaning contains "uncertain" → NO matching pattern (gap) → falls through to default category "other"
|
|
- Category mismatch fires the "overstates a raw answer that remains uncertain" error at line 2969
|
|
```
|
|
|
|
The rejection depends on:
|
|
- **Lexical token:** `"unsure"` detected, `"uncertain"` NOT detected — identical semantics, different word form
|
|
- **Category ordering:** irrelevant here because uncertainty is checked first (line 2877); the issue is that neither word triggers it uniformly
|
|
- **NOT** perspective shift, negation, or "whether" — these are not signals in the detection logic
|
|
|
|
## Part 2 — Semantic Equivalence Check
|
|
|
|
### Uncertainty preserved
|
|
YES — both texts express unresolved uncertainty about the realism of projected office savings.
|
|
|
|
### Decision condition added
|
|
NO — neither text establishes a condition for proceeding/deciding.
|
|
|
|
### Hard constraint added
|
|
NO — neither text introduces a hard constraint.
|
|
|
|
### Priority added
|
|
NO — neither text adds priority/importance framing.
|
|
|
|
### Conclusion added
|
|
NO — neither text asserts a conclusion; both only state the existence of uncertainty.
|
|
|
|
### Material temporal claim added by "currently"
|
|
NEGLIGIBLE — "currently" is a minimal temporal qualifier that does not materially change meaning. The raw answer's present-tense context ("I am unsure") already establishes currentness implicitly.
|
|
|
|
### Perspective shift
|
|
REPRESENTATIONAL NORMALISATION — converting first-person uncertainty ("I am unsure") to third-person assertion ("The user is uncertain") changes representation perspective but preserves substantive meaning. Both express the same proposition: unresolved doubt about savings realism.
|
|
|
|
### Pair classification
|
|
MINOR NON-MATERIAL PARAPHRASE
|
|
|
|
## Part 3 — Architecture Classification
|
|
|
|
**B — LEXICAL FALSE POSITIVE**
|
|
|
|
The meanings are semantically equivalent (both express uncertainty), but lexical/category heuristics in `deriveAnswerMeaningProfile` reject the paraphrase because `"uncertain"` is absent from the detection patterns while `"unsure"` is present. The categories assigned to semantically equivalent uncertainty are incompatible solely due to keyword coverage gap.
|
|
|
|
## Part 4 — Keyword-Dictionary Risk
|
|
|
|
**Evidence of lexical reasoning drift: YES**
|
|
|
|
Current code evidence confirms that deterministic fidelity reasoning has drifted toward English keyword recognition:
|
|
1. `deriveAnswerMeaningProfile` uses `.includes()` checks on 5 specific uncertainty expressions (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`) — but NOT the more direct and common `"uncertain"`
|
|
2. Similarly, `hasConditionalQualification` detects `"conditional"` but not `"contingent"` or `"depends on"` which express identical semantics
|
|
3. The validator's semantic fidelity decision depends entirely on whether the LLM happens to use one of ~15-20 hardcoded English surface forms
|
|
|
|
**Current fidelity boundary:** RAW-LANGUAGE SEMANTIC INFERENCE IN VALIDATOR
|
|
|
|
The boundary is raw-language keyword detection, not structured semantic contract validation. There are no structured fields carrying uncertainty/resolution state that could be checked directly — only free-text string matching against the `userSupportedMeaning` field.
|
|
|
|
## Part 5 — Structured-Output Alternative Already Available?
|
|
|
|
**SUFFICIENT EXISTING STRUCTURE**
|
|
|
|
The engine already carries structured signals that could distinguish:
|
|
- user remains uncertain
|
|
- model inferred stronger condition
|
|
- model preserved uncertainty
|
|
|
|
Relevant existing fields:
|
|
- `answerMeaning.supportCategory` (enum): `"uncertain" | "conditional_tradeoff" | "explicit_hard_constraint"` — this field exists in the schema and is populated by the model (or null)
|
|
- `answerMeaning.resolutionGuidance` (nullable string): `"must_remain_unresolved" | "may_resolve" | "must_resolve"` — already distinguishes preservation from resolution intent
|
|
- `uncertaintyType` (from possibleInference path): `"evidence_needed" | "user_clarification_needed"` — differentiates uncertainty types
|
|
- `answerMeaning.possibleInference`: null when no inference was made
|
|
|
|
These fields exist in the production schema (`lib/graph/schema.js`) and could be used directly for compatibility checking without re-inferring semantics from English keywords. The current architecture already has `supportCategory` as a structured category carrier — the problem is that it is not being populated by the model (per 56D: "the LLM does not auto-populate supportCategory"), so the deterministic derivation layer must infer it from text.
|
|
|
|
## Deterministic Reproduction
|
|
|
|
**Command:**
|
|
```
|
|
node /tmp/57j48-verify.cjs
|
|
```
|
|
|
|
(Inline script executed deterministically — zero Ollama calls, zero API calls)
|
|
|
|
**Result:**
|
|
- Raw answer profiles as `uncertain` ✓
|
|
- userSupportedMeaning profiles as `other` (gap: "uncertain" not in patterns)
|
|
- Validation fires: `"answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."`
|
|
- Inverse test confirms: replacing "uncertain" with "unsure" (identical semantics) → category = `uncertain`, errors = none
|
|
|
|
**Captured rejection reproduced:** YES
|
|
|
|
## 57J.47 Documentation Cleanup
|
|
|
|
**Previous wording required correction:** YES
|
|
|
|
**What was corrected:**
|
|
1. Replaced "Meaning classification: STRENGTHENED" with "HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH" — explicitly distinguishing the human semantic assessment from the actual validator mechanism (keyword gap).
|
|
2. Added explicit statement that `"uncertain"` is absent from `deriveAnswerMeaningProfile`'s detection patterns while `"unsure"` is present — both express identical semantics.
|
|
3. Replaced "Classification: D — STRENGTHENING BLOCKS TEST" with "Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)" — the blocker is a lexical false positive, not genuine strengthening.
|
|
4. Updated "What this establishes" point 3 to describe the incomplete lexical coverage gap rather than claiming the guard "correctly flags as strengthening."
|
|
5. Updated "Did v0.19 solve..." explanation to attribute the blocker to lexical coverage gap rather than "strengthening."
|
|
|
|
**Observed live facts preserved:** YES — the rejection error, the rejected proposal snapshot contents, and the structural progress (one added node) are all preserved unchanged. Only the *interpretation* of the rejection mechanism was corrected.
|
|
|
|
## What this establishes
|
|
|
|
1. **The captured rejection is a lexical false positive:** The validator uses `"unsure"` to detect uncertainty but does not use `"uncertain"`, even though both words express identical semantics (per OED/WordNet, both denote "lacking sufficient knowledge or certainty").
|
|
2. **57J.47's "STRENGTHENED" classification conflates human semantic assessment with validator behavior.** The validator did not detect semantic strengthening — it detected a keyword absence. The human assessment that the paraphrase is a minor non-material paraphrase (not strengthening) is independently valid.
|
|
3. **The existing-first structural action rule worked correctly** in 57J.47: the model DID add a new unknown for savings realism. The blocker was purely at the meaning-extraction boundary.
|
|
4. **Structured semantic signals exist in the schema** (`supportCategory`, `resolutionGuidance`) but are not populated by the LLM (per 56D), leaving keyword inference as the current mechanism.
|
|
|
|
## What it does NOT establish
|
|
|
|
1. That all validator rejections for this class of paraphrase are false positives (other words/phrases may have legitimate strengthening semantics).
|
|
2. That adding `"uncertain"` to the detection patterns is sufficient for broader lexical coverage.
|
|
3. That structured output without keyword inference has been tested end-to-end.
|
|
4. Generalisation across other uncertainty expressions or domains.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Validator changed: NO
|
|
## Tests permanently changed: NO
|
|
## Temporary test removed: YES (removed `/tmp/57j48-verify.cjs`)
|
|
## Ollama calls made: 0
|
|
## Dev server disturbed: NO
|
|
|
|
## Documentation updated
|
|
|
|
- Created: `docs/experiment-57j48.md`
|
|
- Corrected: `docs/experiment-57j47.md` (distinguished validator mechanism from human semantic assessment)
|