confidence-engine/docs/experiment-57j48.md

Experiment 57J.48 — Uncertainty Fidelity False Positive: Lexical Gap in deriveAnswerMeaningProfile

Branch: feature/semantic-to-mutation-contract-v0.19 Starting HEAD: acd1928 experiment: choose structural action-selection rule

Objective

Answer exactly:

Why does the current fidelity validator reject the captured pair "I am unsure whether…" → "The user is currently uncertain whether…" as overstatement, and is that rejection semantically justified or a lexical false positive?

This is the architecture/convergence step after 57J.47. Do not continue prompt tuning.

Part 1 — Exact Deterministic Trace

raw-answer profile/category: uncERtain (matches "unsure" at line 2880 of lib/graph/apply-proposal.js)
userSupportedMeaning profile/category: other (no match for any detection pattern — "uncertain" is ABSENT from patterns)
exact helper/function: deriveAnswerMeaningProfile() → validateAnswerMeaningCompatibilityWithRawAnswer()
exact condition that fires: lines 2966-2970 of apply-proposal.js: rawAnswerProfile.category === "uncertain" && supportedMeaningProfile.category !== "uncertain"
specific token/phrase/signals involved:
  - Raw answer contains "unsure" → matches pattern at line 2880 → category = "uncertain"
  - Supported meaning contains "uncertain" → NO matching pattern (gap) → falls through to default category "other"
  - Category mismatch fires the "overstates a raw answer that remains uncertain" error at line 2969

The rejection depends on:

• Lexical token: "unsure" detected, "uncertain" NOT detected — identical semantics, different word form
• Category ordering: irrelevant here because uncertainty is checked first (line 2877); the issue is that neither word triggers it uniformly
• NOT perspective shift, negation, or "whether" — these are not signals in the detection logic

Part 2 — Semantic Equivalence Check

Uncertainty preserved

YES — both texts express unresolved uncertainty about the realism of projected office savings.

Decision condition added

NO — neither text establishes a condition for proceeding/deciding.

Hard constraint added

NO — neither text introduces a hard constraint.

Priority added

NO — neither text adds priority/importance framing.

Conclusion added

NO — neither text asserts a conclusion; both only state the existence of uncertainty.

Material temporal claim added by "currently"

NEGLIGIBLE — "currently" is a minimal temporal qualifier that does not materially change meaning. The raw answer's present-tense context ("I am unsure") already establishes currentness implicitly.

Perspective shift

REPRESENTATIONAL NORMALISATION — converting first-person uncertainty ("I am unsure") to third-person assertion ("The user is uncertain") changes representation perspective but preserves substantive meaning. Both express the same proposition: unresolved doubt about savings realism.

Pair classification

MINOR NON-MATERIAL PARAPHRASE

Part 3 — Architecture Classification

B — LEXICAL FALSE POSITIVE

The meanings are semantically equivalent (both express uncertainty), but lexical/category heuristics in deriveAnswerMeaningProfile reject the paraphrase because "uncertain" is absent from the detection patterns while "unsure" is present. The categories assigned to semantically equivalent uncertainty are incompatible solely due to keyword coverage gap.

Part 4 — Keyword-Dictionary Risk

Evidence of lexical reasoning drift: YES

Current code evidence confirms that deterministic fidelity reasoning has drifted toward English keyword recognition:

1. deriveAnswerMeaningProfile uses .includes() checks on 5 specific uncertainty expressions ("not really sure" | "not sure" | "unsure" | "do not know" | "don't know") — but NOT the more direct and common "uncertain"
2. Similarly, hasConditionalQualification detects "conditional" but not "contingent" or "depends on" which express identical semantics
3. The validator's semantic fidelity decision depends entirely on whether the LLM happens to use one of ~15-20 hardcoded English surface forms

Current fidelity boundary: RAW-LANGUAGE SEMANTIC INFERENCE IN VALIDATOR

The boundary is raw-language keyword detection, not structured semantic contract validation. There are no structured fields carrying uncertainty/resolution state that could be checked directly — only free-text string matching against the userSupportedMeaning field.

Part 5 — Structured-Output Alternative Already Available?

SUFFICIENT EXISTING STRUCTURE

The engine already carries structured signals that could distinguish:

• user remains uncertain
• model inferred stronger condition
• model preserved uncertainty

Relevant existing fields:

• answerMeaning.supportCategory (enum): "uncertain" | "conditional_tradeoff" | "explicit_hard_constraint" — this field exists in the schema and is populated by the model (or null)
• answerMeaning.resolutionGuidance (nullable string): "must_remain_unresolved" | "may_resolve" | "must_resolve" — already distinguishes preservation from resolution intent
• uncertaintyType (from possibleInference path): "evidence_needed" | "user_clarification_needed" — differentiates uncertainty types
• answerMeaning.possibleInference: null when no inference was made

These fields exist in the production schema (lib/graph/schema.js) and could be used directly for compatibility checking without re-inferring semantics from English keywords. The current architecture already has supportCategory as a structured category carrier — the problem is that it is not being populated by the model (per 56D: "the LLM does not auto-populate supportCategory"), so the deterministic derivation layer must infer it from text.

Deterministic Reproduction

Command:

node /tmp/57j48-verify.cjs

(Inline script executed deterministically — zero Ollama calls, zero API calls)

Result:

• Raw answer profiles as uncertain ✓
• userSupportedMeaning profiles as other (gap: "uncertain" not in patterns)
• Validation fires: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
• Inverse test confirms: replacing "uncertain" with "unsure" (identical semantics) → category = uncertain, errors = none

Captured rejection reproduced: YES

57J.47 Documentation Cleanup

Previous wording required correction: YES

What was corrected:

1. Replaced "Meaning classification: STRENGTHENED" with "HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH" — explicitly distinguishing the human semantic assessment from the actual validator mechanism (keyword gap).
2. Added explicit statement that "uncertain" is absent from deriveAnswerMeaningProfile's detection patterns while "unsure" is present — both express identical semantics.
3. Replaced "Classification: D — STRENGTHENING BLOCKS TEST" with "Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)" — the blocker is a lexical false positive, not genuine strengthening.
4. Updated "What this establishes" point 3 to describe the incomplete lexical coverage gap rather than claiming the guard "correctly flags as strengthening."
5. Updated "Did v0.19 solve..." explanation to attribute the blocker to lexical coverage gap rather than "strengthening."

Observed live facts preserved: YES — the rejection error, the rejected proposal snapshot contents, and the structural progress (one added node) are all preserved unchanged. Only the interpretation of the rejection mechanism was corrected.

What this establishes

1. The captured rejection is a lexical false positive: The validator uses "unsure" to detect uncertainty but does not use "uncertain", even though both words express identical semantics (per OED/WordNet, both denote "lacking sufficient knowledge or certainty").
2. 57J.47's "STRENGTHENED" classification conflates human semantic assessment with validator behavior. The validator did not detect semantic strengthening — it detected a keyword absence. The human assessment that the paraphrase is a minor non-material paraphrase (not strengthening) is independently valid.
3. The existing-first structural action rule worked correctly in 57J.47: the model DID add a new unknown for savings realism. The blocker was purely at the meaning-extraction boundary.
4. Structured semantic signals exist in the schema (supportCategory, resolutionGuidance) but are not populated by the LLM (per 56D), leaving keyword inference as the current mechanism.

What it does NOT establish

1. That all validator rejections for this class of paraphrase are false positives (other words/phrases may have legitimate strengthening semantics).
2. That adding "uncertain" to the detection patterns is sufficient for broader lexical coverage.
3. That structured output without keyword inference has been tested end-to-end.
4. Generalisation across other uncertainty expressions or domains.

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Tests permanently changed: NO

Temporary test removed: YES (removed /tmp/57j48-verify.cjs)

Ollama calls made: 0

Dev server disturbed: NO

Documentation updated

• Created: docs/experiment-57j48.md
• Corrected: docs/experiment-57j47.md (distinguished validator mechanism from human semantic assessment)