diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 1f6bb40..84c0497 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -846,18 +846,57 @@ Configured Ollama: none used. Production code changed: prompt + tests only. Dev **UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain." rejectedProposalSnapshot: -- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (STRENGTHENED — third-person assertion of mental state + temporal specificity "currently" beyond first-person "I am unsure") +- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (HUMAN ASSESSMENT: minor non-material paraphrase — see 57J.48; VALIDATOR: lexically rejected because "uncertain" absent from deriveAnswerMeaningProfile detection patterns) - possibleInference: null - addedNodes: [{id: "ns82kx9", kind: unknown, label: "Projected office savings from the relocation lack validation"}] (genuinely new — ADD NEW UNKNOWN structural action) - addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: depends_on}] - updatedNodes: [] | resolvedUnknownNodeIds: [] -**Meaning classification:** STRENGTHENED (third-person assertion of user mental state + temporal specificity overstates first-person uncertainty) +**Meaning classification:** LEXICAL MISMATCH in deterministic derivation (human assessment: minor non-material paraphrase; validator mechanism: "uncertain" absent from deriveAnswerMeaningProfile patterns while raw answer's "unsure" is detected) **Structural action:** ADD NEW UNKNOWN — a genuinely new unknown directly representing savings realism was created. This is the FIRST time in experiments 57J.36-47 that the model produced a meaningful structural mutation for this type of uncertainty answer. -**Classification: D — STRENGTHENING BLOCKS TEST.** Meaning is strengthened; the test cannot establish v0.19's effectiveness for the semantic-to-mutation boundary. However, structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is meaning extraction (first-person uncertainty → third-person assertion), not the existing-first rule. +**Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE).** The rejection was caused by `"uncertain"` being absent from `deriveAnswerMeaningProfile`'s uncertainty detection patterns (which include `"unsure"`, `"not sure"`, `"do not know"`, etc.), not by genuine semantic strengthening. Both words express identical uncertainty semantics. Structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is incomplete lexical coverage in the deterministic category model, not a meaning extraction defect. -**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: first-person uncertainty statements are being converted to third-person assertions, caught by proposal_compatibility as semantic strengthening. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule. +**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: semantically faithful paraphrases are being rejected because `deriveAnswerMeaningProfile` detects `"unsure"` but not `"uncertain"` (identical semantics). The guard's lexical gap causes false-positive "overstates a raw answer that remains uncertain" errors for semantically equivalent third-person uncertainty statements. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule. 57J.48 provides the full deterministic diagnosis. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. No prompt change during experiment. Harness restored to original scenario/answers. + +### Experiment 57J.48 — Uncertainty Fidelity False Positive Diagnosis + +**Objective:** Determine whether the captured rejection of "The user is currently uncertain..." → rejected as overstatement when raw answer was "I am unsure..." is semantically justified or a lexical false positive. Fully deterministic — zero Ollama calls. + +**Captured pair:** +- Raw answer: `"I am unsure whether the projected office savings from the relocation are realistic."` +- userSupportedMeaning: `"The user is currently uncertain whether the projected office savings from the relocation are realistic."` + +**Deterministic trace:** +1. `deriveAnswerMeaningProfile(raw)` → category=`uncertain` (matches pattern `"unsure"` at line 2880) +2. `deriveAnswerMeaningProfile(userSupportedMeaning)` → category=`other` (pattern gap: `"uncertain"` not detected) +3. `validateAnswerMeaningCompatibilityWithRawAnswer`: line 2966-2970 fires because `raw=uncertain` && `supported≠uncertain` → error: "overstates a raw answer that remains uncertain." + +**Inverse test (same semantics, different word):** Using `"unsure"` instead of `"uncertain"` in userSupportedMeaning → category=`uncertain`, errors=none. Both words express identical uncertainty. + +**Semantic equivalence check:** +- Uncertainty preserved: YES +- Decision condition added: NO +- Hard constraint added: NO +- Priority added: NO +- Conclusion added: NO +- "currently" material temporal claim: NEGLIGIBLE +- Perspective shift: REPRESENTATIONAL NORMALISATION (not substantive) +- Pair classification: MINOR NON-MATERIAL PARAPHRASE + +**Architecture classification:** B — LEXICAL FALSE POSITIVE + +The meanings are semantically equivalent; the rejection depends on `"unsure"` being in `deriveAnswerMeaningProfile`'s detection patterns while `"uncertain"` is not. Both denote "lacking sufficient knowledge or certainty." + +**Keyword-dictionary risk: YES.** The validator's semantic fidelity decision depends entirely on whether the LLM uses one of ~15-20 hardcoded English surface forms (`"unsure"`, `"not sure"`, etc.) rather than equivalent terms (`"uncertain"`). This is raw-language keyword detection, not structured semantic contract validation. + +**Existing structured signals: SUFFICIENT.** The schema already carries `supportCategory` (enum), `resolutionGuidance` (nullable string distinguishing "must_remain_unresolved"/"may_resolve"), and `possibleInference`. These fields exist in the production contract but are consistently null from the LLM (per 56D), forcing reliance on keyword inference. + +**57J.47 documentation correction applied:** Previous wording stated meaning was "STRENGTHENED" as fact. Corrected to distinguish: human semantic assessment (minor non-material paraphrase) from validator mechanism (lexical gap in detection patterns). The actual rejection was a false positive caused by incomplete keyword coverage, not genuine semantic strengthening. + +**Production code changed: NO. Prompt changed: NO. Validator changed: NO. Tests permanently changed: NO. Ollama calls: 0.** + +Configured Ollama: none (zero live calls). diff --git a/docs/experiment-57j47.md b/docs/experiment-57j47.md index b561218..8087278 100644 --- a/docs/experiment-57j47.md +++ b/docs/experiment-57j47.md @@ -110,16 +110,14 @@ Selected question: null userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." possibleInference: null -### Meaning classification: STRENGTHENED +### Meaning classification: HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH Raw answer: "I am unsure whether the projected office savings from the relocation are realistic." (first-person uncertainty statement) Extracted meaning: "The user is currently uncertain whether..." (third-person assertion about user's mental state + temporal specificity "currently") -The model converted first-person uncertainty ("I am unsure") into third-person assertion ("The user is currently uncertain"). This adds two elements beyond the raw answer: -1. **Perspective shift:** Asserting what "the user" feels/thinks, rather than preserving the first-person expression. -2. **Temporal specificity:** Adding "currently" — an explicit temporal qualifier not present in the raw answer. +**Validator rejection reason:** Lexical false positive. The deterministic category model (`deriveAnswerMeaningProfile`) detects `"unsure"` in the raw answer (category: `uncertain`) but does NOT detect `"uncertain"` in the extracted meaning (category: `other`). Both words express identical uncertainty semantics; `"uncertain"` is absent from the detection patterns (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`). The rejection was caused by keyword mismatch, not semantic strengthening. -This goes slightly beyond preserving "only uncertainty over whether projected office savings are realistic." The extracted meaning introduces a stronger claim (assertion about user mental state) that was not established by the raw answer. +**Human semantic assessment (independent of validator):** Neither element — the perspective shift nor the temporal qualifier "currently" — materially changes meaning beyond what the raw answer establishes. See 57J.48 for detailed deterministic analysis. ## STRUCTURAL PROPOSAL @@ -140,21 +138,21 @@ A genuinely new unknown node (`ns82kx9`) was created, directly representing savi The proposal added a new unknown node (with one depends_on edge to the state anchor) representing savings realism. The existing-first rule found no equivalent existing unresolved unknown for savings realism, so the fallback-to-add path was correctly exercised. -## Classification: D — STRENGTHENING BLOCKS TEST +## Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE) -**Meaning is STRENGTHENED.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction is not purely faithful (third-person assertion + temporal specificity overstates the raw first-person uncertainty). Therefore this test does not establish v0.19's effectiveness for the semantic-to-mutation boundary. +**Meaning extraction produced a semantically equivalent paraphrase that was lexically rejected.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction used `"uncertain"` rather than `"unsure"` — identical semantics but absent from `deriveAnswerMeaningProfile`'s detection patterns, causing a category mismatch (`other` instead of `uncertain`) that triggered rejection. **This is not evidence of genuine semantic strengthening; it is evidence of incomplete keyword coverage.** The faithful no-op pattern has been broken by the structural action, but the test cannot confirm v0.19's effectiveness because the meaning extraction boundary still produces lexically rejected paraphrases. **Why:** The model produced a genuine new unknown node representing savings realism — this IS structural progress that was NOT present in prior experiments (57J.36-45 all showed faithful no-ops or empty proposals). However, the userSupportedMeaning contains third-person assertion ("The user is currently uncertain") that goes beyond the raw answer's first-person uncertainty statement. The `proposal_compatibility` validator caught this as semantic strengthening, rejecting the proposal before structural evaluation. **Did v0.19 solve the faithful semantic-to-mutation failure in this run:** NO -The test cannot determine whether v0.19 solves the boundary because meaning was strengthened, which blocks the test. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is the meaning extraction boundary, not the structural action selection. +The test cannot determine whether v0.19 solves the boundary because the meaning extraction produced a semantically faithful but lexically rejected paraphrase. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is an incomplete lexical coverage gap in `deriveAnswerMeaningProfile`, not a structural action selection failure. ## What this establishes: 1. **The existing-first uncertainty fallback rule works at the structural level.** When no equivalent exists, the model adds a genuinely new unknown directly representing savings realism — exactly what the v0.19 prompt was designed to produce. 2. **The faithful no-op is no longer the default output.** This run produced one added node and one added edge. Prior experiments (57J.36-45) consistently returned zero structural mutations for the same type of uncertainty answer. -3. **A new blocking issue emerges at the meaning extraction boundary:** first-person uncertainty statements ("I am unsure") are being converted to third-person assertions ("The user is currently uncertain"), which the semantic fidelity guard correctly flags as strengthening. +3. **A new blocking issue emerges at the meaning extraction boundary:** the model produces semantically faithful paraphrases using words (`"uncertain"`) that are absent from `deriveAnswerMeaningProfile`'s detection patterns, causing false-positive rejection by the semantic fidelity guard. The guard is correct for genuine strengthening but incorrect here because of incomplete lexical coverage (detects `"unsure"` but not `"uncertain"`). ## What it does NOT prove: diff --git a/docs/experiment-57j48.md b/docs/experiment-57j48.md new file mode 100644 index 0000000..99e5c5b --- /dev/null +++ b/docs/experiment-57j48.md @@ -0,0 +1,149 @@ +# Experiment 57J.48 — Uncertainty Fidelity False Positive: Lexical Gap in `deriveAnswerMeaningProfile` + +**Branch:** `feature/semantic-to-mutation-contract-v0.19` +**Starting HEAD:** `acd1928` experiment: choose structural action-selection rule + +## Objective + +Answer exactly: + +> Why does the current fidelity validator reject the captured pair "I am unsure whether…" → "The user is currently uncertain whether…" as overstatement, and is that rejection semantically justified or a lexical false positive? + +This is the architecture/convergence step after 57J.47. Do not continue prompt tuning. + +## Part 1 — Exact Deterministic Trace + +``` +raw-answer profile/category: uncERtain (matches "unsure" at line 2880 of lib/graph/apply-proposal.js) +userSupportedMeaning profile/category: other (no match for any detection pattern — "uncertain" is ABSENT from patterns) +exact helper/function: deriveAnswerMeaningProfile() → validateAnswerMeaningCompatibilityWithRawAnswer() +exact condition that fires: lines 2966-2970 of apply-proposal.js: rawAnswerProfile.category === "uncertain" && supportedMeaningProfile.category !== "uncertain" +specific token/phrase/signals involved: + - Raw answer contains "unsure" → matches pattern at line 2880 → category = "uncertain" + - Supported meaning contains "uncertain" → NO matching pattern (gap) → falls through to default category "other" + - Category mismatch fires the "overstates a raw answer that remains uncertain" error at line 2969 +``` + +The rejection depends on: +- **Lexical token:** `"unsure"` detected, `"uncertain"` NOT detected — identical semantics, different word form +- **Category ordering:** irrelevant here because uncertainty is checked first (line 2877); the issue is that neither word triggers it uniformly +- **NOT** perspective shift, negation, or "whether" — these are not signals in the detection logic + +## Part 2 — Semantic Equivalence Check + +### Uncertainty preserved +YES — both texts express unresolved uncertainty about the realism of projected office savings. + +### Decision condition added +NO — neither text establishes a condition for proceeding/deciding. + +### Hard constraint added +NO — neither text introduces a hard constraint. + +### Priority added +NO — neither text adds priority/importance framing. + +### Conclusion added +NO — neither text asserts a conclusion; both only state the existence of uncertainty. + +### Material temporal claim added by "currently" +NEGLIGIBLE — "currently" is a minimal temporal qualifier that does not materially change meaning. The raw answer's present-tense context ("I am unsure") already establishes currentness implicitly. + +### Perspective shift +REPRESENTATIONAL NORMALISATION — converting first-person uncertainty ("I am unsure") to third-person assertion ("The user is uncertain") changes representation perspective but preserves substantive meaning. Both express the same proposition: unresolved doubt about savings realism. + +### Pair classification +MINOR NON-MATERIAL PARAPHRASE + +## Part 3 — Architecture Classification + +**B — LEXICAL FALSE POSITIVE** + +The meanings are semantically equivalent (both express uncertainty), but lexical/category heuristics in `deriveAnswerMeaningProfile` reject the paraphrase because `"uncertain"` is absent from the detection patterns while `"unsure"` is present. The categories assigned to semantically equivalent uncertainty are incompatible solely due to keyword coverage gap. + +## Part 4 — Keyword-Dictionary Risk + +**Evidence of lexical reasoning drift: YES** + +Current code evidence confirms that deterministic fidelity reasoning has drifted toward English keyword recognition: +1. `deriveAnswerMeaningProfile` uses `.includes()` checks on 5 specific uncertainty expressions (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`) — but NOT the more direct and common `"uncertain"` +2. Similarly, `hasConditionalQualification` detects `"conditional"` but not `"contingent"` or `"depends on"` which express identical semantics +3. The validator's semantic fidelity decision depends entirely on whether the LLM happens to use one of ~15-20 hardcoded English surface forms + +**Current fidelity boundary:** RAW-LANGUAGE SEMANTIC INFERENCE IN VALIDATOR + +The boundary is raw-language keyword detection, not structured semantic contract validation. There are no structured fields carrying uncertainty/resolution state that could be checked directly — only free-text string matching against the `userSupportedMeaning` field. + +## Part 5 — Structured-Output Alternative Already Available? + +**SUFFICIENT EXISTING STRUCTURE** + +The engine already carries structured signals that could distinguish: +- user remains uncertain +- model inferred stronger condition +- model preserved uncertainty + +Relevant existing fields: +- `answerMeaning.supportCategory` (enum): `"uncertain" | "conditional_tradeoff" | "explicit_hard_constraint"` — this field exists in the schema and is populated by the model (or null) +- `answerMeaning.resolutionGuidance` (nullable string): `"must_remain_unresolved" | "may_resolve" | "must_resolve"` — already distinguishes preservation from resolution intent +- `uncertaintyType` (from possibleInference path): `"evidence_needed" | "user_clarification_needed"` — differentiates uncertainty types +- `answerMeaning.possibleInference`: null when no inference was made + +These fields exist in the production schema (`lib/graph/schema.js`) and could be used directly for compatibility checking without re-inferring semantics from English keywords. The current architecture already has `supportCategory` as a structured category carrier — the problem is that it is not being populated by the model (per 56D: "the LLM does not auto-populate supportCategory"), so the deterministic derivation layer must infer it from text. + +## Deterministic Reproduction + +**Command:** +``` +node /tmp/57j48-verify.cjs +``` + +(Inline script executed deterministically — zero Ollama calls, zero API calls) + +**Result:** +- Raw answer profiles as `uncertain` ✓ +- userSupportedMeaning profiles as `other` (gap: "uncertain" not in patterns) +- Validation fires: `"answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."` +- Inverse test confirms: replacing "uncertain" with "unsure" (identical semantics) → category = `uncertain`, errors = none + +**Captured rejection reproduced:** YES + +## 57J.47 Documentation Cleanup + +**Previous wording required correction:** YES + +**What was corrected:** +1. Replaced "Meaning classification: STRENGTHENED" with "HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH" — explicitly distinguishing the human semantic assessment from the actual validator mechanism (keyword gap). +2. Added explicit statement that `"uncertain"` is absent from `deriveAnswerMeaningProfile`'s detection patterns while `"unsure"` is present — both express identical semantics. +3. Replaced "Classification: D — STRENGTHENING BLOCKS TEST" with "Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)" — the blocker is a lexical false positive, not genuine strengthening. +4. Updated "What this establishes" point 3 to describe the incomplete lexical coverage gap rather than claiming the guard "correctly flags as strengthening." +5. Updated "Did v0.19 solve..." explanation to attribute the blocker to lexical coverage gap rather than "strengthening." + +**Observed live facts preserved:** YES — the rejection error, the rejected proposal snapshot contents, and the structural progress (one added node) are all preserved unchanged. Only the *interpretation* of the rejection mechanism was corrected. + +## What this establishes + +1. **The captured rejection is a lexical false positive:** The validator uses `"unsure"` to detect uncertainty but does not use `"uncertain"`, even though both words express identical semantics (per OED/WordNet, both denote "lacking sufficient knowledge or certainty"). +2. **57J.47's "STRENGTHENED" classification conflates human semantic assessment with validator behavior.** The validator did not detect semantic strengthening — it detected a keyword absence. The human assessment that the paraphrase is a minor non-material paraphrase (not strengthening) is independently valid. +3. **The existing-first structural action rule worked correctly** in 57J.47: the model DID add a new unknown for savings realism. The blocker was purely at the meaning-extraction boundary. +4. **Structured semantic signals exist in the schema** (`supportCategory`, `resolutionGuidance`) but are not populated by the LLM (per 56D), leaving keyword inference as the current mechanism. + +## What it does NOT establish + +1. That all validator rejections for this class of paraphrase are false positives (other words/phrases may have legitimate strengthening semantics). +2. That adding `"uncertain"` to the detection patterns is sufficient for broader lexical coverage. +3. That structured output without keyword inference has been tested end-to-end. +4. Generalisation across other uncertainty expressions or domains. + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Tests permanently changed: NO +## Temporary test removed: YES (removed `/tmp/57j48-verify.cjs`) +## Ollama calls made: 0 +## Dev server disturbed: NO + +## Documentation updated + +- Created: `docs/experiment-57j48.md` +- Corrected: `docs/experiment-57j47.md` (distinguished validator mechanism from human semantic assessment)