robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: diagnose uncertainty fidelity false positive

Browse Source
This commit is contained in:
robbond
2026-08-11 14:56:15 +01:00
parent 174e581c23
commit a2c790ee80
3 changed files with 199 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+43 -4
View File
@@ -846,18 +846,57 @@ Configured Ollama: none used. Production code changed: prompt + tests only. Dev
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
rejectedProposalSnapshot:
- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (STRENGTHENED — third-person assertion of mental state + temporal specificity "currently" beyond first-person "I am unsure")
- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (HUMAN ASSESSMENT: minor non-material paraphrase — see 57J.48; VALIDATOR: lexically rejected because "uncertain" absent from deriveAnswerMeaningProfile detection patterns)
- possibleInference: null
- addedNodes: [{id: "ns82kx9", kind: unknown, label: "Projected office savings from the relocation lack validation"}] (genuinely new — ADD NEW UNKNOWN structural action)
- addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: depends_on}]
- updatedNodes: [] | resolvedUnknownNodeIds: []
**Meaning classification:** STRENGTHENED (third-person assertion of user mental state + temporal specificity overstates first-person uncertainty)
**Meaning classification:** LEXICAL MISMATCH in deterministic derivation (human assessment: minor non-material paraphrase; validator mechanism: "uncertain" absent from deriveAnswerMeaningProfile patterns while raw answer's "unsure" is detected)
**Structural action:** ADD NEW UNKNOWN — a genuinely new unknown directly representing savings realism was created. This is the FIRST time in experiments 57J.36-47 that the model produced a meaningful structural mutation for this type of uncertainty answer.
**Classification: D — STRENGTHENING BLOCKS TEST.** Meaning is strengthened; the test cannot establish v0.19's effectiveness for the semantic-to-mutation boundary. However, structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is meaning extraction (first-person uncertainty → third-person assertion), not the existing-first rule.
**Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE).** The rejection was caused by `"uncertain"` being absent from `deriveAnswerMeaningProfile`'s uncertainty detection patterns (which include `"unsure"`, `"not sure"`, `"do not know"`, etc.), not by genuine semantic strengthening. Both words express identical uncertainty semantics. Structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is incomplete lexical coverage in the deterministic category model, not a meaning extraction defect.
**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: first-person uncertainty statements are being converted to third-person assertions, caught by proposal_compatibility as semantic strengthening. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule.
**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: semantically faithful paraphrases are being rejected because `deriveAnswerMeaningProfile` detects `"unsure"` but not `"uncertain"` (identical semantics). The guard's lexical gap causes false-positive "overstates a raw answer that remains uncertain" errors for semantically equivalent third-person uncertainty statements. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule. 57J.48 provides the full deterministic diagnosis.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. No prompt change during experiment. Harness restored to original scenario/answers.
### Experiment 57J.48 — Uncertainty Fidelity False Positive Diagnosis
**Objective:** Determine whether the captured rejection of "The user is currently uncertain..." → rejected as overstatement when raw answer was "I am unsure..." is semantically justified or a lexical false positive. Fully deterministic — zero Ollama calls.
**Captured pair:**
- Raw answer: `"I am unsure whether the projected office savings from the relocation are realistic."`
- userSupportedMeaning: `"The user is currently uncertain whether the projected office savings from the relocation are realistic."`
**Deterministic trace:**
1. `deriveAnswerMeaningProfile(raw)` → category=`uncertain` (matches pattern `"unsure"` at line 2880)
2. `deriveAnswerMeaningProfile(userSupportedMeaning)` → category=`other` (pattern gap: `"uncertain"` not detected)
3. `validateAnswerMeaningCompatibilityWithRawAnswer`: line 2966-2970 fires because `raw=uncertain` && `supported≠uncertain` → error: "overstates a raw answer that remains uncertain."
**Inverse test (same semantics, different word):** Using `"unsure"` instead of `"uncertain"` in userSupportedMeaning → category=`uncertain`, errors=none. Both words express identical uncertainty.
**Semantic equivalence check:**
- Uncertainty preserved: YES
- Decision condition added: NO
- Hard constraint added: NO
- Priority added: NO
- Conclusion added: NO
- "currently" material temporal claim: NEGLIGIBLE
- Perspective shift: REPRESENTATIONAL NORMALISATION (not substantive)
- Pair classification: MINOR NON-MATERIAL PARAPHRASE
**Architecture classification:** B — LEXICAL FALSE POSITIVE
The meanings are semantically equivalent; the rejection depends on `"unsure"` being in `deriveAnswerMeaningProfile`'s detection patterns while `"uncertain"` is not. Both denote "lacking sufficient knowledge or certainty."
**Keyword-dictionary risk: YES.** The validator's semantic fidelity decision depends entirely on whether the LLM uses one of ~15-20 hardcoded English surface forms (`"unsure"`, `"not sure"`, etc.) rather than equivalent terms (`"uncertain"`). This is raw-language keyword detection, not structured semantic contract validation.
**Existing structured signals: SUFFICIENT.** The schema already carries `supportCategory` (enum), `resolutionGuidance` (nullable string distinguishing "must_remain_unresolved"/"may_resolve"), and `possibleInference`. These fields exist in the production contract but are consistently null from the LLM (per 56D), forcing reliance on keyword inference.
**57J.47 documentation correction applied:** Previous wording stated meaning was "STRENGTHENED" as fact. Corrected to distinguish: human semantic assessment (minor non-material paraphrase) from validator mechanism (lexical gap in detection patterns). The actual rejection was a false positive caused by incomplete keyword coverage, not genuine semantic strengthening.
**Production code changed: NO. Prompt changed: NO. Validator changed: NO. Tests permanently changed: NO. Ollama calls: 0.**
Configured Ollama: none (zero live calls).
docs/experiment-57j47.md
+7 -9
View File
@@ -110,16 +110,14 @@ Selected question: null
userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic."
possibleInference: null
### Meaning classification: STRENGTHENED
### Meaning classification: HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH
Raw answer: "I am unsure whether the projected office savings from the relocation are realistic." (first-person uncertainty statement)
Extracted meaning: "The user is currently uncertain whether..." (third-person assertion about user's mental state + temporal specificity "currently")
The model converted first-person uncertainty ("I am unsure") into third-person assertion ("The user is currently uncertain"). This adds two elements beyond the raw answer:
1. **Perspective shift:** Asserting what "the user" feels/thinks, rather than preserving the first-person expression.
2. **Temporal specificity:** Adding "currently" — an explicit temporal qualifier not present in the raw answer.
**Validator rejection reason:** Lexical false positive. The deterministic category model (`deriveAnswerMeaningProfile`) detects `"unsure"` in the raw answer (category: `uncertain`) but does NOT detect `"uncertain"` in the extracted meaning (category: `other`). Both words express identical uncertainty semantics; `"uncertain"` is absent from the detection patterns (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`). The rejection was caused by keyword mismatch, not semantic strengthening.
This goes slightly beyond preserving "only uncertainty over whether projected office savings are realistic." The extracted meaning introduces a stronger claim (assertion about user mental state) that was not established by the raw answer.
**Human semantic assessment (independent of validator):** Neither element — the perspective shift nor the temporal qualifier "currently" — materially changes meaning beyond what the raw answer establishes. See 57J.48 for detailed deterministic analysis.
## STRUCTURAL PROPOSAL
@@ -140,21 +138,21 @@ A genuinely new unknown node (`ns82kx9`) was created, directly representing savi
The proposal added a new unknown node (with one depends_on edge to the state anchor) representing savings realism. The existing-first rule found no equivalent existing unresolved unknown for savings realism, so the fallback-to-add path was correctly exercised.
## Classification: D — STRENGTHENING BLOCKS TEST
## Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)
**Meaning is STRENGTHENED.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction is not purely faithful (third-person assertion + temporal specificity overstates the raw first-person uncertainty). Therefore this test does not establish v0.19's effectiveness for the semantic-to-mutation boundary.
**Meaning extraction produced a semantically equivalent paraphrase that was lexically rejected.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction used `"uncertain"` rather than `"unsure"` — identical semantics but absent from `deriveAnswerMeaningProfile`'s detection patterns, causing a category mismatch (`other` instead of `uncertain`) that triggered rejection. **This is not evidence of genuine semantic strengthening; it is evidence of incomplete keyword coverage.** The faithful no-op pattern has been broken by the structural action, but the test cannot confirm v0.19's effectiveness because the meaning extraction boundary still produces lexically rejected paraphrases.
**Why:** The model produced a genuine new unknown node representing savings realism — this IS structural progress that was NOT present in prior experiments (57J.36-45 all showed faithful no-ops or empty proposals). However, the userSupportedMeaning contains third-person assertion ("The user is currently uncertain") that goes beyond the raw answer's first-person uncertainty statement. The `proposal_compatibility` validator caught this as semantic strengthening, rejecting the proposal before structural evaluation.
**Did v0.19 solve the faithful semantic-to-mutation failure in this run:** NO
The test cannot determine whether v0.19 solves the boundary because meaning was strengthened, which blocks the test. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is the meaning extraction boundary, not the structural action selection.
The test cannot determine whether v0.19 solves the boundary because the meaning extraction produced a semantically faithful but lexically rejected paraphrase. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is an incomplete lexical coverage gap in `deriveAnswerMeaningProfile`, not a structural action selection failure.
## What this establishes:
1. **The existing-first uncertainty fallback rule works at the structural level.** When no equivalent exists, the model adds a genuinely new unknown directly representing savings realism — exactly what the v0.19 prompt was designed to produce.
2. **The faithful no-op is no longer the default output.** This run produced one added node and one added edge. Prior experiments (57J.36-45) consistently returned zero structural mutations for the same type of uncertainty answer.
3. **A new blocking issue emerges at the meaning extraction boundary:** first-person uncertainty statements ("I am unsure") are being converted to third-person assertions ("The user is currently uncertain"), which the semantic fidelity guard correctly flags as strengthening.
3. **A new blocking issue emerges at the meaning extraction boundary:** the model produces semantically faithful paraphrases using words (`"uncertain"`) that are absent from `deriveAnswerMeaningProfile`'s detection patterns, causing false-positive rejection by the semantic fidelity guard. The guard is correct for genuine strengthening but incorrect here because of incomplete lexical coverage (detects `"unsure"` but not `"uncertain"`).
## What it does NOT prove:
docs/experiment-57j48.md
+149
View File
@@ -0,0 +1,149 @@
# Experiment 57J.48 — Uncertainty Fidelity False Positive: Lexical Gap in `deriveAnswerMeaningProfile`
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
**Starting HEAD:** `acd1928` experiment: choose structural action-selection rule
## Objective
Answer exactly:
> Why does the current fidelity validator reject the captured pair "I am unsure whether…" → "The user is currently uncertain whether…" as overstatement, and is that rejection semantically justified or a lexical false positive?
This is the architecture/convergence step after 57J.47. Do not continue prompt tuning.
## Part 1 — Exact Deterministic Trace
```
raw-answer profile/category: uncERtain (matches "unsure" at line 2880 of lib/graph/apply-proposal.js)
userSupportedMeaning profile/category: other (no match for any detection pattern — "uncertain" is ABSENT from patterns)
exact helper/function: deriveAnswerMeaningProfile() → validateAnswerMeaningCompatibilityWithRawAnswer()
exact condition that fires: lines 2966-2970 of apply-proposal.js: rawAnswerProfile.category === "uncertain" && supportedMeaningProfile.category !== "uncertain"
specific token/phrase/signals involved:
- Raw answer contains "unsure" → matches pattern at line 2880 → category = "uncertain"
- Supported meaning contains "uncertain" → NO matching pattern (gap) → falls through to default category "other"
- Category mismatch fires the "overstates a raw answer that remains uncertain" error at line 2969
```
The rejection depends on:
- **Lexical token:** `"unsure"` detected, `"uncertain"` NOT detected — identical semantics, different word form
- **Category ordering:** irrelevant here because uncertainty is checked first (line 2877); the issue is that neither word triggers it uniformly
- **NOT** perspective shift, negation, or "whether" — these are not signals in the detection logic
## Part 2 — Semantic Equivalence Check
### Uncertainty preserved
YES — both texts express unresolved uncertainty about the realism of projected office savings.
### Decision condition added
NO — neither text establishes a condition for proceeding/deciding.
### Hard constraint added
NO — neither text introduces a hard constraint.
### Priority added
NO — neither text adds priority/importance framing.
### Conclusion added
NO — neither text asserts a conclusion; both only state the existence of uncertainty.
### Material temporal claim added by "currently"
NEGLIGIBLE — "currently" is a minimal temporal qualifier that does not materially change meaning. The raw answer's present-tense context ("I am unsure") already establishes currentness implicitly.
### Perspective shift
REPRESENTATIONAL NORMALISATION — converting first-person uncertainty ("I am unsure") to third-person assertion ("The user is uncertain") changes representation perspective but preserves substantive meaning. Both express the same proposition: unresolved doubt about savings realism.
### Pair classification
MINOR NON-MATERIAL PARAPHRASE
## Part 3 — Architecture Classification
**B — LEXICAL FALSE POSITIVE**
The meanings are semantically equivalent (both express uncertainty), but lexical/category heuristics in `deriveAnswerMeaningProfile` reject the paraphrase because `"uncertain"` is absent from the detection patterns while `"unsure"` is present. The categories assigned to semantically equivalent uncertainty are incompatible solely due to keyword coverage gap.
## Part 4 — Keyword-Dictionary Risk
**Evidence of lexical reasoning drift: YES**
Current code evidence confirms that deterministic fidelity reasoning has drifted toward English keyword recognition:
1. `deriveAnswerMeaningProfile` uses `.includes()` checks on 5 specific uncertainty expressions (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`) — but NOT the more direct and common `"uncertain"`
2. Similarly, `hasConditionalQualification` detects `"conditional"` but not `"contingent"` or `"depends on"` which express identical semantics
3. The validator's semantic fidelity decision depends entirely on whether the LLM happens to use one of ~15-20 hardcoded English surface forms
**Current fidelity boundary:** RAW-LANGUAGE SEMANTIC INFERENCE IN VALIDATOR
The boundary is raw-language keyword detection, not structured semantic contract validation. There are no structured fields carrying uncertainty/resolution state that could be checked directly — only free-text string matching against the `userSupportedMeaning` field.
## Part 5 — Structured-Output Alternative Already Available?
**SUFFICIENT EXISTING STRUCTURE**
The engine already carries structured signals that could distinguish:
- user remains uncertain
- model inferred stronger condition
- model preserved uncertainty
Relevant existing fields:
- `answerMeaning.supportCategory` (enum): `"uncertain" | "conditional_tradeoff" | "explicit_hard_constraint"` — this field exists in the schema and is populated by the model (or null)
- `answerMeaning.resolutionGuidance` (nullable string): `"must_remain_unresolved" | "may_resolve" | "must_resolve"` — already distinguishes preservation from resolution intent
- `uncertaintyType` (from possibleInference path): `"evidence_needed" | "user_clarification_needed"` — differentiates uncertainty types
- `answerMeaning.possibleInference`: null when no inference was made
These fields exist in the production schema (`lib/graph/schema.js`) and could be used directly for compatibility checking without re-inferring semantics from English keywords. The current architecture already has `supportCategory` as a structured category carrier — the problem is that it is not being populated by the model (per 56D: "the LLM does not auto-populate supportCategory"), so the deterministic derivation layer must infer it from text.
## Deterministic Reproduction
**Command:**
```
node /tmp/57j48-verify.cjs
```
(Inline script executed deterministically — zero Ollama calls, zero API calls)
**Result:**
- Raw answer profiles as `uncertain` ✓
- userSupportedMeaning profiles as `other` (gap: "uncertain" not in patterns)
- Validation fires: `"answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."`
- Inverse test confirms: replacing "uncertain" with "unsure" (identical semantics) → category = `uncertain`, errors = none
**Captured rejection reproduced:** YES
## 57J.47 Documentation Cleanup
**Previous wording required correction:** YES
**What was corrected:**
1. Replaced "Meaning classification: STRENGTHENED" with "HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH" — explicitly distinguishing the human semantic assessment from the actual validator mechanism (keyword gap).
2. Added explicit statement that `"uncertain"` is absent from `deriveAnswerMeaningProfile`'s detection patterns while `"unsure"` is present — both express identical semantics.
3. Replaced "Classification: D — STRENGTHENING BLOCKS TEST" with "Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)" — the blocker is a lexical false positive, not genuine strengthening.
4. Updated "What this establishes" point 3 to describe the incomplete lexical coverage gap rather than claiming the guard "correctly flags as strengthening."
5. Updated "Did v0.19 solve..." explanation to attribute the blocker to lexical coverage gap rather than "strengthening."
**Observed live facts preserved:** YES — the rejection error, the rejected proposal snapshot contents, and the structural progress (one added node) are all preserved unchanged. Only the *interpretation* of the rejection mechanism was corrected.
## What this establishes
1. **The captured rejection is a lexical false positive:** The validator uses `"unsure"` to detect uncertainty but does not use `"uncertain"`, even though both words express identical semantics (per OED/WordNet, both denote "lacking sufficient knowledge or certainty").
2. **57J.47's "STRENGTHENED" classification conflates human semantic assessment with validator behavior.** The validator did not detect semantic strengthening — it detected a keyword absence. The human assessment that the paraphrase is a minor non-material paraphrase (not strengthening) is independently valid.
3. **The existing-first structural action rule worked correctly** in 57J.47: the model DID add a new unknown for savings realism. The blocker was purely at the meaning-extraction boundary.
4. **Structured semantic signals exist in the schema** (`supportCategory`, `resolutionGuidance`) but are not populated by the LLM (per 56D), leaving keyword inference as the current mechanism.
## What it does NOT establish
1. That all validator rejections for this class of paraphrase are false positives (other words/phrases may have legitimate strengthening semantics).
2. That adding `"uncertain"` to the detection patterns is sufficient for broader lexical coverage.
3. That structured output without keyword inference has been tested end-to-end.
4. Generalisation across other uncertainty expressions or domains.
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Tests permanently changed: NO
## Temporary test removed: YES (removed `/tmp/57j48-verify.cjs`)
## Ollama calls made: 0
## Dev server disturbed: NO
## Documentation updated
- Created: `docs/experiment-57j48.md`
- Corrected: `docs/experiment-57j47.md` (distinguished validator mechanism from human semantic assessment)