experiment: choose structured fidelity migration
This commit is contained in:
@@ -924,3 +924,21 @@ Read-only architecture diagnosis of whether existing `answerMeaning` fields can
|
||||
**What it does NOT establish:** That structured output solves the trust problem in general (Pattern A has no independent verification). That a dedicated `uncertaintyType` field (evidence_needed / user_clarification_needed) is unnecessary — this exists only in test fixtures, not any production schema.
|
||||
|
||||
Full results in `docs/experiment-57j49.md`.
|
||||
|
||||
### Experiment 57J.50 — Structured Fidelity Migration Choice
|
||||
|
||||
**Classification: C — STRUCTURED PRIMARY + NON-LEXICAL CONSISTENCY**
|
||||
|
||||
Read-only design decision selecting the smallest safe migration path from keyword-based fidelity inference to structured semantic fields as primary authority.
|
||||
|
||||
**Decision rationale:** Option C eliminates lexical dictionary dependence for all protected categories where structured fields are populated, while preserving backwards compatibility through null-to-lexical fallback for legacy proposals. It retains model-trust safeguards via cross-field deterministic consistency checks (resolutionGuidance vs proposal state, enum validity) without requiring the validator to re-infer English semantics from raw text or invent new classifiers.
|
||||
|
||||
**Exact next implementation boundary:** One branch — `feature/structured-semantic-fidelity-v0.20`. Four file modifications:
|
||||
1. `lib/graph/prompt-builder.js` — rule 28 replacement (enforce + instruct) + output contract addition (enum value listing for the two fields);
|
||||
2. `lib/graph/schema.js` — `z.enum(Object.values(answerSupportCategory))` and `z.enum(Object.values(answerResolutionGuidance))` constraints on the two zod fields;
|
||||
3. `lib/graph/apply-proposal.js` — migrate `validateAnswerMeaningAlignment()` to read structured values first, lexical fallback for null legacy proposals only; add schema-invariant consistency checks;
|
||||
4. `tests/graph/apply-proposal.test.js` — nine regression tests matching the required list below.
|
||||
|
||||
No new taxonomy. No provider-specific code. No embeddings. No synonym lists. No new LLM call pattern.
|
||||
|
||||
Full results in `docs/experiment-57j50.md`.
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
# Experiment 57J.50 — Structured Fidelity Migration Choice
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
||||
**Starting HEAD:** `f330421` experiment: assess structured semantic fidelity boundary
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> What is the smallest safe production change that makes structured semantic fields the primary fidelity contract for the protected answer-meaning categories, without simply recreating the English keyword dictionary as a verifier?
|
||||
|
||||
This builds on 57J.49's finding: the existing `answerSupportCategory` and `answerResolutionGuidance` enums are fully defined but neither enforced in schema nor populated by the model. The prompt explicitly permits null. The validator re-infers semantics from `userSupportedMeaning` text via keyword detection.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Current Taxonomy (Verified from Production Code)
|
||||
|
||||
### supportCategory
|
||||
|
||||
```text
|
||||
Schema values: relative_priority_only | conditional_tradeoff | uncertain | explicit_hard_constraint | other
|
||||
Defined at: lib/graph/schema.js line 147-153 (answerSupportCategory object)
|
||||
|
||||
Schema form for field: z.string().min(1).nullable().optional() — FREE TEXT, NO ENUM CONSTRAINT
|
||||
Nullable: YES
|
||||
Optional: YES
|
||||
Populated by model in production: NEVER (confirmed by 56D)
|
||||
Prompt instruction: "supportCategory and resolutionGuidance are optional descriptive hints only; if you are unsure of the exact wording, leave them null rather than inventing rigid category labels." (prompt-builder.js line 119)
|
||||
```
|
||||
|
||||
### resolutionGuidance
|
||||
|
||||
```text
|
||||
Schema values: must_remain_unresolved | may_resolve | must_resolve
|
||||
Defined at: lib/graph/schema.js line 155-158 (answerResolutionGuidance object)
|
||||
|
||||
Schema form for field: z.string().min(1).nullable().optional() — FREE TEXT, NO ENUM CONSTRAINT
|
||||
Nullable: YES
|
||||
Optional: YES
|
||||
Populated by model in production: NEVER (same pattern as supportCategory)
|
||||
Prompt instruction: Same line 119 as supportCategory above.
|
||||
```
|
||||
|
||||
### Key observations
|
||||
|
||||
1. Both enums exist and cover all five protected categories and three resolution states. No new taxonomy needed.
|
||||
2. Both fields use `z.string()` not `z.enum()`. Other schema fields (kind, status, relationship, confidence) all use `z.enum(Object.values(...))` — this two is the only exception.
|
||||
3. The prompt does NOT list these enum values in the output contract section. It lists SituationKind, SituationStatus, SituationRelationship, and ConfidenceLevel but not answerSupportCategory or answerResolutionGuidance.
|
||||
4. The prompt explicitly tells the model to leave them null if unsure — this explains zero population in production.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Three Migration Options Evaluated
|
||||
|
||||
### OPTION A — POPULATE + ENUM-CONSTRAIN ONLY
|
||||
|
||||
Change prompt so model MUST populate structured fields when applicable. Change schema to enum-constrain values. Leave existing lexical fidelity validators unchanged and authoritative.
|
||||
|
||||
```text
|
||||
removes 57J.48 unsure/uncertain false positive: PARTIAL
|
||||
- Schema enforcement catches invalid values, preventing garbage categories from being processed
|
||||
- But validator STILL uses keyword detection as primary authority — the false positive mechanism (lexical gap) remains in place for any category not caught by schema validation
|
||||
|
||||
keyword-dictionary dependence: PRIMARY
|
||||
- Validator still runs deriveAnswerMeaningProfile() which is entirely keyword-driven
|
||||
- Structured fields only serve as pass-through; they don't control validator logic
|
||||
|
||||
model-trust risk: MEDIUM
|
||||
- Requires model to reliably populate structured fields (unproven across domains/runs)
|
||||
- If model populates wrong category, validator catches it via keywords — so model misclassification is partially guarded by keywords
|
||||
|
||||
backwards compatibility: HIGH RISK
|
||||
- Breaking change: if model fails to populate (which it has never done reliably), schema enum constraint will cause Zod parse failure at the boundary
|
||||
|
||||
new taxonomy required: NO
|
||||
schema change: YES — z.enum() on both fields + prompt listing of valid values
|
||||
validator change: MINIMAL — no structural logic change needed; validator remains keyword-driven
|
||||
new LLM call: NO
|
||||
provider-specific: NO
|
||||
```
|
||||
|
||||
### OPTION B — STRUCTURED PRIMARY + LEXICAL FALLBACK
|
||||
|
||||
Require and enum-constrain structured fields. When populated, use them as primary semantic profile. Only invoke lexical derivation when structured fields are null for backwards compatibility. Do not cross-check a populated structured category against keywords.
|
||||
|
||||
```text
|
||||
removes 57J.48 unsure/uncertain false positive: YES
|
||||
- The entire deriveAnswerMeaningProfile() path is bypassed when structured fields are populated; no keyword detection occurs
|
||||
- Model says "uncertain" → engine trusts it; no need for "unsure"/"uncertain" keyword in userSupportedMeaning
|
||||
|
||||
keyword-dictionary dependence: FALLBACK ONLY
|
||||
- Keywords only fire for null/legacy proposals (backwards compat)
|
||||
- No populated proposal triggers lexical inference
|
||||
|
||||
model-trust risk: MEDIUM-HIGH
|
||||
- If model populates supportCategory as "uncertain" but means something different, validator has no independent check against userSupportedMeaning text
|
||||
- Mitigated by schema enum constraint catching invalid values
|
||||
- The structured category IS the claim; the engine trusts the model's self-classification for populated cases
|
||||
|
||||
backwards compatibility: HIGH RISK (if model doesn't populate) / MEDIUM (with prompt enforcement)
|
||||
- Schema enum constraint will reject non-populated proposals on first production run after deployment
|
||||
- Migration requires model to learn new instruction immediately — unproven pattern
|
||||
|
||||
new taxonomy required: NO
|
||||
schema change: YES — z.enum() + MUST instruction in prompt + enum listing in output contract
|
||||
validator change: YES — migrate deriveAnswerMeaningProfile() consumer to read structured values first, fall back to keywords for null legacy
|
||||
new LLM call: NO
|
||||
provider-specific: NO
|
||||
```
|
||||
|
||||
### OPTION C — STRUCTURED PRIMARY + NON-LEXICAL CONSISTENCY
|
||||
|
||||
Require and enum-constrain structured fields. Use them as primary semantic profile. Replace lexical verification of protected categories with deterministic consistency checks over structured proposal state where possible. Retain raw-text lexical inference only for legacy/null proposals during migration. Do not invent a new semantic classifier.
|
||||
|
||||
```text
|
||||
removes 57J.48 unsure/uncertain false positive: YES
|
||||
- Structured category "uncertain" + resolutionGuidance "must_remain_unresolved" directly checked against proposal resolved state
|
||||
- No keyword detection in populated path
|
||||
|
||||
keyword-dictionary dependence: NONE (for populated proposals) / FALLBACK ONLY (legacy null)
|
||||
- Zero keyword patterns fire when structured fields are present
|
||||
- Keywords remain only for backwards compat with null legacy proposals
|
||||
|
||||
model-trust risk: LOW-MEDIUM
|
||||
- Model can still misclassify (e.g., "conditional_tradeoff" instead of "uncertain") — but cross-field consistency checks catch internal contradictions
|
||||
- Example: if model says "must_resolve" but proposal resolves nothing → detected as inconsistency
|
||||
- Schema enum constraint catches invalid values
|
||||
|
||||
backwards compatibility: MEDIUM (same migration risk as B regarding prompt compliance)
|
||||
- Same schema enforcement gap during transition — requires model to populate on first run
|
||||
- But the null fallback path preserves existing behavior for any legacy proposal with null fields
|
||||
|
||||
new taxonomy required: NO
|
||||
schema change: YES — z.enum() + MUST instruction + enum listing in output contract
|
||||
validator change: YES — migrate deriveAnswerMeaningProfile consumer; add cross-field consistency checks; retain lexical for null legacy only
|
||||
new LLM call: NO
|
||||
provider-specific: NO
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Trust-Boundary Checks (Non-Lexical)
|
||||
|
||||
The following deterministic checks are possible using ONLY existing structured fields and proposal state, WITHOUT re-reading English semantics:
|
||||
|
||||
### 1. `resolutionGuidance = must_remain_unresolved` while proposal resolves an unknown
|
||||
|
||||
**Classification:** VALID STRUCTURAL CONSISTENCY CHECK
|
||||
**Why:** This is a field-to-field contradiction check against proposal structural state (`resolvedUnknownNodeIds.length > 0` or `updatedNodes.some(n => n.newStatus === "resolved")`). No English semantic inference required. The resolution state and the resolved IDs are both structured values.
|
||||
|
||||
### 2. `resolutionGuidance = must_resolve` while proposal leaves targeted unknown unresolved
|
||||
|
||||
**Classification:** VALID STRUCTURAL CONSISTENCY CHECK
|
||||
**Why:** Same mechanism — if model claims a hard constraint that must resolve, but the proposal doesn't include the node in resolvedUnknownNodeIds or updatedNodes with newStatus=resolved, this is a detectable contradiction between structured claim and structured action. No English reading needed.
|
||||
|
||||
### 3. Invalid `supportCategory` value (not in enum)
|
||||
|
||||
**Classification:** VALID STRUCTURAL CONSISTENCY CHECK
|
||||
**Why:** Zod enum constraint catches this at schema parse time. Zero code change required beyond adding z.enum(). The check is purely structural — does the string value match one of the allowed enum strings?
|
||||
|
||||
### 4. Invalid `resolutionGuidance` value (not in enum)
|
||||
|
||||
**Classification:** VALID STRUCTURAL CONSISTENCY CHECK
|
||||
**Why:** Same mechanism as #3. Zod enum constraint at parse time.
|
||||
|
||||
### 5. `possibleInference` justifying graph mutation unsupported by `userSupportedMeaning`
|
||||
|
||||
**Classification:** NOT POSSIBLE WITH CURRENT STRUCTURE
|
||||
**Why:** `possibleInference` is a free-text nullable string. There is no structured linkage between it and any proposed mutation. The validator already does not consume possibleInference for any decision. Making it authoritative would require either (a) adding structural fields to anchor inference claims to specific nodes, or (b) reading English semantics — both violate the constraint.
|
||||
|
||||
### 6. Populated structured category contradicting raw answer's English wording
|
||||
|
||||
**Classification:** LEXICAL SEMANTIC RE-INFERENCE (if attempted)
|
||||
**Why:** Determining whether "The user is uncertain about X" contradicts a `supportCategory` of "explicit_hard_constraint" requires semantic comparison between the free-text meaning field and the structured category. This IS lexical semantic inference — it reads English to judge consistency. The check is valid as a concept but CANNOT be performed without semantic inference. We are explicitly choosing not to add this check in option C, accepting model-trust risk for misclassification in favor of eliminating dictionary dependence.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Transitional Null/Backwards-Compatibility Policy
|
||||
|
||||
### Options evaluated:
|
||||
|
||||
**Fallback A (Lexical fallback):** Use current lexical derivation only when structured fields are null.
|
||||
**Fallback B (Reject/null):** Reject proposals with null structured fields, require model to regenerate.
|
||||
**Fallback C (Allow + skip validation):** Allow null but skip protected semantic category validation entirely.
|
||||
|
||||
### Preferred transitional policy: Fallback A — LEXICAL FALLBACK for legacy null
|
||||
|
||||
**Why:**
|
||||
- **Fallback B is too harsh:** During migration, any proposal with null fields would fail. Given the model has never been instructed to populate these fields, the first production deployment would break all proposals immediately. No gradual transition path exists.
|
||||
- **Fallback C wastes the migration window:** If we skip validation entirely for null cases, there's no incremental enforcement during transition — it delays the problem with no intermediate signal of whether model compliance is improving.
|
||||
- **Fallback A preserves existing behavior while providing a clear migration signal:** All legacy proposals continue working. Any future proposal that populates structured fields gets structured-path processing. The team can monitor what percentage of proposals populate fields as prompt enforcement takes effect. If population reaches high reliability, the fallback path can be deprecated and eventually removed.
|
||||
|
||||
**Implementation detail:** The validator's null check is: `if (supportCategory === null || resolutionGuidance === null)` → fall through to existing lexical derivation path. This adds zero new error paths during migration and preserves all existing behavior until structured fields are reliably populated.
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Captured-Case Walkthrough
|
||||
|
||||
### Input
|
||||
|
||||
```text
|
||||
Raw:
|
||||
I am unsure whether the projected office savings from the relocation are realistic.
|
||||
|
||||
Structured model meaning:
|
||||
userSupportedMeaning = The user is currently uncertain whether the projected office savings from the relocation are realistic.
|
||||
supportCategory = uncertain
|
||||
resolutionGuidance = must_remain_unresolved
|
||||
possibleInference = null
|
||||
```
|
||||
|
||||
### Option C walkthrough
|
||||
|
||||
1. **Schema parse:** `supportCategory = "uncertain"` → matches answerSupportCategory enum (line 150). ✓ Valid.
|
||||
2. **Schema parse:** `resolutionGuidance = "must_remain_unresolved"` → matches answerResolutionGuidance enum (line 156). ✓ Valid.
|
||||
3. **Structured primary path triggered:** Both fields are populated → use them as primary semantic profile. Skip lexical derivation entirely.
|
||||
4. **Consistency check #1:** `resolutionGuidance = "must_remain_unresolved"`. If proposal contains `resolvedUnknownNodeIds.length > 0`, reject with structured inconsistency error. If resolved, the rejection is: "Proposal resolves an unknown even though answerMeaning resolutionGuidance is must_remain_unresolved." (same error message as current, but source is now structured field not keyword detection).
|
||||
5. **Consistency check #2:** No cross-field contradiction between supportCategory="uncertain" and resolutionGuidance="must_remain_unresolved". ✓ Valid.
|
||||
6. **No lexical fallback triggered:** Both fields populated → keywords never fire.
|
||||
|
||||
### Outcome
|
||||
|
||||
**ACCEPT (if no structural inconsistency with proposal)** or **REJECT (if proposal contradicts must_remain_unresolved)**
|
||||
|
||||
### Depends on "unsure" vs "uncertain" wording?
|
||||
|
||||
**NO.** The raw answer contains "unsure" which is irrelevant under Option C. The validator reads `supportCategory = "uncertain"` from the structured field, not from English keyword detection in userSupportedMeaning or the raw answer. Whether the prose uses "unsure" or "uncertain" has zero impact on the decision path.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
### **C — STRUCTURED PRIMARY + NON-LEXICAL CONSISTENCY**
|
||||
|
||||
### Why C over B:
|
||||
|
||||
1. **Retains model-trust safeguards.** Option B trusts the model's self-classification without any cross-field verification beyond enum validity. Option C adds deterministic consistency checks (resolutionGuidance vs proposal state) that catch internal contradictions — e.g., model says "must_remain_unresolved" but proposal resolves the unknown — without requiring semantic inference.
|
||||
2. **Same implementation complexity.** The cross-field consistency checks are structural comparisons (field values vs resolvedUnknownNodeIds/updatedNodes arrays), not new classifiers. Complexity is bounded and testable.
|
||||
3. **Eliminates all lexical inference for populated proposals.** Like Option B, but with the additional safety net of consistency checks during the model-trust window until population reliability is proven.
|
||||
|
||||
### Why C over A:
|
||||
|
||||
1. **Actually removes keyword dependence.** Option A keeps keywords as the primary authority — structured fields are never consumed by the validator logic. This preserves the false-positive mechanism (lexical coverage gaps) exactly as-is.
|
||||
2. **Structured fields control validation flow, not just pass through values.** In C, the presence of structured values determines which code path executes; in A, the validator always runs keywords and treats structured values as decorative metadata.
|
||||
|
||||
---
|
||||
|
||||
## Required Bounded Implementation Scope (if selected)
|
||||
|
||||
### New branch: `feature/structured-semantic-fidelity-v0.20`
|
||||
|
||||
#### File 1: `lib/graph/prompt-builder.js`
|
||||
- Replace rule 28 with a MUST instruction requiring population of both fields when the answer contains any supported meaning category
|
||||
- Add answerSupportCategory and answerResolutionGuidance values to the output contract section (using formatEnumValues helper)
|
||||
|
||||
#### File 2: `lib/graph/schema.js`
|
||||
- Change line 165: `supportCategory: z.string().min(1).nullable().optional()` → `supportCategory: z.enum(Object.values(answerSupportCategory)).nullable().optional()`
|
||||
- Change line 166: `resolutionGuidance: z.string().min(1).nullable().optional()` → `resolutionGuidance: z.enum(Object.values(answerResolutionGuidance)).nullable().optional()`
|
||||
|
||||
#### File 3: `lib/graph/apply-proposal.js`
|
||||
- Migrate the consumer in `validateAnswerMeaningAlignment()` to read structured values first (`proposal.answerMeaning.supportCategory` / `.resolutionGuidance`)
|
||||
- Add null check: if both fields are null, fall through to existing lexical derivation (deriveAnswerMeaningProfile) for backwards compatibility
|
||||
- When populated, use structured category as the primary signal driving guard logic
|
||||
- Add two consistency checks in the same function:
|
||||
- If resolutionGuidance = "must_remain_unresolved" AND resolvedUnknownNodeIds.length > 0 → reject with specific structured inconsistency message
|
||||
- (The existing check at line 3013 already does this via derived profile — replace that derivation source)
|
||||
|
||||
#### File 4: `tests/graph/apply-proposal.test.js`
|
||||
- Nine focused regression tests (listed below)
|
||||
|
||||
---
|
||||
|
||||
## Required Deterministic Regressions
|
||||
|
||||
1. **`unsure` raw + structured `uncertain` category does not false-reject.** The structured category is authority; the raw word "unsure" is irrelevant. A proposal with supportCategory="uncertain" should not be rejected based on whether the raw answer says "unsure" vs "not sure" vs "I don't know."
|
||||
|
||||
2. **Valid structured category accepted regardless of equivalent paraphrase wording.** Different paraphrases expressing the same semantic meaning (e.g., "unclear whether X is true" / "unsure about X" / "has doubts about X") should all map to the same structured category when populated, and produce identical validator outcomes.
|
||||
|
||||
3. **Invalid category rejected by schema.** A proposal with supportCategory="conditional_qualification" (the value that triggered 56A) fails Zod parse before reaching any validator logic.
|
||||
|
||||
4. **Invalid resolution guidance rejected by schema.** A proposal with resolutionGuidance="needs more nuance" fails Zod parse at the boundary.
|
||||
|
||||
5. **`must_remain_unresolved` cannot coexist with a resolution mutation.** If supportCategory="uncertain" and resolutionGuidance="must_remain_unresolved", a proposal that resolves the unknown is rejected by structured consistency check, not keyword detection.
|
||||
|
||||
6. **possibleInference cannot independently justify mutation.** possibleInference=null remains valid; if populated with "might be hard constraint" but supportCategory="conditional_tradeoff", the inconsistency check does NOT fire because possibleInference has no structured linkage to mutations. The existing non-usage is preserved.
|
||||
|
||||
7. **Null legacy structured fields follow the chosen transitional fallback (A).** When both fields are null, deriveAnswerMeaningProfile() fires as before. Existing test cases continue to pass without modification.
|
||||
|
||||
8. **Existing genuine conditional/hard-constraint protections remain represented through structured categories.** If supportCategory="conditional_tradeoff" and resolutionGuidance="may_resolve", a proposal that resolves the unknown without preserving conditional qualification in proposalText is rejected by structural consistency check (resolved + no qualification preserved). Similarly for explicit_hard_constraint with must_resolve.
|
||||
|
||||
9. **No new keyword/synonym rule added.** The implementation changes zero keyword detection patterns. All five categories and three resolution states are already in the enums; only enforcement path changes.
|
||||
|
||||
---
|
||||
|
||||
## What this intentionally leaves unresolved
|
||||
|
||||
1. **Model population reliability across domains/runs** — unproven whether model reliably populates structured fields under production constraints. This is the primary risk for Option C adoption.
|
||||
2. **The `uncertaintyType` gap** — evidence_needed vs user_clarification_needed distinction (from regression cases E/F) does not exist in any production schema. If this matters, it requires a future field addition.
|
||||
3. **Structured category ↔ English semantic alignment verification** — we accept that the model might misclassify (e.g., "conditional_tradeoff" when "uncertain" is correct). Cross-field consistency catches some contradictions but not wrong-category-with-compatible-text cases. This is the trust boundary of any structured-primary approach.
|
||||
4. **Prompt version increment** — changing rule 28 to a MUST requirement requires a prompt version bump, which cascades through all existing test fixtures that capture prompt versions.
|
||||
|
||||
---
|
||||
|
||||
## Convergence
|
||||
|
||||
This task terminates at concrete Option C selection and bounded implementation scope. No further diagnosis required.
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed: NO
|
||||
Validator changed: NO
|
||||
Schema changed: NO
|
||||
Tests changed: NO
|
||||
Ollama calls made: 0
|
||||
Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user