docs: record structured semantic fidelity implementation
This commit is contained in:
@@ -0,0 +1,145 @@
|
||||
# Experiment 57J.51 — Structured Semantic Fidelity Implementation
|
||||
|
||||
**Branch:** `feature/structured-semantic-fidelity-v0.20`
|
||||
**Starting HEAD:** `b6a232ff6f56b5f1af49d94bb2881190b5bf8345`
|
||||
**Production commit:** `7d06cd3c473cee64c2c371c1e1af1c466cdc32dd`
|
||||
|
||||
## Objective
|
||||
|
||||
Implement Option C from Experiment 57J.50:
|
||||
|
||||
> Use existing structured semantic fields (`supportCategory`, `resolutionGuidance`) as the primary fidelity contract when populated, enforce their allowed enum values, validate only structured cross-field consistency, and retain current lexical derivation only as a temporary fallback when those fields are null.
|
||||
|
||||
## Scope Implemented
|
||||
|
||||
### 1. Schema
|
||||
|
||||
`lib/graph/schema.js`
|
||||
|
||||
- Constrained `answerMeaning.supportCategory` to `z.enum(Object.values(answerSupportCategory)).nullable().optional()`;
|
||||
- Constrained `answerMeaning.resolutionGuidance` to `z.enum(Object.values(answerResolutionGuidance)).nullable().optional()`;
|
||||
- Preserved transitional nullability on both fields;
|
||||
- Reused existing enum constants — no new taxonomy added.
|
||||
|
||||
### 2. Prompt
|
||||
|
||||
`lib/graph/prompt-builder.js`
|
||||
|
||||
- Exposed allowed values for both structured semantic fields in the output contract;
|
||||
- Replaced the old “optional descriptive hints only” instruction with structured population guidance;
|
||||
- Instructed the model to:
|
||||
- populate `supportCategory` whenever the answer fits an existing category,
|
||||
- use `other` when none of the protected categories applies,
|
||||
- avoid leaving `supportCategory` null merely because wording is uncertain,
|
||||
- populate `resolutionGuidance` when one of the existing resolution states genuinely applies,
|
||||
- keep `resolutionGuidance` null only when no existing state actually applies;
|
||||
- Used the existing `formatEnumValues()` helper;
|
||||
- Added no provider-specific wording.
|
||||
|
||||
### 3. Validator — structured first
|
||||
|
||||
`lib/graph/apply-proposal.js`
|
||||
|
||||
- Added `getAnswerMeaningProfile(answerMeaning)` to unify:
|
||||
- structured `supportCategory` / `resolutionGuidance` when populated,
|
||||
- lexical derivation only when those structured fields are null;
|
||||
- Updated `validateAnswerMeaningCompatibilityWithRawAnswer()` so populated structured semantic fields bypass raw-text lexical category verification entirely;
|
||||
- Updated `validateAnswerMeaningAlignment()` so:
|
||||
- structured fields are authoritative when populated,
|
||||
- lexical fallback remains active only for legacy null cases.
|
||||
|
||||
### 4. Non-lexical consistency
|
||||
|
||||
Implemented one deterministic structured consistency check now:
|
||||
|
||||
- `resolutionGuidance = must_remain_unresolved` + proposal resolves an unknown → reject with:
|
||||
- `Proposal resolves an unknown even though answerMeaning.resolutionGuidance is must_remain_unresolved.`
|
||||
|
||||
Deferred one check intentionally:
|
||||
|
||||
- `must_resolve` target-specific enforcement was **deferred** because the current proposal structure does not safely identify the answered/targeted unknown in every valid case without inventing new linkage.
|
||||
|
||||
### 5. possibleInference
|
||||
|
||||
- Preserved current behaviour: `possibleInference` remains non-authoritative;
|
||||
- It does not independently justify mutation;
|
||||
- No validator path was added that treats it as authoritative structure.
|
||||
|
||||
## Captured False Positive
|
||||
|
||||
The exact `unsure` → `uncertain` populated structured-path false positive is now removed.
|
||||
|
||||
### Captured case
|
||||
|
||||
```text
|
||||
raw answer:
|
||||
I am unsure whether the projected office savings from the relocation are realistic.
|
||||
|
||||
userSupportedMeaning:
|
||||
The user is currently uncertain whether the projected office savings from the relocation are realistic.
|
||||
|
||||
supportCategory:
|
||||
uncertain
|
||||
|
||||
resolutionGuidance:
|
||||
must_remain_unresolved
|
||||
```
|
||||
|
||||
### Outcome
|
||||
|
||||
- **Passes** on the populated structured path;
|
||||
- Does **not** depend on synonym logic;
|
||||
- `unsure` vs `uncertain` wording is irrelevant when structured category is present.
|
||||
|
||||
## Tests Added / Updated
|
||||
|
||||
Focused deterministic coverage added or updated in:
|
||||
|
||||
- `tests/graph/schema.test.js`
|
||||
- `tests/graph/prompt-builder.test.js`
|
||||
- `tests/graph/apply-proposal.test.js`
|
||||
- `tests/graph/update-proposal.test.js` (directly related parse-boundary suite due to new enum enforcement)
|
||||
|
||||
### Required outcomes
|
||||
|
||||
1. raw `unsure` + structured `supportCategory=uncertain` does not produce old lexical mismatch rejection — **PASS**
|
||||
2. equivalent paraphrase wording does not change category acceptance when structured category is populated — **PASS**
|
||||
3. invalid `supportCategory` rejected by schema — **PASS**
|
||||
4. invalid `resolutionGuidance` rejected by schema — **PASS**
|
||||
5. `must_remain_unresolved` + relevant resolution mutation rejected — **PASS**
|
||||
6. `must_resolve` + unresolved target rejected if safely implementable — **DEFERRED**
|
||||
7. null structured fields still use existing lexical fallback — **PASS**
|
||||
8. populated `conditional_tradeoff` and `explicit_hard_constraint` use structured path without lexical verification — **PASS**
|
||||
9. `possibleInference` remains non-authoritative — **PASS**
|
||||
10. no new synonym/regex/keyword logic was added — **PASS**
|
||||
|
||||
## Commands Run
|
||||
|
||||
```bash
|
||||
npx vitest run tests/graph/schema.test.js tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js
|
||||
npx vitest run tests/graph/update-proposal.test.js
|
||||
```
|
||||
|
||||
## What this now guarantees
|
||||
|
||||
1. Populated structured semantic fields are now the primary fidelity contract.
|
||||
2. The engine no longer re-derives protected semantic categories lexically when those structured fields are populated.
|
||||
3. Invalid structured category/resolution values fail at schema parse time.
|
||||
4. `must_remain_unresolved` is enforced through deterministic structured consistency rather than English keyword matching.
|
||||
5. Legacy null structured proposals still follow the old lexical fallback path during transition.
|
||||
|
||||
## What remains intentionally unresolved
|
||||
|
||||
1. Safe deterministic enforcement of `must_resolve` against a specific target unknown without inventing new linkage.
|
||||
2. Population reliability of structured fields in live model runs.
|
||||
3. Full retirement of the lexical fallback path once structured population is proven reliable.
|
||||
|
||||
## Constraints respected
|
||||
|
||||
- No new semantic taxonomy;
|
||||
- No synonym or regex expansion;
|
||||
- No new semantic classifier;
|
||||
- No new LLM call;
|
||||
- No provider integration changes;
|
||||
- No Ollama calls;
|
||||
- No graph redesign.
|
||||
Reference in New Issue
Block a user