146 lines
6.1 KiB
Markdown
146 lines
6.1 KiB
Markdown
# Experiment 57J.51 — Structured Semantic Fidelity Implementation
|
|
|
|
**Branch:** `feature/structured-semantic-fidelity-v0.20`
|
|
**Starting HEAD:** `b6a232ff6f56b5f1af49d94bb2881190b5bf8345`
|
|
**Production commit:** `7d06cd3c473cee64c2c371c1e1af1c466cdc32dd`
|
|
|
|
## Objective
|
|
|
|
Implement Option C from Experiment 57J.50:
|
|
|
|
> Use existing structured semantic fields (`supportCategory`, `resolutionGuidance`) as the primary fidelity contract when populated, enforce their allowed enum values, validate only structured cross-field consistency, and retain current lexical derivation only as a temporary fallback when those fields are null.
|
|
|
|
## Scope Implemented
|
|
|
|
### 1. Schema
|
|
|
|
`lib/graph/schema.js`
|
|
|
|
- Constrained `answerMeaning.supportCategory` to `z.enum(Object.values(answerSupportCategory)).nullable().optional()`;
|
|
- Constrained `answerMeaning.resolutionGuidance` to `z.enum(Object.values(answerResolutionGuidance)).nullable().optional()`;
|
|
- Preserved transitional nullability on both fields;
|
|
- Reused existing enum constants — no new taxonomy added.
|
|
|
|
### 2. Prompt
|
|
|
|
`lib/graph/prompt-builder.js`
|
|
|
|
- Exposed allowed values for both structured semantic fields in the output contract;
|
|
- Replaced the old “optional descriptive hints only” instruction with structured population guidance;
|
|
- Instructed the model to:
|
|
- populate `supportCategory` whenever the answer fits an existing category,
|
|
- use `other` when none of the protected categories applies,
|
|
- avoid leaving `supportCategory` null merely because wording is uncertain,
|
|
- populate `resolutionGuidance` when one of the existing resolution states genuinely applies,
|
|
- keep `resolutionGuidance` null only when no existing state actually applies;
|
|
- Used the existing `formatEnumValues()` helper;
|
|
- Added no provider-specific wording.
|
|
|
|
### 3. Validator — structured first
|
|
|
|
`lib/graph/apply-proposal.js`
|
|
|
|
- Added `getAnswerMeaningProfile(answerMeaning)` to unify:
|
|
- structured `supportCategory` / `resolutionGuidance` when populated,
|
|
- lexical derivation only when those structured fields are null;
|
|
- Updated `validateAnswerMeaningCompatibilityWithRawAnswer()` so populated structured semantic fields bypass raw-text lexical category verification entirely;
|
|
- Updated `validateAnswerMeaningAlignment()` so:
|
|
- structured fields are authoritative when populated,
|
|
- lexical fallback remains active only for legacy null cases.
|
|
|
|
### 4. Non-lexical consistency
|
|
|
|
Implemented one deterministic structured consistency check now:
|
|
|
|
- `resolutionGuidance = must_remain_unresolved` + proposal resolves an unknown → reject with:
|
|
- `Proposal resolves an unknown even though answerMeaning.resolutionGuidance is must_remain_unresolved.`
|
|
|
|
Deferred one check intentionally:
|
|
|
|
- `must_resolve` target-specific enforcement was **deferred** because the current proposal structure does not safely identify the answered/targeted unknown in every valid case without inventing new linkage.
|
|
|
|
### 5. possibleInference
|
|
|
|
- Preserved current behaviour: `possibleInference` remains non-authoritative;
|
|
- It does not independently justify mutation;
|
|
- No validator path was added that treats it as authoritative structure.
|
|
|
|
## Captured False Positive
|
|
|
|
The exact `unsure` → `uncertain` populated structured-path false positive is now removed.
|
|
|
|
### Captured case
|
|
|
|
```text
|
|
raw answer:
|
|
I am unsure whether the projected office savings from the relocation are realistic.
|
|
|
|
userSupportedMeaning:
|
|
The user is currently uncertain whether the projected office savings from the relocation are realistic.
|
|
|
|
supportCategory:
|
|
uncertain
|
|
|
|
resolutionGuidance:
|
|
must_remain_unresolved
|
|
```
|
|
|
|
### Outcome
|
|
|
|
- **Passes** on the populated structured path;
|
|
- Does **not** depend on synonym logic;
|
|
- `unsure` vs `uncertain` wording is irrelevant when structured category is present.
|
|
|
|
## Tests Added / Updated
|
|
|
|
Focused deterministic coverage added or updated in:
|
|
|
|
- `tests/graph/schema.test.js`
|
|
- `tests/graph/prompt-builder.test.js`
|
|
- `tests/graph/apply-proposal.test.js`
|
|
- `tests/graph/update-proposal.test.js` (directly related parse-boundary suite due to new enum enforcement)
|
|
|
|
### Required outcomes
|
|
|
|
1. raw `unsure` + structured `supportCategory=uncertain` does not produce old lexical mismatch rejection — **PASS**
|
|
2. equivalent paraphrase wording does not change category acceptance when structured category is populated — **PASS**
|
|
3. invalid `supportCategory` rejected by schema — **PASS**
|
|
4. invalid `resolutionGuidance` rejected by schema — **PASS**
|
|
5. `must_remain_unresolved` + relevant resolution mutation rejected — **PASS**
|
|
6. `must_resolve` + unresolved target rejected if safely implementable — **DEFERRED**
|
|
7. null structured fields still use existing lexical fallback — **PASS**
|
|
8. populated `conditional_tradeoff` and `explicit_hard_constraint` use structured path without lexical verification — **PASS**
|
|
9. `possibleInference` remains non-authoritative — **PASS**
|
|
10. no new synonym/regex/keyword logic was added — **PASS**
|
|
|
|
## Commands Run
|
|
|
|
```bash
|
|
npx vitest run tests/graph/schema.test.js tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js
|
|
npx vitest run tests/graph/update-proposal.test.js
|
|
```
|
|
|
|
## What this now guarantees
|
|
|
|
1. Populated structured semantic fields are now the primary fidelity contract.
|
|
2. The engine no longer re-derives protected semantic categories lexically when those structured fields are populated.
|
|
3. Invalid structured category/resolution values fail at schema parse time.
|
|
4. `must_remain_unresolved` is enforced through deterministic structured consistency rather than English keyword matching.
|
|
5. Legacy null structured proposals still follow the old lexical fallback path during transition.
|
|
|
|
## What remains intentionally unresolved
|
|
|
|
1. Safe deterministic enforcement of `must_resolve` against a specific target unknown without inventing new linkage.
|
|
2. Population reliability of structured fields in live model runs.
|
|
3. Full retirement of the lexical fallback path once structured population is proven reliable.
|
|
|
|
## Constraints respected
|
|
|
|
- No new semantic taxonomy;
|
|
- No synonym or regex expansion;
|
|
- No new semantic classifier;
|
|
- No new LLM call;
|
|
- No provider integration changes;
|
|
- No Ollama calls;
|
|
- No graph redesign.
|