experiment: assess structured semantic fidelity boundary
- Experiment 57J.49: read-only architecture diagnosis showing EXISTING STRUCTURE IS PARTIAL - All classification enums (supportCategory) and resolution enums (resolutionGuidance) already exist in production schema - Gap is population (prompt says leave null if unsure) + enforcement (no enum constraint on Zod fields) - Corrected 57J.48 overstatement of SUFFICIENT → PARTIAL sufficiency in handoff
This commit is contained in:
+25
-1
@@ -893,10 +893,34 @@ The meanings are semantically equivalent; the rejection depends on `"unsure"` be
|
||||
|
||||
**Keyword-dictionary risk: YES.** The validator's semantic fidelity decision depends entirely on whether the LLM uses one of ~15-20 hardcoded English surface forms (`"unsure"`, `"not sure"`, etc.) rather than equivalent terms (`"uncertain"`). This is raw-language keyword detection, not structured semantic contract validation.
|
||||
|
||||
**Existing structured signals: SUFFICIENT.** The schema already carries `supportCategory` (enum), `resolutionGuidance` (nullable string distinguishing "must_remain_unresolved"/"may_resolve"), and `possibleInference`. These fields exist in the production contract but are consistently null from the LLM (per 56D), forcing reliance on keyword inference.
|
||||
**Existing structured signals: PARTIAL.** The schema defines `supportCategory` (enum of five values) and `resolutionGuidance` (enum of three values), but both are free-text Zod strings without enum enforcement, and the LLM consistently produces null for both (per 56D). They can carry the needed semantics in theory but not in practice — a small contract extension (prompt requirement + schema constraint) is needed to make them operational. `possibleInference` exists but has no validator consumer. No production field carries a dedicated uncertainty-type classification (`evidence_needed`/`user_clarification_needed`) used in tests.
|
||||
|
||||
**57J.47 documentation correction applied:** Previous wording stated meaning was "STRENGTHENED" as fact. Corrected to distinguish: human semantic assessment (minor non-material paraphrase) from validator mechanism (lexical gap in detection patterns). The actual rejection was a false positive caused by incomplete keyword coverage, not genuine semantic strengthening.
|
||||
|
||||
**Production code changed: NO. Prompt changed: NO. Validator changed: NO. Tests permanently changed: NO. Ollama calls: 0.**
|
||||
|
||||
Configured Ollama: none (zero live calls).
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.49 — Structured Semantic Fidelity Boundary Assessment
|
||||
|
||||
**Classification: B — EXISTING STRUCTURE IS PARTIAL**
|
||||
|
||||
Read-only architecture diagnosis of whether existing `answerMeaning` fields can replace keyword-based fidelity inference without a contract extension.
|
||||
|
||||
**Key finding:** All five classification values (`relative_priority_only`, `conditional_tradeoff`, `uncertain`, `explicit_hard_constraint`, `other`) and three resolution states (`must_remain_unresolved`, `may_resolve`, `must_resolve`) already exist in production enums (answerSupportCategory, answerResolutionGuidance at lines 147-158 of lib/graph/schema.js). The structured capability is fully designed.
|
||||
|
||||
**Missing piece — two independent gaps:**
|
||||
|
||||
1. **Population gap:** Both `supportCategory` and `resolutionGuidance` are free-text Zod strings not constrained to their respective enums, AND the prompt explicitly tells the model "if you are unsure of the exact wording, leave them null" (prompt-builder.js line 28). Result: LLM never populates them in production. The validator has no structured data to consume — it must re-infer from `userSupportedMeaning` text via keyword detection.
|
||||
|
||||
2. **Enforcement gap:** Neither field uses enum constraint enforcement. Even if populated, there is zero structural guarantee they contain valid category values.
|
||||
|
||||
**What existing structure CAN express (if populated):** `{supportCategory: "uncertain", resolutionGuidance: "must_remain_unresolved"}` — this exact combination expresses "remains unresolved uncertainty" without any lexical inference.
|
||||
|
||||
**Why B not A:** The fields are never reliably populated by the model, and without enum-constrained schema enforcement, they carry no trustworthy information in production today.
|
||||
|
||||
**What it does NOT establish:** That structured output solves the trust problem in general (Pattern A has no independent verification). That a dedicated `uncertaintyType` field (evidence_needed / user_clarification_needed) is unnecessary — this exists only in test fixtures, not any production schema.
|
||||
|
||||
Full results in `docs/experiment-57j49.md`.
|
||||
|
||||
Reference in New Issue
Block a user