Feature/product platform foundation v0.62 #1

Merged
robbond merged 683 commits from feature/product-platform-foundation-v0.62 into feature/emergent-unknowns-v0.5 2026-09-09 07:58:20 +01:00
2 changed files with 289 additions and 1 deletions
Showing only changes of commit f330421294 - Show all commits
+25 -1
View File
@@ -893,10 +893,34 @@ The meanings are semantically equivalent; the rejection depends on `"unsure"` be
**Keyword-dictionary risk: YES.** The validator's semantic fidelity decision depends entirely on whether the LLM uses one of ~15-20 hardcoded English surface forms (`"unsure"`, `"not sure"`, etc.) rather than equivalent terms (`"uncertain"`). This is raw-language keyword detection, not structured semantic contract validation.
**Existing structured signals: SUFFICIENT.** The schema already carries `supportCategory` (enum), `resolutionGuidance` (nullable string distinguishing "must_remain_unresolved"/"may_resolve"), and `possibleInference`. These fields exist in the production contract but are consistently null from the LLM (per 56D), forcing reliance on keyword inference.
**Existing structured signals: PARTIAL.** The schema defines `supportCategory` (enum of five values) and `resolutionGuidance` (enum of three values), but both are free-text Zod strings without enum enforcement, and the LLM consistently produces null for both (per 56D). They can carry the needed semantics in theory but not in practice — a small contract extension (prompt requirement + schema constraint) is needed to make them operational. `possibleInference` exists but has no validator consumer. No production field carries a dedicated uncertainty-type classification (`evidence_needed`/`user_clarification_needed`) used in tests.
**57J.47 documentation correction applied:** Previous wording stated meaning was "STRENGTHENED" as fact. Corrected to distinguish: human semantic assessment (minor non-material paraphrase) from validator mechanism (lexical gap in detection patterns). The actual rejection was a false positive caused by incomplete keyword coverage, not genuine semantic strengthening.
**Production code changed: NO. Prompt changed: NO. Validator changed: NO. Tests permanently changed: NO. Ollama calls: 0.**
Configured Ollama: none (zero live calls).
---
### Experiment 57J.49 — Structured Semantic Fidelity Boundary Assessment
**Classification: B — EXISTING STRUCTURE IS PARTIAL**
Read-only architecture diagnosis of whether existing `answerMeaning` fields can replace keyword-based fidelity inference without a contract extension.
**Key finding:** All five classification values (`relative_priority_only`, `conditional_tradeoff`, `uncertain`, `explicit_hard_constraint`, `other`) and three resolution states (`must_remain_unresolved`, `may_resolve`, `must_resolve`) already exist in production enums (answerSupportCategory, answerResolutionGuidance at lines 147-158 of lib/graph/schema.js). The structured capability is fully designed.
**Missing piece — two independent gaps:**
1. **Population gap:** Both `supportCategory` and `resolutionGuidance` are free-text Zod strings not constrained to their respective enums, AND the prompt explicitly tells the model "if you are unsure of the exact wording, leave them null" (prompt-builder.js line 28). Result: LLM never populates them in production. The validator has no structured data to consume — it must re-infer from `userSupportedMeaning` text via keyword detection.
2. **Enforcement gap:** Neither field uses enum constraint enforcement. Even if populated, there is zero structural guarantee they contain valid category values.
**What existing structure CAN express (if populated):** `{supportCategory: "uncertain", resolutionGuidance: "must_remain_unresolved"}` — this exact combination expresses "remains unresolved uncertainty" without any lexical inference.
**Why B not A:** The fields are never reliably populated by the model, and without enum-constrained schema enforcement, they carry no trustworthy information in production today.
**What it does NOT establish:** That structured output solves the trust problem in general (Pattern A has no independent verification). That a dedicated `uncertaintyType` field (evidence_needed / user_clarification_needed) is unnecessary — this exists only in test fixtures, not any production schema.
Full results in `docs/experiment-57j49.md`.
+264
View File
@@ -0,0 +1,264 @@
# Experiment 57J.49 — Can Existing Structured Semantic Fields Replace Keyword-Based Fidelity Inference?
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
**Starting HEAD:** `a2c790e` experiment: diagnose uncertainty fidelity false positive
## Objective
Answer exactly:
> Can the current answerMeaning/schema contract carry enough structured semantic information to let fidelity validation compare meaning directly, instead of re-inferring uncertainty/constraint/trade-off semantics from English keywords?
This is a **read-only architecture diagnosis** following 57J.48's lexical false positive finding.
Do not call Ollama. Do not run the live API. Do not modify production code, prompts, validators, schema, or tests.
## Part 1 — Inventory of Existing Structured Semantics
For `answerMeaning` and directly related proposal fields:
### Field: userSupportedMeaning
```text
type: z.string().min(1)
required/optional: required (min length 1)
nullable: NO
populated by: LLM — model restates user meaning in third-person descriptive language
consumed by: deriveAnswerMeaningProfile() → keyword detection → category; validateAnswerMeaningCompatibilityWithRawAnswer(); validateAnswerMeaningAlignment()
survives proposal validation: YES (passes Zod schema parse as a required string field)
purpose: Primary carrier of what the user's answer semantically establishes; the sole structured semantic field actually populated by the model in production. All downstream category derivation flows through this text via keyword detection.
```
### Field: possibleInference
```text
type: z.string().nullable()
required/optional: optional (nullable)
nullable: YES — can be null or absent
populated by: LLM — when model wants to express a stronger interpretation beyond user meaning
consumed by: Only through rejectedProposalSnapshot passthrough in orchestrator.js. NOT consumed by any validator, classifier, or fidelity check. No production code examines possibleInference for any decision.
survives proposal validation: YES (passes Zod schema parse as optional nullable)
purpose: Intended for separating stronger model interpretations from user-supported meaning. Currently dead/pass-through — exists in schema and prompt but no validator inspects it.
```
### Field: supportCategory
```text
type: z.string().min(1).nullable() — FREE TEXT (no enum constraint enforced)
required/optional: optional (nullable)
nullable: YES
populated by: Prompt requests it, but LLM consistently produces null in all tested experiments. Confirmed by 56D: "the LLM does not auto-populate supportCategory." The deterministic derivation layer is the sole mechanism for meaning profile category determination.
consumed by: validateAnswerMeaningAlignment() would use it IF populated (lines 3010-3011). deriveAnswerMeaningProfile() derives category from text — NOT from this field.
survives proposal validation: YES (passes Zod as free text), but no schema constraint enforces valid values against answerSupportCategory enum
purpose: Intended as a structured semantic classification carrier that the model self-assigns. In production: never populated by model, so it carries no information. The enum answerSupportCategory exists at lines 147-153 of schema.js but is not used to constrain this field.
```
### Field: resolutionGuidance
```text
type: z.string().min(1).nullable() — FREE TEXT (no enum constraint enforced)
required/optional: optional (nullable)
nullable: YES
populated by: Prompt requests it, but LLM consistently produces null in all tested experiments. Same pattern as supportCategory.
consumed by: validateAnswerMeaningAlignment() would use it IF populated (line 3011, check at 3013). deriveAnswerMeaningProfile() derives guidance from text — NOT from this field.
survives proposal validation: YES (passes Zod as free text), but no schema constraint enforces valid values against answerResolutionGuidance enum
purpose: Intended to convey whether the semantic content requires remaining unresolved, may resolve, or must resolve. In production: never populated by model, so it carries no information.
```
### Field: uncertaintyType
```text
type: NOT PRESENT in production schema — only exists in experimental test fixtures (tests/reconstruction/semantic-regression-e-f*.test.js) and experiment documentation
required/optional: N/A — not in any production contract
populated by: N/A — not part of answerMeaningSchema or prompt instructions
consumed by: N/A — no production code references it
survives proposal validation: N/A
purpose: Experimental concept from 57J.48 documentation describing a potential structured uncertainty classification. Has never existed in the production schema or model contract.
```
## Part 2 — Captured-Case Representation
**Raw answer:** "I am unsure whether the projected office savings from the relocation are realistic."
**Semantically faithful model meaning:** "The user is currently uncertain whether the projected office savings from the relocation are realistic."
**Shared semantic fact to express:** `meaning remains unresolved uncertainty`
### Can current fields express this without lexical inference?
**PARTIAL**
The minimum existing field/value combination that would express it (if populated by the model):
```json
{
"userSupportedMeaning": "The user is currently uncertain whether the projected office savings from the relocation are realistic.",
"supportCategory": "uncertain",
"resolutionGuidance": "must_remain_unresolved"
}
```
- `supportCategory: "uncertain"` — directly expresses the uncertainty classification (one of five values in answerSupportCategory enum)
- `resolutionGuidance: "must_remain_unresolved"` — directly expresses that resolution is not appropriate (one of three values in answerResolutionGuidance enum)
**Why PARTIAL, not YES:** These two fields (supportCategory and resolutionGuidance) are the correct carriers but are **never populated by the model** in production. The validator currently cannot consume them because they are null. The structured capability exists in the schema design but is unreachable — no code path populates these fields with actual classification values, only userSupportedMeaning carries information end-to-end.
Additionally:
- Both fields are free-text Zod types (no enum constraint enforcement), so even if populated, there is no structural guarantee they contain valid category values.
- `uncertaintyType` does not exist in the production schema at all — a dedicated structured uncertainty classifier field would need to be added or supportCategory used for that purpose.
## Part 3 — Current Population Path
### supportCategory: **B — schema exists but prompt does not clearly require population**
**Why:** The prompt (prompt-builder.js line 28) says "supportCategory and resolutionGuidance are optional descriptive hints only; if you are unsure of the exact wording, leave them null rather than inventing rigid category labels." This explicit permission to remain null explains why the LLM consistently produces null. The schema does not enforce population (optional + nullable + free-text). Combined: schema says "nullable," prompt says "leave null if unsure" — no mechanism drives model to populate it.
### resolutionGuidance: **B — schema exists but prompt does not clearly require population**
**Why:** Same mechanism as supportCategory. Prompt line 28 explicitly tells the model it can leave it null. Schema marks it optional + nullable. No enforcement.
### possibleInference: **D — field is derived/populated conditionally by model but has no downstream consumer**
**Why:** The model populates this when it wants to express a stronger interpretation beyond what the user stated. It survives validation as a pass-through field but is never examined by any validator, classifier, or fidelity check. Its existence is effectively cosmetic — it exists in the contract but carries no functional weight.
### uncertaintyType: **NOT PRESENT**
**Why:** This field has never existed in the production answerMeaning schema. It appears only in experimental test fixtures (57J.48 documentation references it as a potential structured signal, and tests for semantic-regression-e/f use it as a model output from inference calls, not from the graph-update contract).
## Part 4 — Current Validator Dependency
### Uncertainty
```text
current source: RAW TEXT
deriveAnswerMeaningProfile() lines 2877-2883: .includes() checks on ["not really sure", "not sure", "unsure", "do not know", "don't know"] → category = "uncertain"
existing structured replacement available: PARTIAL
supportCategory could carry the uncertainty classification (one of five enum values includes "uncertain"), but model never populates it. No other field carries uncertainty classification.
would replacement require new semantic taxonomy: NO
"uncertain" already exists in answerSupportCategory enum at line 150 of schema.js
```
### Conditional/trade-off
```text
current source: MIXED (hasConditionalQualification keyword detection + conditionalPreferenceStructure compound check)
deriveAnswerMeaningProfile() lines 2893-2904 uses hasConditionalQualification(text) [includes("might","normally","for the right opportunity","depends","conditional","under specific")] plus hasDefaultPreferenceSignal + hasExceptionOrOverrideSignal
existing structured replacement available: PARTIAL
supportCategory could carry "conditional_tradeoff" (enum value at line 149 of schema.js). But model never populates it.
would replacement require new semantic taxonomy: NO
"conditional_tradeoff" already exists in answerSupportCategory enum at line 149
```
### Hard constraint
```text
current source: RAW TEXT
mentionsHardConstraint(text) at line 2834: includes("hard constraint","constraint","non negotiable","non-negotiable")
mentionsNegatedHardConstraint(text) at line 2843: included for negation detection
existing structured replacement available: PARTIAL
supportCategory could carry "explicit_hard_constraint" (enum value at line 151 of schema.js). But model never populates it.
would replacement require new semantic taxonomy: NO
"explicit_hard_constraint" already exists in answerSupportCategory enum at line 151
```
### Resolution semantics
```text
current source: RAW TEXT → deriveAnswerMeaningProfile() resolutionGuidance derivation (lines 2886, 2902, 2909, 2922) or fallback null
Derived from text patterns: uncertainty phrases → "must_remain_unresolved", conditional → "may_resolve", hard constraint → "must_resolve", else null
existing structured replacement available: PARTIAL
resolutionGuidance field exists for this purpose, and three valid values exist in answerResolutionGuidance enum. But model never populates it, so deriveAnswerMeaningProfile() must re-derive from text.
would replacement require new semantic taxonomy: NO
"must_remain_unresolved", "may_resolve", "must_resolve" all exist in answerResolutionGuidance enum at lines 156-158
```
### Relative priority (not explicitly asked but relevant)
```text
current source: RAW TEXT
deriveAnswerMeaningProfile() lines 2913-2924: .includes() checks on ["matters more", "more important", "higher priority", "greater relative importance", "relative importance"] → category = "relative_priority_only"
existing structured replacement available: PARTIAL
supportCategory could carry "relative_priority_only" (enum value at line 148 of schema.js). But model never populates it.
would replacement require new semantic taxonomy: NO
"relative_priority_only" already exists in answerSupportCategory enum at line 148
```
## Part 5 — Trust-Boundary Problem
### Pattern A — trust model classification directly
Model supplies structured category; validator compares category to category.
```text
removes lexical dictionary dependence: YES (for all protected categories simultaneously, provided model populates supportCategory)
preserves fidelity protection: PARTIAL (depends on reliable model population; if model lies about its own classification, validator has no independent check — the current keyword inference provides that independent check but with lexical coverage gaps)
requires new schema fields: NO (supportCategory already exists; enum values cover all protected categories)
requires new semantic taxonomy: NO (all five categories + three resolution_guidance values already exist in enums)
```
### Pattern B — model classification + raw-text lexical verification
Structured category is populated, but current keyword inference remains the authority. Validator checks both: model says X, keywords say Y → mismatch flag.
```text
removes lexical dictionary dependence: NO (still uses keyword detection as one of two inputs)
preserves fidelity protection: YES (cross-checks model claim against independent text analysis; catches both lexical gaps AND model hallucination)
requires new schema fields: NO
requires new semantic taxonomy: NO
```
### Pattern C — structured model claim + independent deterministic consistency checks that do NOT attempt full English semantic inference
Examples: schema invariants, cross-field consistency, structural plausibility.
```text
removes lexical dictionary dependence: PARTIAL (removes keyword detection for uncertainty classification where supportCategory is populated; remaining categories still use keywords when supportCategory is null)
preserves fidelity protection: PARTIAL (deterministic checks like "resolutionGuidance=must_remain_unresolved AND resolved=true" catch some contradictions but not all semantic inconsistencies — e.g., a wrong category with compatible text could pass)
requires new schema fields: NO
requires new semantic taxonomy: NO
Specific deterministic consistency checks already possible from existing fields:
1. If resolutionGuidance = "must_remain_unresolved" AND proposal resolves any unknown → CONTRADICTION (currently validated via derived text, would be directly checkable if field populated)
2. If supportCategory = "explicit_hard_constraint" AND userSupportedMeaning contains "rather than a hard constraint" or "not a hard constraint" → CONTRADICTION (cross-field consistency between category and meaning text)
3. If possibleInference is populated but userSupportedMeaning carries no new uncertainty → INCONSISTENCY (inference without meaningful supporting meaning)
4. supportCategory value should be one of answerSupportCategory enum values — currently not enforced by schema
5. resolutionGuidance value should be one of answerResolutionGuidance enum values — currently not enforced by schema
```
## Part 6 — Architecture Decision
### **B — EXISTING STRUCTURE IS PARTIAL**
Current fields cover some protected semantics but cannot replace lexical inference cleanly without a small structured-contract extension.
**What this establishes:**
1. The `supportCategory` enum (answerSupportCategory) already contains all five required classification values: relative_priority_only, conditional_tradeoff, uncertain, explicit_hard_constraint, other.
2. The `resolutionGuidance` enum (answerResolutionGuidance) already contains all three required resolution states: must_remain_unresolved, may_resolve, must_resolve.
3. These fields exist in the production schema and are explicitly requested in the prompt — the structured capability is designed but not operationalized.
4. The missing piece is **reliable model population** (prompt says "optional" and "leave null if unsure") and **schema enforcement** (both are free-text Zod strings, not constrained to their respective enums).
**What it does NOT establish:**
1. That structured output alone solves the trust problem — Pattern A reveals that trusting model classification directly has no independent verification.
2. That the existing enum taxonomy is complete — `uncertaintyType` (evidence_needed / user_clarification_needed) used in tests for regression cases E/F does not exist in any production schema. If this distinction matters, it requires new fields.
3. That adding field requirements to the prompt is sufficient — model compliance with "please fill these fields" has never been proven across repeated runs and domains.
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Tests permanently changed: NO
## Temporary test removed: YES (none created for this read-only diagnosis)
## Ollama calls made: 0
## Dev server disturbed: NO
---