experiment: diagnose null semantic mutation path
This commit is contained in:
@@ -1675,3 +1675,7 @@ This satisfies 57J.77's boundary A recommendation: a committed update-only path
|
||||
### Experiment 57J.83 — Direct Answer-Meaning Capture from updatedProposal
|
||||
|
||||
**Objective:** Verify the production path correctly reads `answerMeaning` and `structuralActionRequired` from inside `updatedProposal` (graphUpdate schema container) and that all test mock boundaries are coherent with this contract. **Classification: IMPLEMENTED.** Fixed mock boundary mismatch where some fixtures placed fields at root level while capture logic read from inside `updatedProposal`. All 49 harness tests pass. Full results in `docs/experiment-57j83.md`. No production code changed; only experiment apparatus (script + test harness).
|
||||
|
||||
### Experiment 57J.85 — Null Semantic/Action Architecture Diagnosis (Read-Only)
|
||||
|
||||
**Objective:** Diagnose why `answerMeaning=null + structuralActionRequired=null + meaningful mutation` is accepted through the current pipeline, and whether this compatibility path should remain open. Read-only diagnosis across prompt contract, schema, validator, and apply-proposal validation layers. **Classification: B — TRANSITION COMPATIBILITY.** The null/null/mutation path persists because: (1) Zod schema allows nullable fields for backward compatibility; (2) validator rules are scoped to only reject when meaning IS populated (rule 1), leaving null-meaning mutations unguarded; (3) the model produces useful data through this path (57J.84: £2m/year on reported_claim node); (4) tightening without a deterministic recovery/retry path would discard that information. Architectural choice A — keep the null transition path for now, pending deterministic recovery capability before tightening becomes safe. Smallest next boundary: implement a deterministic recovery mechanism for proposals with meaningful mutations but unpopulated semantic/action fields. No production code changed; no Ollama calls; documentation-only diagnosis.
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# Experiment 57J.85 — Null Semantic/Action Architecture Diagnosis (Read-Only)
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `eaf3194` (experiment: observe direct meaning/action fields on anchored update)
|
||||
|
||||
## Objective
|
||||
|
||||
Diagnose why `answerMeaning=null + structuralActionRequired=null + meaningful mutation` is accepted through the current pipeline, and whether this compatibility path should remain open.
|
||||
|
||||
Read-only diagnosis. No code changes. No Ollama calls. No live API calls.
|
||||
|
||||
## Context Files Read
|
||||
|
||||
1. `docs/current-handoff.md` (handoff state through 57J.84)
|
||||
2. `docs/experiment-57j84.md` (the live case: null meaning + null action + £2m/year mutation accepted)
|
||||
3. `lib/graph/schema.js` (schema truth for all relevant fields)
|
||||
4. `lib/graph/prompt-builder.js` (current HEAD — prompt contract rules)
|
||||
5. `lib/graph/utils.js` (validator logic at line 868+)
|
||||
6. `lib/graph/apply-proposal.js` (validation/parsing section: lines 2937–3146, applyValidatedProposal entry at 3182)
|
||||
|
||||
---
|
||||
|
||||
## PROMPT CONTRACT TRACE
|
||||
|
||||
### userSupportedMeaning required on every answer?
|
||||
**CONDITIONAL** — Required *when you have semantic intent that requires graph progress* (rule #6). The prompt says "If answerMeaning.userSupportedMeaning contains consequential information... you MUST express its effect through structural mutation." It also has rules 26–31 governing how to populate userSupportedMeaning when present. However, the prompt does not say "you MUST always populate userSupportedMeaning" — it leaves open the possibility of answerMeaning=null when the answer contains no user-supported meaning that requires graph progress (rule #14 in Additional Guidance: "If rule #6 does not apply... return empty arrays").
|
||||
|
||||
### structuralActionRequired required on every proposal?
|
||||
**CONDITIONAL** — Required when userSupportedMeaning is populated (Declaration Rule section: "When answerMeaning.userSupportedMeaning is populated you MUST set structuralActionRequired to match what your proposal outputs"). However, when answerMeaning=null or userSupportedMeaning is null/unpopulated, the prompt does not explicitly require structuralActionRequired. The contract says it's a declaration tied to semantic intent.
|
||||
|
||||
### Meaningful mutation + answerMeaning=null explicitly permitted?
|
||||
**AMBIGUOUS** — The prompt implies that if rule #6 doesn't apply (no consequential user-supported meaning), the model should return empty arrays with null meaning. But a *meaningful* mutation with null meaning falls in no explicit category: not rule #6 (which requires userSupportedMeaning to be populated), and not "no semantic intent" (since there's clearly semantic content). The prompt silently allows this combination through omission.
|
||||
|
||||
### Meaningful mutation + structuralActionRequired=null explicitly permitted?
|
||||
**AMBIGUOUS** — Same reasoning as above. When answerMeaning is null, the Declaration Rule does not trigger, so structuralActionRequired is unmentioned for this case.
|
||||
|
||||
---
|
||||
|
||||
## SCHEMA TRUTH
|
||||
|
||||
From `lib/graph/schema.js`:
|
||||
|
||||
### answerMeaning
|
||||
```js
|
||||
answerMeaningSchema.nullable().default(null)
|
||||
└── userSupportedMeaning: z.string().min(1) [REQUIRED within object]
|
||||
└── possibleInference: z.string().nullable().optional() [OPTIONAL/NULLABLE, defaults to null via Zod]
|
||||
└── supportCategory: z.enum(...).nullable().optional() [OPTIONAL/NULLABLE, defaults to null]
|
||||
└── resolutionGuidance: z.enum(...).nullable().optional() [OPTIONAL/NULLABLE, defaults to null]
|
||||
```
|
||||
|
||||
**Classification:** answerMeaning is OPTIONAL (can be omitted from JSON), NULLABLE (can be explicitly null), DEFAULTED (null if absent). userSupportedMeaning is REQUIRED *within a non-null object* but the outer container is optional.
|
||||
|
||||
### structuralActionRequired
|
||||
```js
|
||||
structuralActionRequired: z.boolean().nullable().optional()
|
||||
```
|
||||
|
||||
**Classification:** OPTIONAL, NULLABLE, defaults to null when omitted.
|
||||
|
||||
### answerMeaning omission/null while proposal schema-valid?
|
||||
**YES** — `answerMeaningSchema.nullable().default(null)` means the entire answerMeaning field can be null and the schema still passes. Even if answerMeaning object is present, only userSupportedMeaning is required within it; possibleInference, supportCategory, and resolutionGuidance are all nullable+optional.
|
||||
|
||||
### structuralActionRequired omission/null while proposal schema-valid?
|
||||
**YES** — `z.boolean().nullable().optional()` means the field can be omitted entirely or set to null, and Zod will accept it. No schema constraint prevents this.
|
||||
|
||||
---
|
||||
|
||||
## NULL VS OMISSION BOUNDARY
|
||||
|
||||
### answerMeaning: NOT DISTINGUISHABLE
|
||||
- Model omits field → Zod defaults to `null`
|
||||
- Model emits `null` → stays `null`
|
||||
- Code sees: `answerMeaning === null` — both indistinguishable
|
||||
|
||||
The information-loss boundary is at Zod schema application. Once parsed, there is no trace of whether the model omitted the field or emitted null.
|
||||
|
||||
### structuralActionRequired: NOT DISTINGUISHABLE
|
||||
- Model omits field → stays `undefined` (optional + nullable)
|
||||
- Model emits `null` → stays `null`
|
||||
- Code checks both with `=== null || === undefined` — treats them identically
|
||||
|
||||
The information-loss boundary is at Zod schema application. Both omission and explicit null converge to an effective "not set" state that the validator cannot differentiate.
|
||||
|
||||
---
|
||||
|
||||
## VALIDATOR MATRIX (using validateGraphUpdate at HEAD)
|
||||
|
||||
Current validation logic in utils.js:
|
||||
```js
|
||||
meaningPopulated = !!update.answerMeaning?.userSupportedMeaning;
|
||||
hasMeaningfulChange = [addedNodes, statusChanged, valueChanged, addedEdges, removedEdges];
|
||||
fieldAbsent = structuralActionRequired === null || undefined;
|
||||
|
||||
// Rule 1: missing field + meaning populated → REJECT
|
||||
if (fieldAbsent && meaningPopulated) → reject
|
||||
|
||||
// Rule 2: true + no mutation → REJECT
|
||||
if (structuralActionRequired === true && !hasMeaningfulChange) → reject
|
||||
|
||||
// Rule 3: false + mutation → REJECT
|
||||
if (structuralActionRequired === false && hasMeaningfulChange) → reject
|
||||
|
||||
// Rule 4: no mutation + absent field + no meaning → REJECT ("Update contains no meaningful change")
|
||||
if (!hasMeaningfulChange && fieldAbsent && !meaningPopulated) → reject
|
||||
```
|
||||
|
||||
### A: populated meaning + true + mutation
|
||||
**PASS** — All three rules are satisfied (meaningPopulated=true doesn't trigger rule 1 because fieldAbsent=false; rules 2 and 3 don't apply because structuralActionRequired===true AND hasMeaningfulChange=true; rule 4 doesn't apply because hasMeaningfulChange=true).
|
||||
|
||||
### B: populated meaning + false + no mutation
|
||||
**PASS** — All checks pass. Rule 1 doesn't trigger (fieldAbsent=false). Rules 2/3 don't trigger (true is not false). Rule 4 requires !hasMeaningfulChange AND fieldAbsent AND !meaningPopulated — but meaningPopulated=true, so rule 4 doesn't fire.
|
||||
|
||||
### C: populated meaning + null action
|
||||
**REJECT** — Rule 1 fires: meaningPopulated=true && fieldAbsent=true → "structuralActionRequired must be present when userSupportedMeaning is populated".
|
||||
|
||||
### D: null meaning + null action + mutation
|
||||
**PASS** — Rule 1 doesn't trigger (meaningPopulated=false). Rules 2/3 don't trigger (fieldAbsent=true, not === true/false). Rule 4 doesn't trigger (hasMeaningfulChange=true). **Escape hatch.**
|
||||
|
||||
### E: null meaning + null action + no mutation
|
||||
**REJECT** — Rule 4 fires: !hasMeaningfulChange=true && fieldAbsent=true && !meaningPopulated=true → "Update contains no meaningful change".
|
||||
|
||||
### F: null meaning + true + mutation
|
||||
**PASS** — No rules fire. Rules 1/3 check structuralActionRequired===true (rule 3 fails because hasMeaningfulChange=true). Rule 4 doesn't trigger (hasMeaningfulChange=true). The true declaration is inconsistent with null meaning but not explicitly checked.
|
||||
|
||||
### G: null meaning + false + no mutation
|
||||
**PASS** — No rules fire. Rules 1/2 don't apply for the same reasons as F and E respectively. Rule 4 doesn't trigger (hasMeaningfulChange=false AND fieldAbsent=true AND !meaningPopulated=true... wait, that's rule 4 which should REJECT).
|
||||
|
||||
Correction: Rule 4 fires: !hasMeaningfulChange && fieldAbsent && !meaningPopulated → "Update contains no meaningful change". **REJECT**.
|
||||
|
||||
### H: null meaning + false + mutation
|
||||
**REJECT** — Rule 3 fires: structuralActionRequired===false && hasMeaningfulChange=true → "structuralActionRequired is false but proposal contains meaningful mutations".
|
||||
|
||||
### I: null meaning + true + no mutation
|
||||
**REJECT** — Rule 2 fires: structuralActionRequired===true && !hasMeaningfulChange=true → "structuralActionRequired is true but proposal contains no graph mutation".
|
||||
|
||||
---
|
||||
|
||||
## 57J.84 PATH CLASSIFICATION
|
||||
|
||||
**Classification: B — TRANSITION COMPATIBILITY**
|
||||
|
||||
### Why
|
||||
The null/nullable fields are schema-legal and the validator rules are carefully scoped to only reject when meaning IS populated (rule 1) or when the boolean is explicitly true/false but contradicts mutation state (rules 2/3). The specific combination of answerMeaning=null + structuralActionRequired=null + meaningful mutation falls through all rules because:
|
||||
1. Rule 1 requires meaningPopulated=true — not met
|
||||
2. Rules 2/3 require structuralActionRequired to be ===true or ===false — fieldAbsent=true prevents this
|
||||
3. Rule 4 requires !hasMeaningfulChange — not met
|
||||
|
||||
This is not accidental (C would mean the rules were written carelessly), because the rules are explicitly structured with these exact conditions. It's not first-class design (A) because no prompt rule encourages it, and no architecture document describes it as a feature. It exists because during transition, nullable fields remained for compatibility while structured field population was incomplete — tightening would reject live proposals that contain useful data.
|
||||
|
||||
### Schema-valid: YES
|
||||
Zod schema accepts null/absent for both answerMeaning and structuralActionRequired.
|
||||
|
||||
### Prompt-compliant: AMBIGUOUS
|
||||
The prompt does not explicitly permit this path (no rule says "you may produce mutation without semantic declarations"), but it also doesn't explicitly forbid it — the prompt's constraints on structuralActionRequired only activate when userSupportedMeaning is populated. This creates a silent gap.
|
||||
|
||||
### Validator-accepted: YES
|
||||
All four validator rules are satisfied for the null/null/mutation case.
|
||||
|
||||
### Architecturally intended: TRANSITION ONLY
|
||||
The combination exists because of incomplete transition, not deliberate design. The field-absence rule (rule 1) only triggers when meaning is populated — intentionally limiting its scope during transition.
|
||||
|
||||
### Deterministic accountability: SHAPE ONLY
|
||||
What IS validated: node/edge structure validity, ID consistency, size limit, no-op guard (when meaning absent and no mutation). What is NOT validated for this path: any semantic intent check, any structural action declaration check, any answer-meaning alignment check. The validator confirms shape only — that addedNodes has correct fields, that edges reference valid nodes, etc.
|
||||
|
||||
### What is still verified:
|
||||
- Schema structure of all nodes/edges in the proposal
|
||||
- No duplicate IDs against existing graph
|
||||
- No update to non-existent nodes
|
||||
- Size < 100KB
|
||||
- If structuralActionRequired===true/false: contradiction with actual mutation state (rules 2/3)
|
||||
- If meaningPopulated+fieldAbsent: rejection (rule 1)
|
||||
- If no mutation + fieldAbsent + !meaningPopulated: rejection (rule 4)
|
||||
|
||||
---
|
||||
|
||||
## MIGRATION READINESS
|
||||
|
||||
### answerMeaning population reliability: PARTIAL
|
||||
57J.84 proves the model can produce null when it should populate meaningful content (it implicitly captured meaning via structure). However, other experiments show the model can populate userSupportedMeaning in some cases. Reliability is proven to be inconsistent — sometimes populated, sometimes null for consequential answers.
|
||||
|
||||
### structuralActionRequired population reliability: PARTIAL
|
||||
57J.84 proves null production alongside meaningful mutation. 57J.71 proved true+mutation is possible (same model). But the consistent null production on the "meaning via structure" path means population is not reliable when meaning flows through implicit representation.
|
||||
|
||||
### true + mutation path: PROVEN
|
||||
57J.71 demonstrated the model can produce `structuralActionRequired=true` with meaningful mutation in a single pass. The validator accepts it cleanly. But this only works when answerMeaning IS populated — proving that the model can follow the declaration rule WHEN triggered.
|
||||
|
||||
### false + no-op path: PROVEN
|
||||
Multiple experiments show the validator correctly accepts and rejects false+no-op combinations. The contract is clean for this path.
|
||||
|
||||
### null/null + mutation still exercised live: YES
|
||||
57J.84 is direct evidence — £2m/year savings data was structurally preserved via a new reported_claim node with supports edge, all semantic/action fields were null, and the update was accepted at HTTP 200.
|
||||
|
||||
---
|
||||
|
||||
## ARCHITECTURAL CHOICE
|
||||
|
||||
**Choice: A — KEEP NULL TRANSITION PATH FOR NOW**
|
||||
|
||||
### Why
|
||||
Population reliability/recovery is not strong enough to tighten safely. The 57J.84 case demonstrates that meaningful, consequential data (£2m/year + lease-basis) flows through this path successfully — it IS preserved in the graph even without semantic field population. Tightening would reject such proposals, and there is no deterministic recovery/retry path to get that information back from the model (mutations go directly to applyValidatedProposal → graph persistence with no re-attempt mechanism).
|
||||
|
||||
---
|
||||
|
||||
## 57J.84 UNDER CHOSEN CONTRACT
|
||||
|
||||
If structuralActionRequired were required for any meaningful mutation:
|
||||
**REJECT because action declaration missing**
|
||||
|
||||
If answerMeaning were required when userSupportedMeaning should be populated:
|
||||
Also applicable, but the stronger issue is structuralActionRequired — that's the direct gate on mutations.
|
||||
|
||||
### Would useful £2m/year + lease-basis information be discarded?
|
||||
**YES** — The proposal contains structured data (value=2,000,000, unit=GBP/year, description with "lease elimination") embedded in a new reported_claim node and supports edge. Rejecting the proposal discards this entire piece of evidence from the graph.
|
||||
|
||||
### Does a deterministic recovery/retry path currently exist?
|
||||
**NO** — The mutation applies directly via `applyValidatedProposal` → `applyMutation()` with no retry mechanism. Once rejected, there's no bounded repair loop or re-attempt path that would ask the model to repopulate semantic fields while preserving the mutation.
|
||||
|
||||
---
|
||||
|
||||
## SMALLEST NEXT BOUNDARY
|
||||
|
||||
**State the one missing capability required before tightening becomes safe:**
|
||||
|
||||
A deterministic recovery/retry path for proposals with meaningful mutations but unpopulated semantic/action fields. Specifically: when the validator rejects a proposal containing hasMeaningfulChange=true but answerMeaning=null, the engine must be able to ask the model to populate the semantic/action declarations *without* regenerating the entire mutation (or provide a mechanism to recover the structurally-represented meaning for downstream use). Without this capability, tightening creates information loss rather than contract compliance.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The null/null/mutation path is a transition compatibility gap, not an intended feature. It persists because:
|
||||
1. Schema allows nullable fields for backward compatibility during structured population improvement
|
||||
2. Validator rules were intentionally scoped to only reject when meaning IS populated (avoiding over-rejection)
|
||||
3. The model produces useful data through this path (57J.84: £2m/year on reported_claim node)
|
||||
4. Tightening without a recovery path would discard that data
|
||||
|
||||
The architecture should keep this path open until deterministic recovery/retry is in place, then tighten with minimal impact to live proposals containing meaningful structural changes.
|
||||
Reference in New Issue
Block a user