Files

T

robbond a40a3e343e experiment: diagnose null semantic mutation path

2026-08-12 13:19:37 +01:00

15 KiB

Raw Blame History

Experiment 57J.85 — Null Semantic/Action Architecture Diagnosis (Read-Only)

Branch: feature/semantic-action-contract-v0.23 Starting HEAD: eaf3194 (experiment: observe direct meaning/action fields on anchored update)

Objective

Diagnose why answerMeaning=null + structuralActionRequired=null + meaningful mutation is accepted through the current pipeline, and whether this compatibility path should remain open.

Read-only diagnosis. No code changes. No Ollama calls. No live API calls.

Context Files Read

docs/current-handoff.md (handoff state through 57J.84)
docs/experiment-57j84.md (the live case: null meaning + null action + £2m/year mutation accepted)
lib/graph/schema.js (schema truth for all relevant fields)
lib/graph/prompt-builder.js (current HEAD — prompt contract rules)
lib/graph/utils.js (validator logic at line 868+)
lib/graph/apply-proposal.js (validation/parsing section: lines 2937–3146, applyValidatedProposal entry at 3182)

PROMPT CONTRACT TRACE

userSupportedMeaning required on every answer?

CONDITIONAL — Required when you have semantic intent that requires graph progress (rule #6). The prompt says "If answerMeaning.userSupportedMeaning contains consequential information... you MUST express its effect through structural mutation." It also has rules 26–31 governing how to populate userSupportedMeaning when present. However, the prompt does not say "you MUST always populate userSupportedMeaning" — it leaves open the possibility of answerMeaning=null when the answer contains no user-supported meaning that requires graph progress (rule #14 in Additional Guidance: "If rule #6 does not apply... return empty arrays").

structuralActionRequired required on every proposal?

CONDITIONAL — Required when userSupportedMeaning is populated (Declaration Rule section: "When answerMeaning.userSupportedMeaning is populated you MUST set structuralActionRequired to match what your proposal outputs"). However, when answerMeaning=null or userSupportedMeaning is null/unpopulated, the prompt does not explicitly require structuralActionRequired. The contract says it's a declaration tied to semantic intent.

Meaningful mutation + answerMeaning=null explicitly permitted?

AMBIGUOUS — The prompt implies that if rule #6 doesn't apply (no consequential user-supported meaning), the model should return empty arrays with null meaning. But a meaningful mutation with null meaning falls in no explicit category: not rule #6 (which requires userSupportedMeaning to be populated), and not "no semantic intent" (since there's clearly semantic content). The prompt silently allows this combination through omission.

Meaningful mutation + structuralActionRequired=null explicitly permitted?

AMBIGUOUS — Same reasoning as above. When answerMeaning is null, the Declaration Rule does not trigger, so structuralActionRequired is unmentioned for this case.

SCHEMA TRUTH

From lib/graph/schema.js:

answerMeaning

answerMeaningSchema.nullable().default(null)
  └── userSupportedMeaning: z.string().min(1)          [REQUIRED within object]
  └── possibleInference:   z.string().nullable().optional() [OPTIONAL/NULLABLE, defaults to null via Zod]
  └── supportCategory:     z.enum(...).nullable().optional() [OPTIONAL/NULLABLE, defaults to null]
  └── resolutionGuidance:  z.enum(...).nullable().optional() [OPTIONAL/NULLABLE, defaults to null]

Classification: answerMeaning is OPTIONAL (can be omitted from JSON), NULLABLE (can be explicitly null), DEFAULTED (null if absent). userSupportedMeaning is REQUIRED within a non-null object but the outer container is optional.

structuralActionRequired

structuralActionRequired: z.boolean().nullable().optional()

Classification: OPTIONAL, NULLABLE, defaults to null when omitted.

answerMeaning omission/null while proposal schema-valid?

YES — answerMeaningSchema.nullable().default(null) means the entire answerMeaning field can be null and the schema still passes. Even if answerMeaning object is present, only userSupportedMeaning is required within it; possibleInference, supportCategory, and resolutionGuidance are all nullable+optional.

structuralActionRequired omission/null while proposal schema-valid?

YES — z.boolean().nullable().optional() means the field can be omitted entirely or set to null, and Zod will accept it. No schema constraint prevents this.

NULL VS OMISSION BOUNDARY

answerMeaning: NOT DISTINGUISHABLE

Model omits field → Zod defaults to null
Model emits null → stays null
Code sees: answerMeaning === null — both indistinguishable

The information-loss boundary is at Zod schema application. Once parsed, there is no trace of whether the model omitted the field or emitted null.

structuralActionRequired: NOT DISTINGUISHABLE

Model omits field → stays undefined (optional + nullable)
Model emits null → stays null
Code checks both with === null || === undefined — treats them identically

The information-loss boundary is at Zod schema application. Both omission and explicit null converge to an effective "not set" state that the validator cannot differentiate.

VALIDATOR MATRIX (using validateGraphUpdate at HEAD)

Current validation logic in utils.js:

meaningPopulated = !!update.answerMeaning?.userSupportedMeaning;
hasMeaningfulChange = [addedNodes, statusChanged, valueChanged, addedEdges, removedEdges];
fieldAbsent = structuralActionRequired === null || undefined;

// Rule 1: missing field + meaning populated → REJECT
if (fieldAbsent && meaningPopulated) → reject

// Rule 2: true + no mutation → REJECT
if (structuralActionRequired === true && !hasMeaningfulChange) → reject

// Rule 3: false + mutation → REJECT  
if (structuralActionRequired === false && hasMeaningfulChange) → reject

// Rule 4: no mutation + absent field + no meaning → REJECT ("Update contains no meaningful change")
if (!hasMeaningfulChange && fieldAbsent && !meaningPopulated) → reject

A: populated meaning + true + mutation

PASS — All three rules are satisfied (meaningPopulated=true doesn't trigger rule 1 because fieldAbsent=false; rules 2 and 3 don't apply because structuralActionRequired===true AND hasMeaningfulChange=true; rule 4 doesn't apply because hasMeaningfulChange=true).

B: populated meaning + false + no mutation

PASS — All checks pass. Rule 1 doesn't trigger (fieldAbsent=false). Rules 2/3 don't trigger (true is not false). Rule 4 requires !hasMeaningfulChange AND fieldAbsent AND !meaningPopulated — but meaningPopulated=true, so rule 4 doesn't fire.

C: populated meaning + null action

REJECT — Rule 1 fires: meaningPopulated=true && fieldAbsent=true → "structuralActionRequired must be present when userSupportedMeaning is populated".

D: null meaning + null action + mutation

PASS — Rule 1 doesn't trigger (meaningPopulated=false). Rules 2/3 don't trigger (fieldAbsent=true, not === true/false). Rule 4 doesn't trigger (hasMeaningfulChange=true). Escape hatch.

E: null meaning + null action + no mutation

REJECT — Rule 4 fires: !hasMeaningfulChange=true && fieldAbsent=true && !meaningPopulated=true → "Update contains no meaningful change".

F: null meaning + true + mutation

PASS — No rules fire. Rules 1/3 check structuralActionRequired===true (rule 3 fails because hasMeaningfulChange=true). Rule 4 doesn't trigger (hasMeaningfulChange=true). The true declaration is inconsistent with null meaning but not explicitly checked.

G: null meaning + false + no mutation

PASS — No rules fire. Rules 1/2 don't apply for the same reasons as F and E respectively. Rule 4 doesn't trigger (hasMeaningfulChange=false AND fieldAbsent=true AND !meaningPopulated=true... wait, that's rule 4 which should REJECT).

Correction: Rule 4 fires: !hasMeaningfulChange && fieldAbsent && !meaningPopulated → "Update contains no meaningful change". REJECT.

H: null meaning + false + mutation

REJECT — Rule 3 fires: structuralActionRequired===false && hasMeaningfulChange=true → "structuralActionRequired is false but proposal contains meaningful mutations".

I: null meaning + true + no mutation

REJECT — Rule 2 fires: structuralActionRequired===true && !hasMeaningfulChange=true → "structuralActionRequired is true but proposal contains no graph mutation".

57J.84 PATH CLASSIFICATION

Classification: B — TRANSITION COMPATIBILITY

Why

The null/nullable fields are schema-legal and the validator rules are carefully scoped to only reject when meaning IS populated (rule 1) or when the boolean is explicitly true/false but contradicts mutation state (rules 2/3). The specific combination of answerMeaning=null + structuralActionRequired=null + meaningful mutation falls through all rules because:

Rule 1 requires meaningPopulated=true — not met
Rules 2/3 require structuralActionRequired to be ===true or ===false — fieldAbsent=true prevents this
Rule 4 requires !hasMeaningfulChange — not met

This is not accidental (C would mean the rules were written carelessly), because the rules are explicitly structured with these exact conditions. It's not first-class design (A) because no prompt rule encourages it, and no architecture document describes it as a feature. It exists because during transition, nullable fields remained for compatibility while structured field population was incomplete — tightening would reject live proposals that contain useful data.

Schema-valid: YES

Zod schema accepts null/absent for both answerMeaning and structuralActionRequired.

Prompt-compliant: AMBIGUOUS

The prompt does not explicitly permit this path (no rule says "you may produce mutation without semantic declarations"), but it also doesn't explicitly forbid it — the prompt's constraints on structuralActionRequired only activate when userSupportedMeaning is populated. This creates a silent gap.

Validator-accepted: YES

All four validator rules are satisfied for the null/null/mutation case.

Architecturally intended: TRANSITION ONLY

The combination exists because of incomplete transition, not deliberate design. The field-absence rule (rule 1) only triggers when meaning is populated — intentionally limiting its scope during transition.

Deterministic accountability: SHAPE ONLY

What IS validated: node/edge structure validity, ID consistency, size limit, no-op guard (when meaning absent and no mutation). What is NOT validated for this path: any semantic intent check, any structural action declaration check, any answer-meaning alignment check. The validator confirms shape only — that addedNodes has correct fields, that edges reference valid nodes, etc.

What is still verified:

Schema structure of all nodes/edges in the proposal
No duplicate IDs against existing graph
No update to non-existent nodes
Size < 100KB
If structuralActionRequired===true/false: contradiction with actual mutation state (rules 2/3)
If meaningPopulated+fieldAbsent: rejection (rule 1)
If no mutation + fieldAbsent + !meaningPopulated: rejection (rule 4)

MIGRATION READINESS

answerMeaning population reliability: PARTIAL

57J.84 proves the model can produce null when it should populate meaningful content (it implicitly captured meaning via structure). However, other experiments show the model can populate userSupportedMeaning in some cases. Reliability is proven to be inconsistent — sometimes populated, sometimes null for consequential answers.

structuralActionRequired population reliability: PARTIAL

57J.84 proves null production alongside meaningful mutation. 57J.71 proved true+mutation is possible (same model). But the consistent null production on the "meaning via structure" path means population is not reliable when meaning flows through implicit representation.

true + mutation path: PROVEN

57J.71 demonstrated the model can produce structuralActionRequired=true with meaningful mutation in a single pass. The validator accepts it cleanly. But this only works when answerMeaning IS populated — proving that the model can follow the declaration rule WHEN triggered.

false + no-op path: PROVEN

Multiple experiments show the validator correctly accepts and rejects false+no-op combinations. The contract is clean for this path.

null/null + mutation still exercised live: YES

57J.84 is direct evidence — £2m/year savings data was structurally preserved via a new reported_claim node with supports edge, all semantic/action fields were null, and the update was accepted at HTTP 200.

ARCHITECTURAL CHOICE

Choice: A — KEEP NULL TRANSITION PATH FOR NOW

Why

Population reliability/recovery is not strong enough to tighten safely. The 57J.84 case demonstrates that meaningful, consequential data (£2m/year + lease-basis) flows through this path successfully — it IS preserved in the graph even without semantic field population. Tightening would reject such proposals, and there is no deterministic recovery/retry path to get that information back from the model (mutations go directly to applyValidatedProposal → graph persistence with no re-attempt mechanism).

57J.84 UNDER CHOSEN CONTRACT

If structuralActionRequired were required for any meaningful mutation: REJECT because action declaration missing

If answerMeaning were required when userSupportedMeaning should be populated: Also applicable, but the stronger issue is structuralActionRequired — that's the direct gate on mutations.

Would useful £2m/year + lease-basis information be discarded?

YES — The proposal contains structured data (value=2,000,000, unit=GBP/year, description with "lease elimination") embedded in a new reported_claim node and supports edge. Rejecting the proposal discards this entire piece of evidence from the graph.

Does a deterministic recovery/retry path currently exist?

NO — The mutation applies directly via applyValidatedProposal → applyMutation() with no retry mechanism. Once rejected, there's no bounded repair loop or re-attempt path that would ask the model to repopulate semantic fields while preserving the mutation.

SMALLEST NEXT BOUNDARY

State the one missing capability required before tightening becomes safe:

A deterministic recovery/retry path for proposals with meaningful mutations but unpopulated semantic/action fields. Specifically: when the validator rejects a proposal containing hasMeaningfulChange=true but answerMeaning=null, the engine must be able to ask the model to populate the semantic/action declarations without regenerating the entire mutation (or provide a mechanism to recover the structurally-represented meaning for downstream use). Without this capability, tightening creates information loss rather than contract compliance.

Conclusion

The null/null/mutation path is a transition compatibility gap, not an intended feature. It persists because:

Schema allows nullable fields for backward compatibility during structured population improvement
Validator rules were intentionally scoped to only reject when meaning IS populated (avoiding over-rejection)
The model produces useful data through this path (57J.84: £2m/year on reported_claim node)
Tightening without a recovery path would discard that data

The architecture should keep this path open until deterministic recovery/retry is in place, then tighten with minimal impact to live proposals containing meaningful structural changes.

15 KiB Raw Blame History Unescape Escape