291 lines
17 KiB
Markdown
291 lines
17 KiB
Markdown
# Experiment 57J.67 — `structuralActionRequired` Contract Semantics Finalized
|
|
|
|
**Branch:** `feature/selected-question-contract-v0.22`
|
|
**Starting HEAD:** `9425e7b` (experiment: define semantic action contract)
|
|
|
|
## Objective
|
|
|
|
Settle the final ambiguity from Experiment 57J.66:
|
|
|
|
> **Is `structuralActionRequired` a strict consistency contract or merely advisory intent?**
|
|
|
|
This task settles that question and produces a complete, unambiguous v0.23 implementation contract.
|
|
|
|
**Classification: READ-ONLY ARCHITECTURE DECISION. No production code changed.**
|
|
|
|
---
|
|
|
|
## Part 1 — Boolean Definition Chosen
|
|
|
|
### Comparison
|
|
|
|
**Definition A — EXACT STRUCTURAL CLAIM** (chosen):
|
|
```
|
|
true → proposal contains meaningful mutation (hasMeaningfulChange === true)
|
|
false → no meaningful mutation is needed (hasMeaningfulChange === false)
|
|
Declaration matches proposal shape exactly.
|
|
```
|
|
|
|
**Definition B — MINIMUM-ACTION CLAIM** (rejected):
|
|
```
|
|
true → at least some structural mutation occurs
|
|
false → no minimum required, but extra mutation is allowed
|
|
Declaration is a floor, not a boundary.
|
|
```
|
|
|
|
### Decision: EXACT STRUCTURAL CLAIM
|
|
|
|
**Why:**
|
|
|
|
1. **Field name semantics.** `structuralActionRequired` uses the word "required" — which denotes necessity, not suggestion. Under Definition B, `false` means "no *minimum* action required" which is awkward and contradicts the natural reading of "action [is] required = false."
|
|
|
|
2. **Full determinism.** Definition A produces exactly four deterministic outcomes (one per contradiction pair) with no ambiguity about what passes or fails. Definition B requires distinguishing "more than necessary but harmless" from "contract fulfilled," which introduces softness into a field designed for hard validation.
|
|
|
|
3. **Prevents the most damaging error class.** `false + mutation` under exact claim rejects a model that declared "no structural change needed" while producing meaningful mutations — either it misunderstood the answer or over-produced structure. Under advisory semantics, this goes undetected and becomes silent degradation.
|
|
|
|
4. **57J.66's advisory recommendation was premature.** It was made without resolving whether false + mutation genuinely harms the contract. Analysis shows it does: a declaration that "no action is required" followed by actual structural production creates an inconsistency that semantic interpretation cannot resolve deterministically.
|
|
|
|
---
|
|
|
|
## Part 2 — Contradiction Matrix (Exact Structural Claim)
|
|
|
|
| `structuralActionRequired` | hasMeaningfulChange | Outcome | Rationale |
|
|
|---|---|---|---|
|
|
| true | true | **PASS** | Declaration fulfilled. Action declared and produced. Contract satisfied. |
|
|
| true | false | **REJECT** | Model claims action is required but produces zero mutations. Either the model misunderstood the answer's implications, or failed to execute on its own declaration. Deterministic error: contract violation. |
|
|
| false | false | **PASS** | Intentional no-op. Model explicitly declared that no structural action is needed, and zero mutations confirm the declaration. Deterministic code trusts this structured declaration. |
|
|
| false | true | **REJECT** | Declaration says "no structural change needed" but proposal produces meaningful changes. Under exact claim, this is inconsistent — the model either misunderstood the user's meaning (claimed no action when one was needed) or over-produced structure beyond what the answer warrants. This is not harmless extra progress; it is a broken contract between declaration and output shape. |
|
|
|
|
**Why false + mutation rejects without being advisory:** If the model truly believed the user's supported meaning didn't require any structural change, then producing meaningful mutations means either: (a) the model changed its mind mid-production without updating `structuralActionRequired`, or (b) the model misunderstood what "no action required" means. In either case, the inconsistency is actionable by deterministic validation — the field exists to surface exactly this class of error.
|
|
|
|
---
|
|
|
|
## Part 3 — What `false` Actually Means
|
|
|
|
### Chosen: A
|
|
|
|
```
|
|
The user's supported meaning is already fully represented in graph state,
|
|
so no graph mutation is needed.
|
|
```
|
|
|
|
**Why A over B:** Option B ("The proposal intentionally performs no graph progress for this answer") is too narrow — it only covers cases where the model *chooses* to do nothing. It excludes the primary case: semantic agreement with existing graph state. Option A covers both the intentional no-op (the model evaluates and finds nothing to change) and semantic agreement (an equivalent unresolved uncertainty already exists).
|
|
|
|
**Why A over C:** Option C ("Either A or another legitimate no-op case") is intentionally vague and would require semantic parsing at validation time to determine which sub-case applies — defeating the purpose of a deterministic boolean field.
|
|
|
|
Option A is precise: when `structuralActionRequired = false`, the model asserts that **the user's supported meaning does not necessitate any graph change**. This assertion can be either true or false (semantic correctness is unprovable), but the declaration itself is deterministically checkable against proposal shape.
|
|
|
|
---
|
|
|
|
## Part 4 — Populated Meaning + False + Empty
|
|
|
|
```
|
|
userSupportedMeaning: populated (non-empty string)
|
|
structuralActionRequired: false
|
|
hasMeaningfulChange: false
|
|
```
|
|
|
|
### Deterministic Validation: PASS
|
|
|
|
**Rationale:** The model explicitly declared that no structural action is needed (`false`) and the proposal confirms zero mutations. Deterministic code verifies contract consistency — declaration matches reality. No semantic parsing of the userSupportedMeaning content is required or performed.
|
|
|
|
### Does this prove the model's semantic judgment was correct?
|
|
|
|
**NO**
|
|
|
|
**What it proves:**
|
|
1. The model made an explicit structural intent declaration (no silence).
|
|
2. The proposal shape matches that declaration (consistency verified).
|
|
3. The model intentionally chose a no-op path with populated meaning extraction.
|
|
|
|
**What it does NOT prove:**
|
|
- Whether the user's supported meaning genuinely didn't warrant graph mutation.
|
|
- Whether useful graph structure was omitted.
|
|
- Whether the answer warranted more than zero mutations.
|
|
|
|
The boolean field's purpose is precisely to avoid requiring semantic proof — it delegates semantic judgment to the model and only checks structural consistency.
|
|
|
|
---
|
|
|
|
## Part 5 — Populated Meaning + False + Mutation
|
|
|
|
```
|
|
userSupportedMeaning: populated (non-empty string)
|
|
structuralActionRequired: false
|
|
hasMeaningfulChange: true
|
|
```
|
|
|
|
### Deterministic Validation: REJECT
|
|
|
|
**Why (contract terms):** Under exact structural claim, `false` means "no meaningful mutation is needed." The presence of meaningful mutations contradicts this declaration. The model either:
|
|
- Claimed no action was needed but then produced structure anyway (mid-production state change), or
|
|
- Misunderstood the user's meaning and over-produced beyond what the answer warranted.
|
|
|
|
This is not a case of "more progress is harmless." A field named `structuralActionRequired` must be truthful about its own claim: if it says `false`, the proposal should contain zero mutations. Any deviation breaks the contract deterministically — no semantic interpretation needed.
|
|
|
|
**Note:** Under 57J.66's advisory recommendation, this would have been accepted with a diagnostic note. This experiment rejects that approach because:
|
|
- It defeats the purpose of having a boolean field with crisp semantics.
|
|
- A model can always produce "more" structure regardless of what it declares, making `false` meaningless as a signal.
|
|
- The inconsistency is actionable by validation and should be surfaced to the developer/model for correction.
|
|
|
|
---
|
|
|
|
## Part 6 — Missing/Null Transition Rule
|
|
|
|
### Chosen: A
|
|
|
|
```
|
|
missing/null + populated userSupportedMeaning → reject
|
|
missing/null + no userSupportedMeaning → retain existing behaviour
|
|
```
|
|
|
|
**Why A over B:** Policy B (always retain existing behavior for missing/null) creates a silent degradation window during transition. Any proposal with populated `userSupportedMeaning` and missing `structuralActionRequired` would bypass the new contract entirely, allowing noncompliant outputs to pass validation until the prompt change ships.
|
|
|
|
**Why A over C:** While the field should ultimately be mandatory on every proposal (C), enforcing it at the validator level during transition is premature without the prompt requiring it first. Policy A provides a minimal safety net: the contract activates whenever there is meaningful content that could justify structural action. The transition to full mandatory enforcement (C) happens when the prompt change ships in v0.23.
|
|
|
|
**Specific transitions:**
|
|
- `structuralActionRequired` absent + `userSupportedMeaning` populated → **REJECT** ("structuralActionRequired must be present when userSupportedMeaning is populated")
|
|
- `structuralActionRequired` null + `userSupportedMeaning` populated → **REJECT** (same as absent)
|
|
- `structuralActionRequired` absent/null + `userSupportedMeaning` not populated → existing behavior ("Update contains no meaningful change" if zero mutations; pass if mutations present)
|
|
|
|
---
|
|
|
|
## Part 7 — Legacy No-Op Guard Status
|
|
|
|
### Decision: REPLACED BY structuralActionRequired CONTRACT
|
|
|
|
**Rationale:** The existing legacy guard rejects any proposal where `userSupportedMeaning` is populated but `hasMeaningfulChange` is false. Under the new exact contract:
|
|
- When `structuralActionRequired = false` + zero mutations → this should PASS as a valid intentional no-op (the model declared no action needed, and it produced none).
|
|
- The legacy guard would incorrectly reject this valid case.
|
|
|
|
**Implementation approach:** The legacy guard's semantic-only-no-op rejection (`"answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"`) is replaced by the `structuralActionRequired` contract check:
|
|
- If `structuralActionRequired === false` → skip legacy guard (intentional no-op is valid).
|
|
- If `structuralActionRequired === true` → it would already be rejected by the `true + no mutation` rule.
|
|
- If `structuralActionRequired` is missing/null + populated meaning → reject for field absence, not for structural mismatch.
|
|
|
|
**Result:** The legacy guard's specific semantic-no-op rejection is removed from the new-contract path and effectively replaced by the `structuralActionRequired` contract. Its generic "no meaningful change" rejection remains for cases where `userSupportedMeaning` is null/non-populated.
|
|
|
|
---
|
|
|
|
## Part 8 — Prompt Wording Boundary
|
|
|
|
### Minimum Semantic Instructions (2 sentences):
|
|
|
|
1. **"Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)."**
|
|
|
|
2. **"Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed."**
|
|
|
|
These two sentences are sufficient because:
|
|
- Sentence 1 gives an *output-based* criterion (truth = proposal has mutations), which the model can verify against its own output without requiring semantic analysis.
|
|
- Sentence 2 gives a *semantic* criterion for false only (the user's meaning is already in the graph), which is the legitimate case for no-op.
|
|
- No third action taxonomy is introduced; the boolean maps directly to `hasMeaningfulChange`.
|
|
- The prompt does not need to explain every edge case — deterministic validation handles those at the contract level.
|
|
|
|
---
|
|
|
|
## Part 9 — Exact v0.23 Implementation Contract
|
|
|
|
```
|
|
Field location: top-level in graphUpdateSchema (lib/graph/schema.js line ~184)
|
|
Type: z.boolean().nullable().optional()
|
|
Nullable: YES during transition; becomes mandatory once prompt ships
|
|
Meaning of true: The model declares that the user's supported meaning requires meaningful graph mutation
|
|
Meaning of false: The user's supported meaning is already fully represented in existing graph state, so no graph mutation is needed
|
|
true + mutation: PASS — declaration fulfilled
|
|
true + no mutation: REJECT — contract violation; "structuralActionRequired is true but proposal contains no graph mutation"
|
|
false + no mutation: PASS — intentional no-op; declaration matches zero mutations
|
|
false + mutation: REJECT — contract violation; declaration contradicts output shape
|
|
missing + populated meaning: REJECT — field required when userSupportedMeaning is populated
|
|
missing + no meaning: RETAIN existing "no meaningful change" behavior (unchanged)
|
|
legacy no-op guard: REPLACED BY structuralActionRequired CONTRACT for new-contract path; generic non-meaning rejection retained
|
|
```
|
|
|
|
---
|
|
|
|
## Required Regression Test Matrix
|
|
|
|
1. **true + meaningful mutation** → PASS. Validator confirms declaration matches mutations present.
|
|
2. **true + zero mutation** → REJECT. Error: "structuralActionRequired is true but proposal contains no graph mutation."
|
|
3. **false + zero mutation** → PASS. Valid intentional no-op with populated userSupportedMeaning.
|
|
4. **false + meaningful mutation** → REJECT. Error: "structuralActionRequired is false but proposal contains meaningful mutations."
|
|
5. **null + populated userSupportedMeaning** → REJECT. Error: "structuralActionRequired must be present when userSupportedMeaning is populated."
|
|
6. **null + no userSupportedMeaning** → PASS/REJECT based on hasMeaningfulChange (existing behavior preserved).
|
|
7. **populated meaning + false does not imply semantic truth was proven** → documented in test as explicit assertion: validation passes but this proves only contract consistency, not semantic correctness.
|
|
8. **existing hasMeaningfulChange logic unchanged** → all existing mutation-detection tests pass identically (verified against current 64-test suite).
|
|
9. **supportCategory remains independent of structuralActionRequired** → no cross-dependency; supportCategory = null with any structuralActionRequired value is valid.
|
|
10. **no keyword/synonym/raw-English logic added** → validation compares boolean against hasMeaningfulChange boolean result only. Zero semantic parsing in the contract check.
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
**A — Strict exact structural contract**
|
|
|
|
**Why:** `structuralActionRequired` uses "required" which denotes necessity. A boolean named "required" should mean what it says: an action is required (true) or not required (false). The EXACT STRUCTURAL CLAIM provides crisp, deterministic semantics in all four cases, prevents the most damaging error class (false + mutation), and enables intentional no-ops as a valid contract-consistent path rather than requiring semantic proof.
|
|
|
|
This does NOT require:
|
|
- New semantic taxonomy: NO
|
|
- Keyword/synonym logic: NO
|
|
- Provider-specific behavior
|
|
|
|
This preserves:
|
|
- Provider-agnostic design: YES
|
|
- Existing hasMeaningfulChange semantics: unchanged (only new boolean check added)
|
|
- supportCategory independence: maintained
|
|
|
|
---
|
|
|
|
## Convergence
|
|
|
|
**READY FOR BOUNDED IMPLEMENTATION: YES**
|
|
|
|
All previously ambiguous decisions from 57J.66 are now settled:
|
|
- Field location: top-level graphUpdateSchema
|
|
- Semantics: EXACT STRUCTURAL CLAIM (strict, not advisory)
|
|
- Null transition: Policy A (reject when meaning populated, retain otherwise)
|
|
- Contradiction matrix: all four cases fully specified
|
|
- Legacy guard: replaced by contract for new path
|
|
|
|
---
|
|
|
|
## Required Implementation Boundary (if READY)
|
|
|
|
### Files changed:
|
|
1. `lib/graph/schema.js` — add `structuralActionRequired` to `graphUpdateSchema` (line ~184), as `z.boolean().nullable().optional()`
|
|
2. `lib/graph/utils.js` — in `validateGraphUpdate()`, add exact structural contract check alongside existing hasMeaningfulChange logic; replace semantic-only-no-op rejection with contract-based logic
|
|
3. `lib/graph/prompt-builder.js` — add field to "Required JSON Field Names" list, to "Required Shapes" section, and add two prompt sentences under "Proposal Rules"
|
|
4. `tests/graph/utils.test.js` — 6 new tests for the contract matrix + regression assertions
|
|
|
|
### New tests:
|
|
1. true + meaningful mutation → pass;
|
|
2. true + zero mutation → reject with specific error message;
|
|
3. false + zero mutation (with populated userSupportedMeaning) → pass (intentional no-op);
|
|
4. false + meaningful mutation → reject with specific error message;
|
|
5. null + populated userSupportedMeaning → reject (field required);
|
|
6. null + no userSupportedMeaning → retain existing "no meaningful change" rejection;
|
|
7. documented assertion: PASS on populated meaning + false does not prove semantic correctness — only contract consistency;
|
|
8. existing hasMeaningfulChange semantics remain unchanged for non-contract paths;
|
|
9. supportCategory remains independent of structuralActionRequired (any combination valid);
|
|
10. no keyword/synonym/raw-English logic added anywhere in contract check.
|
|
|
|
### Scope exclusions (intentionally out of scope):
|
|
- retry/regeneration
|
|
- mutation enums or categories
|
|
- scoring
|
|
- evidence linkage
|
|
- provider-specific behaviour
|
|
- semantic similarity detection
|
|
- keyword classifiers
|
|
- changing `hasMeaningfulChange` computation itself
|
|
- changing `supportCategory` behavior
|
|
|
|
### What this intentionally leaves unresolved:
|
|
- Whether the strict false/mutation path should eventually log a diagnostic before rejecting
|
|
- Whether `structuralActionRequired` should eventually carry additional fields (e.g., `structuralReason`)
|
|
- Whether the prompt rule needs refinement based on live model behavior under the contract
|
|
- Migration of existing prompts that reference the old schema field list
|
|
|
|
---
|
|
|
|
**Classification:** READ-ONLY ARCHITECTURE DECISION. No production code changed. No Ollama calls. No tests modified. All decisions settled for bounded implementation.
|