diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 3e30baa..f6ad0e5 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1228,4 +1228,25 @@ The contract semantics are **ADVISORY** — the boolean is a minimum intent decl **READY FOR BOUNVED IMPLEMENTATION: YES.** All decisions resolved: field location, strict/advisory semantics, null transition, contradiction matrix. No remaining ambiguity for bounded implementation. -Full record in `docs/experiment-57j66.md`. \ No newline at end of file +Full record in `docs/experiment-57j66.md`. + +--- + +### Experiment 57J.67 — semanticActionRequired Contract Semantics Finalized + +**Classification: A — SEMANTICS SETTLED.** Resolved the final ambiguity from 57J.66: is the boolean an *exact structural claim* (strict contract) or a *minimum-action claim* (advisory)? **Decision: EXACT STRUCTURAL CLAIM.** The field name "structuralActionRequired" semantically implies necessity, not suggestion. Definition A provides cleaner semantics, fully deterministic validation in all four cases, and prevents the most damaging error class (model declares no action but produces structure). Advisory (57J.66's recommendation) is rejected: `false + mutation` violates contract consistency — if the model declares "no structural action required" but produces meaningful mutations, it has either misunderstood the answer or over-produced unnecessary structure. This is not harmless. + +- **Boolean definition:** EXACT STRUCTURAL CLAIM (Definition A). true = meaningful mutation present; false = no meaningful mutation needed. +- **true + mutation:** PASS. true + no mutation: REJECT. false + no mutation: PASS. false + mutation: REJECT (under exact claim). +- **False semantics:** Option A — "The user's supported meaning is already fully represented in graph state, so no graph mutation is needed." This covers semantic agreement with existing state and other legitimate no-op cases. +- **Populated meaning + false + empty:** PASS. Does NOT prove semantic correctness (NO). Proves only: model explicitly declared intent + declaration matches zero mutations = contract consistent. Semantic truth remains unproven. +- **Populated meaning + false + mutation:** REJECT under exact claim. Declaration says "no structural change needed" but proposal contradicts by producing meaningful changes. Deterministic validation catches this inconsistency. +- **Transition rule:** A — missing/null + populated userSupportedMeaning → reject; missing/null + no meaning → retain existing behavior. Mandatory on every proposal (C) deferred to prompt-only enforcement. +- **Legacy no-op guard:** REMAINS ONLY FOR LEGACY/MISSING FIELD. Under exact contract, `false + zero mutation` is valid intentional no-op — legacy "semantic-only no-op rejection" would incorrectly block it. Legacy guard stays for when structuralActionRequired is absent. +- **Prompt wording (2 sentences):** + 1. "Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)." + 2. "Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed." + +**Final v0.23 contract:** top-level field in `graphUpdateSchema`, boolean | nullable, exact structural semantics. Ready for bounded implementation. + +New branch: `feature/semantic-action-contract-v0.23` \ No newline at end of file diff --git a/docs/experiment-57j67.md b/docs/experiment-57j67.md new file mode 100644 index 0000000..1af8b1a --- /dev/null +++ b/docs/experiment-57j67.md @@ -0,0 +1,290 @@ +# Experiment 57J.67 — `structuralActionRequired` Contract Semantics Finalized + +**Branch:** `feature/selected-question-contract-v0.22` +**Starting HEAD:** `9425e7b` (experiment: define semantic action contract) + +## Objective + +Settle the final ambiguity from Experiment 57J.66: + +> **Is `structuralActionRequired` a strict consistency contract or merely advisory intent?** + +This task settles that question and produces a complete, unambiguous v0.23 implementation contract. + +**Classification: READ-ONLY ARCHITECTURE DECISION. No production code changed.** + +--- + +## Part 1 — Boolean Definition Chosen + +### Comparison + +**Definition A — EXACT STRUCTURAL CLAIM** (chosen): +``` +true → proposal contains meaningful mutation (hasMeaningfulChange === true) +false → no meaningful mutation is needed (hasMeaningfulChange === false) +Declaration matches proposal shape exactly. +``` + +**Definition B — MINIMUM-ACTION CLAIM** (rejected): +``` +true → at least some structural mutation occurs +false → no minimum required, but extra mutation is allowed +Declaration is a floor, not a boundary. +``` + +### Decision: EXACT STRUCTURAL CLAIM + +**Why:** + +1. **Field name semantics.** `structuralActionRequired` uses the word "required" — which denotes necessity, not suggestion. Under Definition B, `false` means "no *minimum* action required" which is awkward and contradicts the natural reading of "action [is] required = false." + +2. **Full determinism.** Definition A produces exactly four deterministic outcomes (one per contradiction pair) with no ambiguity about what passes or fails. Definition B requires distinguishing "more than necessary but harmless" from "contract fulfilled," which introduces softness into a field designed for hard validation. + +3. **Prevents the most damaging error class.** `false + mutation` under exact claim rejects a model that declared "no structural change needed" while producing meaningful mutations — either it misunderstood the answer or over-produced structure. Under advisory semantics, this goes undetected and becomes silent degradation. + +4. **57J.66's advisory recommendation was premature.** It was made without resolving whether false + mutation genuinely harms the contract. Analysis shows it does: a declaration that "no action is required" followed by actual structural production creates an inconsistency that semantic interpretation cannot resolve deterministically. + +--- + +## Part 2 — Contradiction Matrix (Exact Structural Claim) + +| `structuralActionRequired` | hasMeaningfulChange | Outcome | Rationale | +|---|---|---|---| +| true | true | **PASS** | Declaration fulfilled. Action declared and produced. Contract satisfied. | +| true | false | **REJECT** | Model claims action is required but produces zero mutations. Either the model misunderstood the answer's implications, or failed to execute on its own declaration. Deterministic error: contract violation. | +| false | false | **PASS** | Intentional no-op. Model explicitly declared that no structural action is needed, and zero mutations confirm the declaration. Deterministic code trusts this structured declaration. | +| false | true | **REJECT** | Declaration says "no structural change needed" but proposal produces meaningful changes. Under exact claim, this is inconsistent — the model either misunderstood the user's meaning (claimed no action when one was needed) or over-produced structure beyond what the answer warrants. This is not harmless extra progress; it is a broken contract between declaration and output shape. | + +**Why false + mutation rejects without being advisory:** If the model truly believed the user's supported meaning didn't require any structural change, then producing meaningful mutations means either: (a) the model changed its mind mid-production without updating `structuralActionRequired`, or (b) the model misunderstood what "no action required" means. In either case, the inconsistency is actionable by deterministic validation — the field exists to surface exactly this class of error. + +--- + +## Part 3 — What `false` Actually Means + +### Chosen: A + +``` +The user's supported meaning is already fully represented in graph state, +so no graph mutation is needed. +``` + +**Why A over B:** Option B ("The proposal intentionally performs no graph progress for this answer") is too narrow — it only covers cases where the model *chooses* to do nothing. It excludes the primary case: semantic agreement with existing graph state. Option A covers both the intentional no-op (the model evaluates and finds nothing to change) and semantic agreement (an equivalent unresolved uncertainty already exists). + +**Why A over C:** Option C ("Either A or another legitimate no-op case") is intentionally vague and would require semantic parsing at validation time to determine which sub-case applies — defeating the purpose of a deterministic boolean field. + +Option A is precise: when `structuralActionRequired = false`, the model asserts that **the user's supported meaning does not necessitate any graph change**. This assertion can be either true or false (semantic correctness is unprovable), but the declaration itself is deterministically checkable against proposal shape. + +--- + +## Part 4 — Populated Meaning + False + Empty + +``` +userSupportedMeaning: populated (non-empty string) +structuralActionRequired: false +hasMeaningfulChange: false +``` + +### Deterministic Validation: PASS + +**Rationale:** The model explicitly declared that no structural action is needed (`false`) and the proposal confirms zero mutations. Deterministic code verifies contract consistency — declaration matches reality. No semantic parsing of the userSupportedMeaning content is required or performed. + +### Does this prove the model's semantic judgment was correct? + +**NO** + +**What it proves:** +1. The model made an explicit structural intent declaration (no silence). +2. The proposal shape matches that declaration (consistency verified). +3. The model intentionally chose a no-op path with populated meaning extraction. + +**What it does NOT prove:** +- Whether the user's supported meaning genuinely didn't warrant graph mutation. +- Whether useful graph structure was omitted. +- Whether the answer warranted more than zero mutations. + +The boolean field's purpose is precisely to avoid requiring semantic proof — it delegates semantic judgment to the model and only checks structural consistency. + +--- + +## Part 5 — Populated Meaning + False + Mutation + +``` +userSupportedMeaning: populated (non-empty string) +structuralActionRequired: false +hasMeaningfulChange: true +``` + +### Deterministic Validation: REJECT + +**Why (contract terms):** Under exact structural claim, `false` means "no meaningful mutation is needed." The presence of meaningful mutations contradicts this declaration. The model either: +- Claimed no action was needed but then produced structure anyway (mid-production state change), or +- Misunderstood the user's meaning and over-produced beyond what the answer warranted. + +This is not a case of "more progress is harmless." A field named `structuralActionRequired` must be truthful about its own claim: if it says `false`, the proposal should contain zero mutations. Any deviation breaks the contract deterministically — no semantic interpretation needed. + +**Note:** Under 57J.66's advisory recommendation, this would have been accepted with a diagnostic note. This experiment rejects that approach because: +- It defeats the purpose of having a boolean field with crisp semantics. +- A model can always produce "more" structure regardless of what it declares, making `false` meaningless as a signal. +- The inconsistency is actionable by validation and should be surfaced to the developer/model for correction. + +--- + +## Part 6 — Missing/Null Transition Rule + +### Chosen: A + +``` +missing/null + populated userSupportedMeaning → reject +missing/null + no userSupportedMeaning → retain existing behaviour +``` + +**Why A over B:** Policy B (always retain existing behavior for missing/null) creates a silent degradation window during transition. Any proposal with populated `userSupportedMeaning` and missing `structuralActionRequired` would bypass the new contract entirely, allowing noncompliant outputs to pass validation until the prompt change ships. + +**Why A over C:** While the field should ultimately be mandatory on every proposal (C), enforcing it at the validator level during transition is premature without the prompt requiring it first. Policy A provides a minimal safety net: the contract activates whenever there is meaningful content that could justify structural action. The transition to full mandatory enforcement (C) happens when the prompt change ships in v0.23. + +**Specific transitions:** +- `structuralActionRequired` absent + `userSupportedMeaning` populated → **REJECT** ("structuralActionRequired must be present when userSupportedMeaning is populated") +- `structuralActionRequired` null + `userSupportedMeaning` populated → **REJECT** (same as absent) +- `structuralActionRequired` absent/null + `userSupportedMeaning` not populated → existing behavior ("Update contains no meaningful change" if zero mutations; pass if mutations present) + +--- + +## Part 7 — Legacy No-Op Guard Status + +### Decision: REPLACED BY structuralActionRequired CONTRACT + +**Rationale:** The existing legacy guard rejects any proposal where `userSupportedMeaning` is populated but `hasMeaningfulChange` is false. Under the new exact contract: +- When `structuralActionRequired = false` + zero mutations → this should PASS as a valid intentional no-op (the model declared no action needed, and it produced none). +- The legacy guard would incorrectly reject this valid case. + +**Implementation approach:** The legacy guard's semantic-only-no-op rejection (`"answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"`) is replaced by the `structuralActionRequired` contract check: +- If `structuralActionRequired === false` → skip legacy guard (intentional no-op is valid). +- If `structuralActionRequired === true` → it would already be rejected by the `true + no mutation` rule. +- If `structuralActionRequired` is missing/null + populated meaning → reject for field absence, not for structural mismatch. + +**Result:** The legacy guard's specific semantic-no-op rejection is removed from the new-contract path and effectively replaced by the `structuralActionRequired` contract. Its generic "no meaningful change" rejection remains for cases where `userSupportedMeaning` is null/non-populated. + +--- + +## Part 8 — Prompt Wording Boundary + +### Minimum Semantic Instructions (2 sentences): + +1. **"Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)."** + +2. **"Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed."** + +These two sentences are sufficient because: +- Sentence 1 gives an *output-based* criterion (truth = proposal has mutations), which the model can verify against its own output without requiring semantic analysis. +- Sentence 2 gives a *semantic* criterion for false only (the user's meaning is already in the graph), which is the legitimate case for no-op. +- No third action taxonomy is introduced; the boolean maps directly to `hasMeaningfulChange`. +- The prompt does not need to explain every edge case — deterministic validation handles those at the contract level. + +--- + +## Part 9 — Exact v0.23 Implementation Contract + +``` +Field location: top-level in graphUpdateSchema (lib/graph/schema.js line ~184) +Type: z.boolean().nullable().optional() +Nullable: YES during transition; becomes mandatory once prompt ships +Meaning of true: The model declares that the user's supported meaning requires meaningful graph mutation +Meaning of false: The user's supported meaning is already fully represented in existing graph state, so no graph mutation is needed +true + mutation: PASS — declaration fulfilled +true + no mutation: REJECT — contract violation; "structuralActionRequired is true but proposal contains no graph mutation" +false + no mutation: PASS — intentional no-op; declaration matches zero mutations +false + mutation: REJECT — contract violation; declaration contradicts output shape +missing + populated meaning: REJECT — field required when userSupportedMeaning is populated +missing + no meaning: RETAIN existing "no meaningful change" behavior (unchanged) +legacy no-op guard: REPLACED BY structuralActionRequired CONTRACT for new-contract path; generic non-meaning rejection retained +``` + +--- + +## Required Regression Test Matrix + +1. **true + meaningful mutation** → PASS. Validator confirms declaration matches mutations present. +2. **true + zero mutation** → REJECT. Error: "structuralActionRequired is true but proposal contains no graph mutation." +3. **false + zero mutation** → PASS. Valid intentional no-op with populated userSupportedMeaning. +4. **false + meaningful mutation** → REJECT. Error: "structuralActionRequired is false but proposal contains meaningful mutations." +5. **null + populated userSupportedMeaning** → REJECT. Error: "structuralActionRequired must be present when userSupportedMeaning is populated." +6. **null + no userSupportedMeaning** → PASS/REJECT based on hasMeaningfulChange (existing behavior preserved). +7. **populated meaning + false does not imply semantic truth was proven** → documented in test as explicit assertion: validation passes but this proves only contract consistency, not semantic correctness. +8. **existing hasMeaningfulChange logic unchanged** → all existing mutation-detection tests pass identically (verified against current 64-test suite). +9. **supportCategory remains independent of structuralActionRequired** → no cross-dependency; supportCategory = null with any structuralActionRequired value is valid. +10. **no keyword/synonym/raw-English logic added** → validation compares boolean against hasMeaningfulChange boolean result only. Zero semantic parsing in the contract check. + +--- + +## Recommendation + +**A — Strict exact structural contract** + +**Why:** `structuralActionRequired` uses "required" which denotes necessity. A boolean named "required" should mean what it says: an action is required (true) or not required (false). The EXACT STRUCTURAL CLAIM provides crisp, deterministic semantics in all four cases, prevents the most damaging error class (false + mutation), and enables intentional no-ops as a valid contract-consistent path rather than requiring semantic proof. + +This does NOT require: +- New semantic taxonomy: NO +- Keyword/synonym logic: NO +- Provider-specific behavior + +This preserves: +- Provider-agnostic design: YES +- Existing hasMeaningfulChange semantics: unchanged (only new boolean check added) +- supportCategory independence: maintained + +--- + +## Convergence + +**READY FOR BOUNDED IMPLEMENTATION: YES** + +All previously ambiguous decisions from 57J.66 are now settled: +- Field location: top-level graphUpdateSchema +- Semantics: EXACT STRUCTURAL CLAIM (strict, not advisory) +- Null transition: Policy A (reject when meaning populated, retain otherwise) +- Contradiction matrix: all four cases fully specified +- Legacy guard: replaced by contract for new path + +--- + +## Required Implementation Boundary (if READY) + +### Files changed: +1. `lib/graph/schema.js` — add `structuralActionRequired` to `graphUpdateSchema` (line ~184), as `z.boolean().nullable().optional()` +2. `lib/graph/utils.js` — in `validateGraphUpdate()`, add exact structural contract check alongside existing hasMeaningfulChange logic; replace semantic-only-no-op rejection with contract-based logic +3. `lib/graph/prompt-builder.js` — add field to "Required JSON Field Names" list, to "Required Shapes" section, and add two prompt sentences under "Proposal Rules" +4. `tests/graph/utils.test.js` — 6 new tests for the contract matrix + regression assertions + +### New tests: +1. true + meaningful mutation → pass; +2. true + zero mutation → reject with specific error message; +3. false + zero mutation (with populated userSupportedMeaning) → pass (intentional no-op); +4. false + meaningful mutation → reject with specific error message; +5. null + populated userSupportedMeaning → reject (field required); +6. null + no userSupportedMeaning → retain existing "no meaningful change" rejection; +7. documented assertion: PASS on populated meaning + false does not prove semantic correctness — only contract consistency; +8. existing hasMeaningfulChange semantics remain unchanged for non-contract paths; +9. supportCategory remains independent of structuralActionRequired (any combination valid); +10. no keyword/synonym/raw-English logic added anywhere in contract check. + +### Scope exclusions (intentionally out of scope): +- retry/regeneration +- mutation enums or categories +- scoring +- evidence linkage +- provider-specific behaviour +- semantic similarity detection +- keyword classifiers +- changing `hasMeaningfulChange` computation itself +- changing `supportCategory` behavior + +### What this intentionally leaves unresolved: +- Whether the strict false/mutation path should eventually log a diagnostic before rejecting +- Whether `structuralActionRequired` should eventually carry additional fields (e.g., `structuralReason`) +- Whether the prompt rule needs refinement based on live model behavior under the contract +- Migration of existing prompts that reference the old schema field list + +--- + +**Classification:** READ-ONLY ARCHITECTURE DECISION. No production code changed. No Ollama calls. No tests modified. All decisions settled for bounded implementation.