experiment: finalize semantic action contract semantics
This commit is contained in:
+22
-1
@@ -1228,4 +1228,25 @@ The contract semantics are **ADVISORY** — the boolean is a minimum intent decl
|
||||
|
||||
**READY FOR BOUNVED IMPLEMENTATION: YES.** All decisions resolved: field location, strict/advisory semantics, null transition, contradiction matrix. No remaining ambiguity for bounded implementation.
|
||||
|
||||
Full record in `docs/experiment-57j66.md`.
|
||||
Full record in `docs/experiment-57j66.md`.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.67 — semanticActionRequired Contract Semantics Finalized
|
||||
|
||||
**Classification: A — SEMANTICS SETTLED.** Resolved the final ambiguity from 57J.66: is the boolean an *exact structural claim* (strict contract) or a *minimum-action claim* (advisory)? **Decision: EXACT STRUCTURAL CLAIM.** The field name "structuralActionRequired" semantically implies necessity, not suggestion. Definition A provides cleaner semantics, fully deterministic validation in all four cases, and prevents the most damaging error class (model declares no action but produces structure). Advisory (57J.66's recommendation) is rejected: `false + mutation` violates contract consistency — if the model declares "no structural action required" but produces meaningful mutations, it has either misunderstood the answer or over-produced unnecessary structure. This is not harmless.
|
||||
|
||||
- **Boolean definition:** EXACT STRUCTURAL CLAIM (Definition A). true = meaningful mutation present; false = no meaningful mutation needed.
|
||||
- **true + mutation:** PASS. true + no mutation: REJECT. false + no mutation: PASS. false + mutation: REJECT (under exact claim).
|
||||
- **False semantics:** Option A — "The user's supported meaning is already fully represented in graph state, so no graph mutation is needed." This covers semantic agreement with existing state and other legitimate no-op cases.
|
||||
- **Populated meaning + false + empty:** PASS. Does NOT prove semantic correctness (NO). Proves only: model explicitly declared intent + declaration matches zero mutations = contract consistent. Semantic truth remains unproven.
|
||||
- **Populated meaning + false + mutation:** REJECT under exact claim. Declaration says "no structural change needed" but proposal contradicts by producing meaningful changes. Deterministic validation catches this inconsistency.
|
||||
- **Transition rule:** A — missing/null + populated userSupportedMeaning → reject; missing/null + no meaning → retain existing behavior. Mandatory on every proposal (C) deferred to prompt-only enforcement.
|
||||
- **Legacy no-op guard:** REMAINS ONLY FOR LEGACY/MISSING FIELD. Under exact contract, `false + zero mutation` is valid intentional no-op — legacy "semantic-only no-op rejection" would incorrectly block it. Legacy guard stays for when structuralActionRequired is absent.
|
||||
- **Prompt wording (2 sentences):**
|
||||
1. "Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)."
|
||||
2. "Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed."
|
||||
|
||||
**Final v0.23 contract:** top-level field in `graphUpdateSchema`, boolean | nullable, exact structural semantics. Ready for bounded implementation.
|
||||
|
||||
New branch: `feature/semantic-action-contract-v0.23`
|
||||
@@ -0,0 +1,290 @@
|
||||
# Experiment 57J.67 — `structuralActionRequired` Contract Semantics Finalized
|
||||
|
||||
**Branch:** `feature/selected-question-contract-v0.22`
|
||||
**Starting HEAD:** `9425e7b` (experiment: define semantic action contract)
|
||||
|
||||
## Objective
|
||||
|
||||
Settle the final ambiguity from Experiment 57J.66:
|
||||
|
||||
> **Is `structuralActionRequired` a strict consistency contract or merely advisory intent?**
|
||||
|
||||
This task settles that question and produces a complete, unambiguous v0.23 implementation contract.
|
||||
|
||||
**Classification: READ-ONLY ARCHITECTURE DECISION. No production code changed.**
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Boolean Definition Chosen
|
||||
|
||||
### Comparison
|
||||
|
||||
**Definition A — EXACT STRUCTURAL CLAIM** (chosen):
|
||||
```
|
||||
true → proposal contains meaningful mutation (hasMeaningfulChange === true)
|
||||
false → no meaningful mutation is needed (hasMeaningfulChange === false)
|
||||
Declaration matches proposal shape exactly.
|
||||
```
|
||||
|
||||
**Definition B — MINIMUM-ACTION CLAIM** (rejected):
|
||||
```
|
||||
true → at least some structural mutation occurs
|
||||
false → no minimum required, but extra mutation is allowed
|
||||
Declaration is a floor, not a boundary.
|
||||
```
|
||||
|
||||
### Decision: EXACT STRUCTURAL CLAIM
|
||||
|
||||
**Why:**
|
||||
|
||||
1. **Field name semantics.** `structuralActionRequired` uses the word "required" — which denotes necessity, not suggestion. Under Definition B, `false` means "no *minimum* action required" which is awkward and contradicts the natural reading of "action [is] required = false."
|
||||
|
||||
2. **Full determinism.** Definition A produces exactly four deterministic outcomes (one per contradiction pair) with no ambiguity about what passes or fails. Definition B requires distinguishing "more than necessary but harmless" from "contract fulfilled," which introduces softness into a field designed for hard validation.
|
||||
|
||||
3. **Prevents the most damaging error class.** `false + mutation` under exact claim rejects a model that declared "no structural change needed" while producing meaningful mutations — either it misunderstood the answer or over-produced structure. Under advisory semantics, this goes undetected and becomes silent degradation.
|
||||
|
||||
4. **57J.66's advisory recommendation was premature.** It was made without resolving whether false + mutation genuinely harms the contract. Analysis shows it does: a declaration that "no action is required" followed by actual structural production creates an inconsistency that semantic interpretation cannot resolve deterministically.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Contradiction Matrix (Exact Structural Claim)
|
||||
|
||||
| `structuralActionRequired` | hasMeaningfulChange | Outcome | Rationale |
|
||||
|---|---|---|---|
|
||||
| true | true | **PASS** | Declaration fulfilled. Action declared and produced. Contract satisfied. |
|
||||
| true | false | **REJECT** | Model claims action is required but produces zero mutations. Either the model misunderstood the answer's implications, or failed to execute on its own declaration. Deterministic error: contract violation. |
|
||||
| false | false | **PASS** | Intentional no-op. Model explicitly declared that no structural action is needed, and zero mutations confirm the declaration. Deterministic code trusts this structured declaration. |
|
||||
| false | true | **REJECT** | Declaration says "no structural change needed" but proposal produces meaningful changes. Under exact claim, this is inconsistent — the model either misunderstood the user's meaning (claimed no action when one was needed) or over-produced structure beyond what the answer warrants. This is not harmless extra progress; it is a broken contract between declaration and output shape. |
|
||||
|
||||
**Why false + mutation rejects without being advisory:** If the model truly believed the user's supported meaning didn't require any structural change, then producing meaningful mutations means either: (a) the model changed its mind mid-production without updating `structuralActionRequired`, or (b) the model misunderstood what "no action required" means. In either case, the inconsistency is actionable by deterministic validation — the field exists to surface exactly this class of error.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — What `false` Actually Means
|
||||
|
||||
### Chosen: A
|
||||
|
||||
```
|
||||
The user's supported meaning is already fully represented in graph state,
|
||||
so no graph mutation is needed.
|
||||
```
|
||||
|
||||
**Why A over B:** Option B ("The proposal intentionally performs no graph progress for this answer") is too narrow — it only covers cases where the model *chooses* to do nothing. It excludes the primary case: semantic agreement with existing graph state. Option A covers both the intentional no-op (the model evaluates and finds nothing to change) and semantic agreement (an equivalent unresolved uncertainty already exists).
|
||||
|
||||
**Why A over C:** Option C ("Either A or another legitimate no-op case") is intentionally vague and would require semantic parsing at validation time to determine which sub-case applies — defeating the purpose of a deterministic boolean field.
|
||||
|
||||
Option A is precise: when `structuralActionRequired = false`, the model asserts that **the user's supported meaning does not necessitate any graph change**. This assertion can be either true or false (semantic correctness is unprovable), but the declaration itself is deterministically checkable against proposal shape.
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Populated Meaning + False + Empty
|
||||
|
||||
```
|
||||
userSupportedMeaning: populated (non-empty string)
|
||||
structuralActionRequired: false
|
||||
hasMeaningfulChange: false
|
||||
```
|
||||
|
||||
### Deterministic Validation: PASS
|
||||
|
||||
**Rationale:** The model explicitly declared that no structural action is needed (`false`) and the proposal confirms zero mutations. Deterministic code verifies contract consistency — declaration matches reality. No semantic parsing of the userSupportedMeaning content is required or performed.
|
||||
|
||||
### Does this prove the model's semantic judgment was correct?
|
||||
|
||||
**NO**
|
||||
|
||||
**What it proves:**
|
||||
1. The model made an explicit structural intent declaration (no silence).
|
||||
2. The proposal shape matches that declaration (consistency verified).
|
||||
3. The model intentionally chose a no-op path with populated meaning extraction.
|
||||
|
||||
**What it does NOT prove:**
|
||||
- Whether the user's supported meaning genuinely didn't warrant graph mutation.
|
||||
- Whether useful graph structure was omitted.
|
||||
- Whether the answer warranted more than zero mutations.
|
||||
|
||||
The boolean field's purpose is precisely to avoid requiring semantic proof — it delegates semantic judgment to the model and only checks structural consistency.
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Populated Meaning + False + Mutation
|
||||
|
||||
```
|
||||
userSupportedMeaning: populated (non-empty string)
|
||||
structuralActionRequired: false
|
||||
hasMeaningfulChange: true
|
||||
```
|
||||
|
||||
### Deterministic Validation: REJECT
|
||||
|
||||
**Why (contract terms):** Under exact structural claim, `false` means "no meaningful mutation is needed." The presence of meaningful mutations contradicts this declaration. The model either:
|
||||
- Claimed no action was needed but then produced structure anyway (mid-production state change), or
|
||||
- Misunderstood the user's meaning and over-produced beyond what the answer warranted.
|
||||
|
||||
This is not a case of "more progress is harmless." A field named `structuralActionRequired` must be truthful about its own claim: if it says `false`, the proposal should contain zero mutations. Any deviation breaks the contract deterministically — no semantic interpretation needed.
|
||||
|
||||
**Note:** Under 57J.66's advisory recommendation, this would have been accepted with a diagnostic note. This experiment rejects that approach because:
|
||||
- It defeats the purpose of having a boolean field with crisp semantics.
|
||||
- A model can always produce "more" structure regardless of what it declares, making `false` meaningless as a signal.
|
||||
- The inconsistency is actionable by validation and should be surfaced to the developer/model for correction.
|
||||
|
||||
---
|
||||
|
||||
## Part 6 — Missing/Null Transition Rule
|
||||
|
||||
### Chosen: A
|
||||
|
||||
```
|
||||
missing/null + populated userSupportedMeaning → reject
|
||||
missing/null + no userSupportedMeaning → retain existing behaviour
|
||||
```
|
||||
|
||||
**Why A over B:** Policy B (always retain existing behavior for missing/null) creates a silent degradation window during transition. Any proposal with populated `userSupportedMeaning` and missing `structuralActionRequired` would bypass the new contract entirely, allowing noncompliant outputs to pass validation until the prompt change ships.
|
||||
|
||||
**Why A over C:** While the field should ultimately be mandatory on every proposal (C), enforcing it at the validator level during transition is premature without the prompt requiring it first. Policy A provides a minimal safety net: the contract activates whenever there is meaningful content that could justify structural action. The transition to full mandatory enforcement (C) happens when the prompt change ships in v0.23.
|
||||
|
||||
**Specific transitions:**
|
||||
- `structuralActionRequired` absent + `userSupportedMeaning` populated → **REJECT** ("structuralActionRequired must be present when userSupportedMeaning is populated")
|
||||
- `structuralActionRequired` null + `userSupportedMeaning` populated → **REJECT** (same as absent)
|
||||
- `structuralActionRequired` absent/null + `userSupportedMeaning` not populated → existing behavior ("Update contains no meaningful change" if zero mutations; pass if mutations present)
|
||||
|
||||
---
|
||||
|
||||
## Part 7 — Legacy No-Op Guard Status
|
||||
|
||||
### Decision: REPLACED BY structuralActionRequired CONTRACT
|
||||
|
||||
**Rationale:** The existing legacy guard rejects any proposal where `userSupportedMeaning` is populated but `hasMeaningfulChange` is false. Under the new exact contract:
|
||||
- When `structuralActionRequired = false` + zero mutations → this should PASS as a valid intentional no-op (the model declared no action needed, and it produced none).
|
||||
- The legacy guard would incorrectly reject this valid case.
|
||||
|
||||
**Implementation approach:** The legacy guard's semantic-only-no-op rejection (`"answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"`) is replaced by the `structuralActionRequired` contract check:
|
||||
- If `structuralActionRequired === false` → skip legacy guard (intentional no-op is valid).
|
||||
- If `structuralActionRequired === true` → it would already be rejected by the `true + no mutation` rule.
|
||||
- If `structuralActionRequired` is missing/null + populated meaning → reject for field absence, not for structural mismatch.
|
||||
|
||||
**Result:** The legacy guard's specific semantic-no-op rejection is removed from the new-contract path and effectively replaced by the `structuralActionRequired` contract. Its generic "no meaningful change" rejection remains for cases where `userSupportedMeaning` is null/non-populated.
|
||||
|
||||
---
|
||||
|
||||
## Part 8 — Prompt Wording Boundary
|
||||
|
||||
### Minimum Semantic Instructions (2 sentences):
|
||||
|
||||
1. **"Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)."**
|
||||
|
||||
2. **"Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed."**
|
||||
|
||||
These two sentences are sufficient because:
|
||||
- Sentence 1 gives an *output-based* criterion (truth = proposal has mutations), which the model can verify against its own output without requiring semantic analysis.
|
||||
- Sentence 2 gives a *semantic* criterion for false only (the user's meaning is already in the graph), which is the legitimate case for no-op.
|
||||
- No third action taxonomy is introduced; the boolean maps directly to `hasMeaningfulChange`.
|
||||
- The prompt does not need to explain every edge case — deterministic validation handles those at the contract level.
|
||||
|
||||
---
|
||||
|
||||
## Part 9 — Exact v0.23 Implementation Contract
|
||||
|
||||
```
|
||||
Field location: top-level in graphUpdateSchema (lib/graph/schema.js line ~184)
|
||||
Type: z.boolean().nullable().optional()
|
||||
Nullable: YES during transition; becomes mandatory once prompt ships
|
||||
Meaning of true: The model declares that the user's supported meaning requires meaningful graph mutation
|
||||
Meaning of false: The user's supported meaning is already fully represented in existing graph state, so no graph mutation is needed
|
||||
true + mutation: PASS — declaration fulfilled
|
||||
true + no mutation: REJECT — contract violation; "structuralActionRequired is true but proposal contains no graph mutation"
|
||||
false + no mutation: PASS — intentional no-op; declaration matches zero mutations
|
||||
false + mutation: REJECT — contract violation; declaration contradicts output shape
|
||||
missing + populated meaning: REJECT — field required when userSupportedMeaning is populated
|
||||
missing + no meaning: RETAIN existing "no meaningful change" behavior (unchanged)
|
||||
legacy no-op guard: REPLACED BY structuralActionRequired CONTRACT for new-contract path; generic non-meaning rejection retained
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Required Regression Test Matrix
|
||||
|
||||
1. **true + meaningful mutation** → PASS. Validator confirms declaration matches mutations present.
|
||||
2. **true + zero mutation** → REJECT. Error: "structuralActionRequired is true but proposal contains no graph mutation."
|
||||
3. **false + zero mutation** → PASS. Valid intentional no-op with populated userSupportedMeaning.
|
||||
4. **false + meaningful mutation** → REJECT. Error: "structuralActionRequired is false but proposal contains meaningful mutations."
|
||||
5. **null + populated userSupportedMeaning** → REJECT. Error: "structuralActionRequired must be present when userSupportedMeaning is populated."
|
||||
6. **null + no userSupportedMeaning** → PASS/REJECT based on hasMeaningfulChange (existing behavior preserved).
|
||||
7. **populated meaning + false does not imply semantic truth was proven** → documented in test as explicit assertion: validation passes but this proves only contract consistency, not semantic correctness.
|
||||
8. **existing hasMeaningfulChange logic unchanged** → all existing mutation-detection tests pass identically (verified against current 64-test suite).
|
||||
9. **supportCategory remains independent of structuralActionRequired** → no cross-dependency; supportCategory = null with any structuralActionRequired value is valid.
|
||||
10. **no keyword/synonym/raw-English logic added** → validation compares boolean against hasMeaningfulChange boolean result only. Zero semantic parsing in the contract check.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**A — Strict exact structural contract**
|
||||
|
||||
**Why:** `structuralActionRequired` uses "required" which denotes necessity. A boolean named "required" should mean what it says: an action is required (true) or not required (false). The EXACT STRUCTURAL CLAIM provides crisp, deterministic semantics in all four cases, prevents the most damaging error class (false + mutation), and enables intentional no-ops as a valid contract-consistent path rather than requiring semantic proof.
|
||||
|
||||
This does NOT require:
|
||||
- New semantic taxonomy: NO
|
||||
- Keyword/synonym logic: NO
|
||||
- Provider-specific behavior
|
||||
|
||||
This preserves:
|
||||
- Provider-agnostic design: YES
|
||||
- Existing hasMeaningfulChange semantics: unchanged (only new boolean check added)
|
||||
- supportCategory independence: maintained
|
||||
|
||||
---
|
||||
|
||||
## Convergence
|
||||
|
||||
**READY FOR BOUNDED IMPLEMENTATION: YES**
|
||||
|
||||
All previously ambiguous decisions from 57J.66 are now settled:
|
||||
- Field location: top-level graphUpdateSchema
|
||||
- Semantics: EXACT STRUCTURAL CLAIM (strict, not advisory)
|
||||
- Null transition: Policy A (reject when meaning populated, retain otherwise)
|
||||
- Contradiction matrix: all four cases fully specified
|
||||
- Legacy guard: replaced by contract for new path
|
||||
|
||||
---
|
||||
|
||||
## Required Implementation Boundary (if READY)
|
||||
|
||||
### Files changed:
|
||||
1. `lib/graph/schema.js` — add `structuralActionRequired` to `graphUpdateSchema` (line ~184), as `z.boolean().nullable().optional()`
|
||||
2. `lib/graph/utils.js` — in `validateGraphUpdate()`, add exact structural contract check alongside existing hasMeaningfulChange logic; replace semantic-only-no-op rejection with contract-based logic
|
||||
3. `lib/graph/prompt-builder.js` — add field to "Required JSON Field Names" list, to "Required Shapes" section, and add two prompt sentences under "Proposal Rules"
|
||||
4. `tests/graph/utils.test.js` — 6 new tests for the contract matrix + regression assertions
|
||||
|
||||
### New tests:
|
||||
1. true + meaningful mutation → pass;
|
||||
2. true + zero mutation → reject with specific error message;
|
||||
3. false + zero mutation (with populated userSupportedMeaning) → pass (intentional no-op);
|
||||
4. false + meaningful mutation → reject with specific error message;
|
||||
5. null + populated userSupportedMeaning → reject (field required);
|
||||
6. null + no userSupportedMeaning → retain existing "no meaningful change" rejection;
|
||||
7. documented assertion: PASS on populated meaning + false does not prove semantic correctness — only contract consistency;
|
||||
8. existing hasMeaningfulChange semantics remain unchanged for non-contract paths;
|
||||
9. supportCategory remains independent of structuralActionRequired (any combination valid);
|
||||
10. no keyword/synonym/raw-English logic added anywhere in contract check.
|
||||
|
||||
### Scope exclusions (intentionally out of scope):
|
||||
- retry/regeneration
|
||||
- mutation enums or categories
|
||||
- scoring
|
||||
- evidence linkage
|
||||
- provider-specific behaviour
|
||||
- semantic similarity detection
|
||||
- keyword classifiers
|
||||
- changing `hasMeaningfulChange` computation itself
|
||||
- changing `supportCategory` behavior
|
||||
|
||||
### What this intentionally leaves unresolved:
|
||||
- Whether the strict false/mutation path should eventually log a diagnostic before rejecting
|
||||
- Whether `structuralActionRequired` should eventually carry additional fields (e.g., `structuralReason`)
|
||||
- Whether the prompt rule needs refinement based on live model behavior under the contract
|
||||
- Migration of existing prompts that reference the old schema field list
|
||||
|
||||
---
|
||||
|
||||
**Classification:** READ-ONLY ARCHITECTURE DECISION. No production code changed. No Ollama calls. No tests modified. All decisions settled for bounded implementation.
|
||||
Reference in New Issue
Block a user