263 lines
13 KiB
Markdown
263 lines
13 KiB
Markdown
# Experiment 57J.65 — Smallest Enforceable Semantic-to-Mutation Contract
|
|
|
|
**Branch:** `feature/selected-question-contract-v0.22`
|
|
**Starting HEAD:** `d7cb343` (experiment: diagnose semantic-to-mutation action ownership)
|
|
|
|
## Objective
|
|
|
|
Answer one question:
|
|
|
|
> What is the smallest structured contract that lets the model declare whether graph action is required, and lets deterministic code verify that the actual proposal fulfils that declaration?
|
|
|
|
57J.64 established that further prompt-only wording is not the next boundary. This experiment answers with data-contract analysis only.
|
|
|
|
---
|
|
|
|
## Part 1 — Are Existing Fields Enough?
|
|
|
|
**Classification: C — NEW ACTION DECLARATION REQUIRED**
|
|
|
|
The existing fields provide these capabilities:
|
|
|
|
| Field | What it expresses |
|
|
|-------|-------------------|
|
|
| `userSupportedMeaning` | Semantic content (text) of what the user supports |
|
|
| `supportCategory` | Category label for semantic content |
|
|
| `resolutionGuidance` | Resolution instruction |
|
|
| `updatedNodes` | Nodes modified |
|
|
| `resolvedUnknownNodeIds` | Unknowns resolved |
|
|
| `addedNodes` | New nodes created |
|
|
| `addedEdges` | New edges created |
|
|
| `selectedQuestion` | Follow-up question candidate |
|
|
|
|
**Why they are insufficient:**
|
|
|
|
These fields encode *what changed* but not *what was intended*. When a model intends "I agree with the semantic content, no structural change is needed," it returns empty mutation arrays. There is no explicit field saying "I intentionally declare zero graph action." The validator's current check (line 886 of utils.js) derives intent from:
|
|
|
|
```
|
|
userSupportedMeaning populated + all mutation arrays empty → REJECT
|
|
```
|
|
|
|
This treats the model's silence as an error rather than accepting a valid intentional no-op declaration. It cannot distinguish between "model forgot to mutate" and "model intentionally chose no mutation."
|
|
|
|
---
|
|
|
|
## Part 2 — Minimum Required Distinction
|
|
|
|
**What deterministic validation actually needs:**
|
|
|
|
The validator does not need to know *why* the model made its choice. It only needs to verify that the model's declared intent matches the proposal shape.
|
|
|
|
| Intended Action | How validator checks | Classification |
|
|
|-----------------|----------------------|----------------|
|
|
| Reuse/refine existing | `updatedNodes` references existing node with status/value change | DERIVABLE FROM PROPOSAL SHAPE |
|
|
| Add new unknown | `addedUnknownCount > 0` | DERIVABLE FROM PROPOSAL SHAPE |
|
|
| Resolve existing | `resolvedUnknownNodeIds.length > 0` | DERIVABLE FROM PROPOSAL SHAPE |
|
|
| Other structural mutation | Any non-empty mutation array or addedEdges | DERIVABLE FROM PROPOSAL SHAPE |
|
|
| No structural change | All mutation arrays empty | MUST BE DECLARED (by the model) |
|
|
|
|
**Conclusion: The minimum distinction is `MUTATION REQUIRED` vs `NO MUTATION REQUIRED`.**
|
|
|
|
Deterministic validation does not need to know *which* mutation type was intended because it checks the actual proposal shape for each possible mutation independently. The only gap is: when all arrays are empty, how do we know the model intentionally chose no-op vs failed to produce one?
|
|
|
|
---
|
|
|
|
## Part 3 — Compare Three Designs
|
|
|
|
### Option A — Boolean Contract
|
|
|
|
A single field: `structuralActionRequired: true | false`
|
|
|
|
| Criterion | Answer |
|
|
|-----------|--------|
|
|
| Prevents ambiguous semantic-only no-op | PARTIAL — declares intent, but model can always choose the "safe" value without verifying |
|
|
| Checks actual mutation | YES — validator compares declared value against proposal shape |
|
|
| Requires re-reading English semantics | NO — only compares structured field against structured arrays |
|
|
| New schema concept | BOOLEAN |
|
|
| Validator complexity | LOW — two boolean checks (true→non-empty, false→empty) |
|
|
| Model-compliance risk | MEDIUM — model may default to one value under pressure; binary choice is simplest for the model |
|
|
|
|
### Option B — Small Action Enum
|
|
|
|
A field: `semanticAction: "add_new_unknown" | "reuse_or_refine_existing" | "resolve_existing" | "other_structural_mutation" | "no_change_needed"`
|
|
|
|
| Criterion | Answer |
|
|
|-----------|--------|
|
|
| Prevents ambiguous semantic-only no-op | PARTIAL — more categories than validation needs, but declares explicit intent |
|
|
| Checks actual mutation | YES — validator maps each enum value to specific proposal shape requirements |
|
|
| Requires re-reading English semantics | NO — only compares structured field against structured arrays |
|
|
| New schema concept | SMALL ENUM (5 values) |
|
|
| Validator complexity | MEDIUM — five mapping rules plus cross-validation |
|
|
| Model-compliance risk | MEDIUM-HIGH — more categories increase noncompliance risk; model must pick from five options deterministically |
|
|
|
|
### Option C — Existing Fields Only
|
|
|
|
No new field. Use `userSupportedMeaning` populated + empty mutation arrays to mean "intentional semantic agreement, no graph change."
|
|
|
|
| Criterion | Answer |
|
|
|-----------|--------|
|
|
| Prevents ambiguous semantic-only no-op | PARTIAL — currently rejects this case; treating it as valid would accept noncompliant outputs silently |
|
|
| Checks actual mutation | YES — proposal shape is always checkable |
|
|
| Requires re-reading English semantics | NO — existing behavior already works without semantic parsing |
|
|
| New schema concept | NONE |
|
|
| Validator complexity | LOW — no new logic needed |
|
|
| Model-compliance risk | HIGH — treating empty-mutation-as-intentional would accept every noncompliant zero-mutation output, making the boundary unenforceable |
|
|
|
|
---
|
|
|
|
## Part 4 — The No-Change Case
|
|
|
|
**Can no-change be verified without re-reading English?**
|
|
|
|
**YES — but only with a new structured declaration**
|
|
|
|
With existing fields:
|
|
- `userSupportedMeaning` populated + all mutation arrays empty → current code REJECTS
|
|
- We cannot distinguish "model intended no-op" from "model forgot to mutate"
|
|
- This is NOT verifiable as intentional without knowing what the model *meant*
|
|
|
|
With a new declaration field:
|
|
- Model sets `structuralActionRequired: false` + all mutation arrays empty → validation PASSES (model explicitly declared no action)
|
|
- Model sets `structuralActionRequired: true` + all mutation arrays empty → validation REJECTS (contradiction between intent and proposal)
|
|
- The declaration itself is the verification mechanism
|
|
|
|
---
|
|
|
|
## Part 5 — Relationship to supportCategory
|
|
|
|
**Relationship: INDEPENDENT OF supportCategory**
|
|
|
|
Reasoning:
|
|
|
|
- `supportCategory = "uncertain"` does NOT necessarily mean `add new unknown`
|
|
- An equivalent uncertainty may already exist and should be reused (v0.21 identity rule)
|
|
- `supportCategory` classifies the *semantic content* of the answer
|
|
- The structural action declaration classifies the *proposed graph change*
|
|
- These are orthogonal: the same supportCategory can map to different structural actions depending on current graph state
|
|
|
|
The v0.21 identity rule must be preserved: when an equivalent unresolved uncertainty already exists, reuse/refine that existing node — do not add a duplicate.
|
|
|
|
---
|
|
|
|
## Part 6 — Deterministic Invariants
|
|
|
|
For Option A (boolean contract), the invariants are:
|
|
|
|
1. **`structuralActionRequired = true` + all mutation arrays empty → REJECT**
|
|
The model declared intent for structural action but produced none.
|
|
|
|
2. **`structuralActionRequired = false` + meaningful mutation present → ACCEPT (diagnostic note)**
|
|
Model declared no change but produced one. This is not a contradiction — it may be the model doing extra work beyond what was needed. Log a warning.
|
|
|
|
3. **`structuralActionRequired` missing + `userSupportedMeaning` populated → REJECT**
|
|
Cannot verify intent when required field is absent.
|
|
|
|
4. **No invariant needed for `structuralActionRequired = false` + empty mutations**
|
|
This is the valid "semantic agreement, no structural change" case. The model explicitly declared its intention; validation passes because it can do so deterministically without semantic parsing.
|
|
|
|
---
|
|
|
|
## Part 7 — 57J.63 Walkthrough
|
|
|
|
### Case A: Successful proposal with dedicated unknown
|
|
|
|
```text
|
|
userSupportedMeaning: "uncertainty about projected office savings realism"
|
|
proposal: adds dedicated savings-realism unknown
|
|
Declaration: structuralActionRequired = true
|
|
```
|
|
|
|
**Why validation passes:**
|
|
- Model declares `true` → expects meaningful mutation
|
|
- `addedNodes` contains a new unknown node (non-empty)
|
|
- Validator compares: declared `true` + actual mutation present → PASS
|
|
|
|
### Case B: Semantic-only no-op with same meaning
|
|
|
|
```text
|
|
userSupportedMeaning: "uncertainty about projected office savings realism"
|
|
proposal: no meaningful graph mutation
|
|
Declaration options: structuralActionRequired = false (intentional) or structuralActionRequired = true (noncompliant)
|
|
```
|
|
|
|
**What the model can declare:**
|
|
- If equivalent uncertainty already exists in the graph → `structuralActionRequired = false` is valid. The model has legitimately determined no new structure is needed.
|
|
- If no equivalent exists and the answer introduces genuinely new material → `structuralActionRequired = true` is required by rule #6.
|
|
|
|
**Exactly what deterministic validation does:**
|
|
1. Check `structuralActionRequired` is populated (not null) because `userSupportedMeaning` is populated
|
|
2. Compare declared value against proposal shape:
|
|
- `true` + empty mutations → REJECT (contradiction)
|
|
- `false` + empty mutations → PASS (explicit no-op declaration validated against zero mutation)
|
|
- `false` + non-empty mutations → ACCEPT with diagnostic note (model did more than declared)
|
|
|
|
**Classification of preferred design:**
|
|
|
|
**A — ACTUAL CONTRACT ENFORCEMENT**
|
|
|
|
This is a contract at the structural level: the model declares its intent in a structured field, and code verifies that the proposal shape matches. If the model declares `false` (no change needed), validation passes because it checks the actual empty mutation state — not semantic similarity. The boundary between "intentional no-op" and "noncompliant no-op" is enforced by requiring the explicit declaration.
|
|
|
|
This solves the boundary because:
|
|
- Noncompliant zero-mutation outputs cannot hide behind empty arrays (they must also declare `true`, which fails validation)
|
|
- Intentional no-ops are valid when equivalent structure already exists (model declares `false`, validation confirms empty mutation)
|
|
|
|
---
|
|
|
|
## Part 8 — Recommendation
|
|
|
|
**Recommended option: B — boolean structural-action contract**
|
|
|
|
### Exact new field
|
|
|
|
```
|
|
structuralActionRequired: boolean | null
|
|
nullable during transition: YES (but rejected if userSupportedMeaning is populated and field is null)
|
|
```
|
|
|
|
### Location in schema
|
|
|
|
Add to `answerMeaningSchema` in `lib/graph/schema.js`:
|
|
|
|
```javascript
|
|
export const answerMeaningSchema = z.object({
|
|
userSupportedMeaning: z.string().min(1),
|
|
possibleInference: z.string().nullable().optional(),
|
|
supportCategory: z.enum(...).nullable().optional(),
|
|
resolutionGuidance: z.enum(...).nullable().optional(),
|
|
structuralActionRequired: z.boolean().nullable().optional(), // NEW
|
|
});
|
|
```
|
|
|
|
### Exact validator invariants (in `lib/graph/utils.js`, in `validateGraphUpdate`)
|
|
|
|
After the existing `hasMeaningfulChange` check (around line 876):
|
|
|
|
```javascript
|
|
if (!hasMeaningfulChange) {
|
|
if (update.answerMeaning?.structuralActionRequired === false) {
|
|
// Intentional no-op — model declared no change needed, and proposal confirms it
|
|
// PASS — this is the "semantic agreement, no structural change" case
|
|
} else if (!update.answerMeaning?.structuralActionRequired) {
|
|
errors.push(
|
|
"answerMeaning.structuralActionRequired must be populated when userSupportedMeaning is present."
|
|
);
|
|
} else if (update.answerMeaning?.structuralActionRequired === true) {
|
|
errors.push(
|
|
"answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
|
);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Transition policy: B — missing new field + populated userSupportedMeaning is rejected
|
|
|
|
Reason: The current architecture has `userSupportedMeaning` as a commitment signal. If we accept zero-mutation proposals without the new field, every noncompliant output becomes valid again. During transition, reject until the model produces the new field. After the field is present, allow it as the enforcement mechanism.
|
|
|
|
---
|
|
|
|
## Convergence
|
|
|
|
**This is ready for bounded implementation.** The design is minimal: one boolean field and one invariant check. It does not invent new semantic taxonomies. It does not require keyword/synonym logic. It is provider-agnostic because it validates structured output fields, not model behavior.
|
|
|
|
The boundary it solves: the gap between "model understands the meaning" and "model declares its structural intent deterministically." With this contract, the validator checks a declared boolean against actual proposal shape — no semantic parsing needed.
|