diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 8f618f5..86b748a 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1679,3 +1679,7 @@ This satisfies 57J.77's boundary A recommendation: a committed update-only path ### Experiment 57J.85 — Null Semantic/Action Architecture Diagnosis (Read-Only) **Objective:** Diagnose why `answerMeaning=null + structuralActionRequired=null + meaningful mutation` is accepted through the current pipeline, and whether this compatibility path should remain open. Read-only diagnosis across prompt contract, schema, validator, and apply-proposal validation layers. **Classification: B — TRANSITION COMPATIBILITY.** The null/null/mutation path persists because: (1) Zod schema allows nullable fields for backward compatibility; (2) validator rules are scoped to only reject when meaning IS populated (rule 1), leaving null-meaning mutations unguarded; (3) the model produces useful data through this path (57J.84: £2m/year on reported_claim node); (4) tightening without a deterministic recovery/retry path would discard that information. Architectural choice A — keep the null transition path for now, pending deterministic recovery capability before tightening becomes safe. Smallest next boundary: implement a deterministic recovery mechanism for proposals with meaningful mutations but unpopulated semantic/action fields. No production code changed; no Ollama calls; documentation-only diagnosis. + +### Experiment 57J.86 — Smallest Recovery Contract for Null Semantic/Action with Good Mutation (Read-Only Design) + +**Objective:** What is the smallest recovery contract that lets the engine preserve a good mutation while recovering missing semantic/action declarations, without regenerating or discarding the mutation? Read-only design evaluation of four options (deterministic action fill, declaration-only repair call, full regeneration, keep transition path). **Classification: B — DECLARATION-ONLY REPAIR CALL.** structuralActionRequired is PARTIALLY recoverable from structure via hasMeaningfulChange=true (but this changes field semantics from model declaration to engine inference). answerMeaning fields are NOT recoverable from mutation structure alone. Existing code has zero repair capability — validator only validates, orchestrator returns errors on rejection with no retry/repair path. Option B chosen: one bounded second-stage repair call that preserves mutation arrays exactly and recovers all missing declarations (answerMeaning + structuralActionRequired) through model declaration. Repair receives raw answer + original proposal as context; forbidden from changing any mutation arrays. Repair classified as SECOND-STAGE REPAIR, not RETRY or NORMAL SECOND CALL — existing call accounting cannot cleanly distinguish repair calls without tooling change. Non-negotiable invariants all met: original mutation preserved, no keyword logic, no regeneration, exactly 1 bounded additional call, provider-agnostic, 57J.84 information survives full recovery. No production code changed; no Ollama calls; documentation-only design. diff --git a/docs/experiment-57j86.md b/docs/experiment-57j86.md new file mode 100644 index 0000000..c3c6d10 --- /dev/null +++ b/docs/experiment-57j86.md @@ -0,0 +1,284 @@ +# Experiment 57J.86 — Smallest Recovery Contract for Null Semantic/Action with Good Mutation (Read-Only Design) + +**Branch:** `feature/semantic-action-contract-v0.23` +**Starting HEAD:** `a40a3e3` (experiment: diagnose null semantic mutation path) + +## Objective + +Answer exactly: + +> What is the smallest recovery contract that lets the engine preserve a good mutation while recovering missing semantic/action declarations, without regenerating or discarding the mutation? + +Read-only architecture design. No code changes. No Ollama calls. No live API calls. + +## Context Files Read + +1. `docs/current-handoff.md` (handoff state through 57J.85) +2. `docs/experiment-57j85.md` (transition compatibility diagnosis) +3. `lib/graph/schema.js` (schema truth for answerMeaning, structuralActionRequired, graphUpdateSchema) +4. `lib/graph/utils.js` (validator logic at line 868+) +5. `lib/graph/apply-proposal.js` (validation/parsing section: lines 2937–3146; applyValidatedProposal entry at 3182) +6. `lib/graph/orchestrator.js` (proposal rejection path and diagnostics snapshot construction) +7. `app/api/cases/update/route.js` (production API boundary — no retry/repair logic) + +--- + +## 1 — Separate the Two Missing Declarations + +### A. structuralActionRequired + +**Can deterministic code recover it from proposal structure alone?** + +PARTIAL — YES for the true direction only. + +**Test: `hasMeaningfulChange=true` → `structuralActionRequired=true`** + +From `lib/graph/utils.js` lines 878–883: + +```js +const hasMeaningfulChange = + update.addedNodes.length > 0 || + statusChanged || + valueChanged || + update.addedEdges.length > 0 || + update.removedEdgeIds.length > 0; +``` + +If `hasMeaningfulChange=true`, then at least one of these conditions holds: +- addedNodes.length > 0 (new nodes were added) +- statusChanged (at least one node's status was changed) +- valueChanged (at least one node's value was changed) +- addedEdges.length > 0 (new edges were added) +- removedEdgeIds.length > 0 (edges were removed) + +Each of these is by definition a structural action. The model declared that it intended to act (via the mutation itself). Setting `structuralActionRequired=true` when hasMeaningfulChange=true is a deterministic mapping from "mutation present" → "action was required." + +No semantic inference is needed. This is purely structural: if nodes/edges were added or changed, structural action occurred. + +**Would doing so preserve the original meaning of structuralActionRequired as a model declaration, or would it change the field into an engine-derived fact?** + +CHANGES FIELD SEMANTICS. + +`structuralActionRequired` was designed as a *model declaration* — the model telling the engine "I know I must act structurally." Deriving it from mutation presence converts it to an *engine-inferred fact*. The semantic shift is: + +- Before (declaration): "The model consciously chose to declare action is required" +- After (inference): "There was structural change, therefore action must have been needed" + +The practical effect for this experiment's scope is identical (action flows forward either way). But the contract semantics shift from declaration → inference. The field no longer reflects model intent; it reflects engine observation. + +This matters for future contract work because: +- A model declaring `structuralActionRequired=false` with mutation would still be a contradiction (rule 3 checks structuralActionRequired===false) +- An engine-derived `structuralActionRequired=true` from mutation cannot be "wrong" — it is tautologically true by definition of the mutation + +### B. answerMeaning + +**Can existing structured mutation fields recover `userSupportedMeaning`, `supportCategory`, `resolutionGuidance`?** + +NOT RECOVERABLE. + +Reasoning: +- `userSupportedMeaning` is the model's semantic interpretation of the raw user answer — a natural language summary of what the user established. No graph field captures this. +- `supportCategory` classifies the reasoning pattern (relative_priority_only, conditional_tradeoff, uncertain, explicit_hard_constraint, other). This requires understanding the raw answer text, not just the structural result. +- `resolutionGuidance` (must_remain_unresolved, may_resolve, must_resolve) is a judgment about what downstream processing should do — a control signal, not derivable from mutation shape. + +The mutation arrays (addedNodes, updatedNodes, addedEdges) capture WHAT was done to the graph but not WHY or WHAT THE USER ESTABLISHED. The same mutation shape (new reported_claim node) could result from radically different answer meanings (explicit fact vs. estimate vs. uncertainty). There is no deterministic mapping from mutation structure back to semantic intent. + +--- + +## 2 — Existing Recovery Capabilities + +**Can validator mutate/repair proposal: NO** + +`validateGraphUpdate` (utils.js line 868) returns `{ valid, errors }` only. It has no side effects on the input proposal and no repair logic. + +**Can validator preserve rejected proposal and continue: PARTIAL** + +The orchestrator captures a `rejectedProposalSnapshot` (orchestrator.js lines 693–725) for diagnostics when rejection occurs at `proposal_compatibility`. This is write-only diagnostic evidence — it does not feed back into any repair mechanism. + +**Can orchestrator issue a bounded repair call: NO** + +The orchestrator flow (orchestrator.js lines 620–800) is strictly linear: +1. Build prompt → model call +2. Parse proposal (Zod + normalisation) +3. Apply validated proposal → direct graph mutation +4. Return success or error + +There is no retry, repair, or secondary call path. On rejection at `proposal_compatibility`, the orchestrator returns an error with diagnostic snapshot and terminates. + +**Can existing code call the model again with the original proposal attached: NO** + +No mechanism exists to re-invoke the model with any proposal content. The raw response is parsed once and never retained after parsing. No prompt-building path accepts a previous proposal as context. + +**Can a repaired proposal reuse the exact original mutation arrays: REQUIRES NEW PATH** + +Currently, rejected proposals are discarded. Only a diagnostic snapshot (subset of fields) survives. To preserve and reuse the exact mutation arrays through repair would require new plumbing: retention of parsed proposal past rejection, plus a repair call path that accepts mutation-arrays-as-immutable-context. + +--- + +## 3 — Compare Four Recovery Designs + +### Option A — deterministic action-field fill only + +When `hasMeaningfulChange=true` and `structuralActionRequired=null`, engine sets `structuralActionRequired=true`. Leaves answerMeaning unchanged (null). + +| Criterion | Answer | +|---|---| +| preserves useful original mutation | YES | +| requires new LLM call | NO | +| can change graph mutation | NO — only fills one boolean field on the proposal; mutation arrays untouched | +| requires English keyword inference | NO | +| retains semantic accountability | PARTIAL — recovers structural accountability (action = required, inferred from mutation); leaves answerMeaning unaccountable (null) | +| new failure surface | LOW — deterministic fill cannot produce incorrect values. If hasMeaningfulChange=true, structuralActionRequired MUST be true by definition. No hallucination risk. | + +### Option B — one bounded declaration-only repair call + +Preserve original mutation arrays exactly. Ask model to populate only: +- answerMeaning (userSupportedMeaning, supportCategory, resolutionGuidance) +- structuralActionRequired + +Forbidden from changing addedNodes, updatedNodes, resolvedUnknownNodeIds, addedEdges, removedEdgeIds. + +| Criterion | Answer | +|---|---| +| preserves useful original mutation | YES — mutation arrays are passed as immutable context to the repair call | +| requires new LLM call | YES — one additional bounded call | +| can change graph mutation | NO — forbidden by contract boundary of the repair call | +| requires English keyword inference | NO — repair receives raw answer text + original proposal; must produce structured semantics, not derive from keywords | +| retains semantic accountability | FULL — all three missing fields (answerMeaning + structuralActionRequired) are recovered through model declaration, not engine inference | +| new failure surface | MEDIUM — second LLM call introduces latency/cost variance; repair prompt must be carefully constrained to prevent mutation drift | + +### Option C — full proposal regeneration + +Reject original proposal. Ask model to regenerate everything from raw answer + graph state. + +| Criterion | Answer | +|---|---| +| preserves useful original mutation | NO — entirely discarded; new mutation may differ materially | +| requires new LLM call | YES | +| can change graph mutation | YES — full regeneration allows different nodes, edges, values | +| requires English keyword inference | NO — but introduces cold-start variance across two generations from same input | +| retains semantic accountability | FULL — regenerated proposal is fully accountable (model produces fresh declarations for everything) | +| new failure surface | HIGH — double the cost; double the variance; original good data is lost | + +### Option D — keep transition compatibility unchanged + +No repair architecture. Accept null/null + mutation as-is. + +| Criterion | Answer | +|---|---| +| preserves useful original mutation | YES — current path accepts it | +| requires new LLM call | NO | +| can change graph mutation | NO | +| requires English keyword inference | NO | +| retains semantic accountability | NONE — no semantic declarations, no action declaration. The graph records what happened but not why or what the user meant. | +| new failure surface | LOW — no new code; existing path already exercised | + +--- + +## 4 — 57J.84 Walkthrough + +Apply each option to the exact 57J.84 shape: + +``` +answerMeaning = null +structuralActionRequired = null +addedNodes = [n_lease_savings_claim (reported_claim, £2M/year)] +addedEdges = [supports edge → n_savings_realism] +existing uncertainty preserved (n_savings_realism) +``` + +**Option A:** PRESERVED + +Engine sees hasMeaningfulChange=true (new node + new edge). Sets structuralActionRequired=true deterministically. Mutation arrays pass through unchanged. answerMeaning remains null but mutation is preserved. + +**Option B:** PRESERVED + +Repair call receives original mutation arrays as immutable context. Produces answerMeaning with userSupportedMeaning ("User claims £2M/year savings from lease elimination, remaining uncertain about realism"), supportCategory="other", resolutionGuidance="may_resolve". structuralActionRequired=true. Mutation preserved exactly. + +**Option C:** REGENERATED + +Original mutation discarded. New proposal generated — might produce different node IDs, slightly different label/description for the claim, potentially different edge relationships. £2m information survives only if model regenerates it faithfully. + +**Option D:** ACCEPTED UNCHANGED + +Proposal accepted as-is through transition compatibility path. Mutation applied. answerMeaning=null and structuralActionRequired=null persist on the graph with no recovery. + +--- + +## 5 — Repair-Call Ownership + +If option B is chosen, the narrowest possible contract: + +**Should repair be allowed to reconsider semantic meaning? YES** + +The entire purpose of the repair call is to recover meaning declarations. It must produce userSupportedMeaning, supportCategory, and resolutionGuidance. + +**Should repair be allowed to alter mutation arrays? NO** + +Mutation integrity is the core invariant. The repair call must treat addedNodes/updatedNodes/addedEdges as frozen input context. Only semantic/action fields may be populated or changed. + +**Should repair be allowed to alter selectedQuestion? NO** + +selectedQuestion is derived from the mutation (nodeId references an unresolved unknown created or preserved by the mutation). Altering it would create a mismatch with the frozen mutation. Keep as-is. + +**Should repair receive raw user answer? YES** + +answerMeaning fields require understanding of the raw answer text. The repair call cannot produce userSupportedMeaning without the source material. + +**Should repair receive original proposal? YES** + +The repair call needs to see the original mutation arrays (as frozen context) and any existing non-null fields (to avoid overwriting). It must know what was already produced. + +--- + +## 6 — Call-Budget Consequence + +**Repair classification: SECOND-STAGE REPAIR** + +This is not a RETRY (retry implies failure + repetition of the same operation). This is not a NORMAL SECOND MODEL CALL (implies independent decision-making). This is a repair: it operates on an accepted-but-incomplete primary proposal, adding missing declarations without regenerating. + +**Can existing call accounting distinguish primary vs repair calls? NO** + +Current call accounting tracks `startCalls` and `updateCalls`. There is no distinction between primary proposals and repair sub-calls within those counts. A bounded repair would be invisible to current accounting unless classified as either start or update. + +**Requires tooling change: YES — for complete distinction, but minimal.** + +Existing accounting can approximate the distinction by noting that repairs only occur on `proposal_compatibility` rejections (as opposed to `proposal_validation`, `provider`, or `application` failures). No new counters needed if using stage-diagnosis as proxy. But clean separation would require a call type field. + +--- + +## 7 — Chose the Smallest Next Boundary + +### Choice: B — DECLARATION-ONLY REPAIR CALL + +**Why:** Option A (deterministic action fill) solves only half the problem (structuralActionRequired) and changes field semantics from declaration to inference. Option C (full regeneration) discards the entire point of this experiment (preserving good mutation). Option D (keep transition path) accepts the gap indefinitely without resolving it. + +Option B is the smallest design that: +1. Preserves the exact original mutation (no regeneration, no discard) +2. Recovers ALL missing fields (not just structuralActionRequired) +3. Retains declaration semantics (model still produces the declarations) +4. Is bounded (one call, forbidden from changing mutations) +5. Solves the 57J.84 case fully (both meaning and action recovered) + +The trade-off: one additional LLM call per affected proposal vs. semantic accountability gap. This trade-off is justified by the volume of proposals flowing through the null/null/mutation path (confirmed in 57J.84 as the dominant pattern for "implicit meaning through structure" cases). + +--- + +## 8 — Non-Negotiable Invariants + +For option B (declaration-only repair call): + +``` +original useful mutation preserved: YES +no keyword/synonym logic: YES +no mutation regeneration: YES +bounded additional model calls: 1 +provider-agnostic: YES +57J.84 information would survive: YES +``` + +--- + +## Chosen Boundary Summary + +The smallest recovery contract that preserves good mutation while recovering missing declarations is a bounded second-stage repair call that receives the raw answer and original proposal as context, produces only answerMeaning and structuralActionRequired fields, and is forbidden from touching any mutation arrays. This converts a null/null/mutation acceptance (57J.84) into a fully declared proposal with zero mutation change.