Files
confidence-engine/docs/experiment-57j68.md
T

6.3 KiB
Raw Blame History

Experiment 57J.68 — structuralActionRequired Implementation

Branch: feature/semantic-action-contract-v0.23 Starting HEAD: 5f9e8eb (experiment: finalize semantic action contract semantics) Parent design experiment: 57J.67 (contract semantics finalized — exact structural claim)

Objective

Bounded implementation of the settled structuralActionRequired contract from Experiment 57J.67 across production files, schema, validator, prompt, and deterministic test suite.

Classification: BOUNDED IMPLEMENTATION. All design decisions from 57J.67 implemented verbatim. No live Ollama calls. Zero semantic model invocations. Fully deterministic.


Implementation Summary

Production files changed (3 files):

  1. lib/graph/schema.js — Added structuralActionRequired: z.boolean().nullable().optional() to graphUpdateSchema.
  2. lib/graph/utils.js — Replaced the old semantic-only-no-op guard in validateGraphUpdate() with the full four-case contract validator. New checks (in order of evaluation):
    • Field-presence check: null/absent + populated userSupportedMeaning → reject
    • Four contradiction pairs evaluated: (true, no-mutation) REJECT, (false, mutation) REJECT, (true, mutation) PASS, (false, zero) PASS
    • Legacy "no meaningful change" guard retained only for non-contract paths (no userSupportedMeaning)
  3. lib/graph/prompt-builder.js — Added field name to Required JSON Field Names and Required Shapes sections; inserted new contract declaration section between numbered rules and Additional Guidance with two mandatory sentences telling the model when to set true vs false.

Schema transition behaviour:

  • Field is z.boolean().nullable().optional() — accepts true, false, null, or omission.
  • Missing/absent + populated userSupportedMeaning → contract-level rejection (not schema error).
  • Fully backward-compatible: old proposals without the field behave identically to the legacy path.

Strict four-case validator contract:

structuralActionRequired hasMeaningfulChange Outcome Error
true true PASS
true false REJECT "structuralActionRequired is true but proposal contains no graph mutation"
false false PASS (intentional no-op)
false true REJECT "structuralActionRequired is false but proposal contains meaningful mutations"

Prompt contract:

Two mandatory sentences inserted into the prompt under a new ## Contract: structuralActionRequired Declaration Rule section:

  1. "Set to true when your proposal contains any meaningful graph change (new nodes, updated nodes, resolved unknowns, or changed edges)."
  2. "Set to false only when the user's supported meaning is already fully represented in existing graph state and no graph mutation is needed."

Focused deterministic tests (50 new + 8 migrated):

tests/graph/schema.test.js (+4 tests):

  • Allows structuralActionRequired: true
  • Allows structuralActionRequired: false
  • Allows null structuralActionRequired
  • Omits by default (undefined is valid)

tests/graph/prompt-builder.test.js (+10 tests):

  • Field name appears in Required JSON Field Names
  • Contract section heading exists with exact text
  • First sentence references userSupportedMeaning trigger
  • true condition references addedNodes.length
  • false condition references zero structural mutations
  • Existing semantic fidelity rules remain intact (supportCategory, resolutionGuidance)
  • No provider-specific wording added
  • Additional Guidance section preserved
  • Rule numbering unchanged (132 contiguous)
  • Contract section positioned between rules and Additional Guidance

tests/graph/utils.test.js (+10 tests, 8 migrated):

  • true + meaningful mutation → PASS
  • true + zero mutation → REJECT
  • false + zero mutation with populated meaning → PASS (valid intentional no-op)
  • false + meaningful mutation → REJECT
  • null structuralActionRequired + populated meaning → REJECT (field required)
  • absent structuralActionRequired + populated meaning → REJECT (field required)
  • null structuralActionRequired + no meaning → retain existing behavior ("no meaningful change")
  • false+zero validation passes only for contract consistency, not semantic truth
  • hasMeaningfulChange logic unchanged for non-contract paths
  • supportCategory remains independent of structuralActionRequired

Migrated 8 existing tests that previously used userSupportedMeaning assertions to use structuralActionRequired: true where answerMeaning is populated.

Test accounting:

Category Count
New tests added 24 (4 in schema + 10 in prompt-builder + 10 in utils)
Existing tests migrated/modified 8 (in utils.test.js and prompt-builder.test.js test comments/data)
Pre-existing unchanged All other existing tests pass as-is

Test results:

All 197 graph tests pass across schema.test.js, prompt-builder.test.js, and utils.test.js.


What the implementation guarantees:

  1. Any proposal with populated userSupportedMeaning MUST include structuralActionRequired as a boolean.
  2. The declaration is an exact claim about output shape: true iff meaningful mutation exists; false iff zero mutations are intentional.
  3. false + zero is a valid no-op (contract-consistent) — the old "semantic-only rejection" no longer blocks it under contract.
  4. true/false mismatch on output shape is deterministically rejected with specific error messages.

What remains intentionally unresolved:

  • Prompt enforcement without runtime validation of model outputs (models may still send wrong values; the schema-level guard only helps downstream consumers).
  • The semantic correctness of false + zero is not validated — the validator confirms contract consistency, not whether the model's judgment was actually correct.
  • No migration plan for callers that currently produce answerMeaning without structuralActionRequired.

Live regression readiness:

  • All existing schema, prompt-builder, and utils tests pass.
  • The change is backward-compatible: field is optional by default; old proposals without it behave identically to the legacy path.

Classification: IMPLEMENTATION COMPLETE. Design decisions from 57J.67 applied verbatim. No live model calls. No semantic modifications. Ready for clean live regression on feature/semantic-action-contract-v0.23.