confidence-engine/docs/experiment-57j40.md

Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation

Objective: On one fresh live run, does the v0.17 prompt contract cause a faithful userSupportedMeaning to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?

Branch: feature/semantic-to-mutation-contract-v0.17 Starting HEAD: 712c0c4 docs: experiment 57J.39 record and handoff update

Pre-written expectation

The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful userSupportedMeaning should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.

A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17. The specific failure under test is faithful userSupportedMeaning plus zero structural mutation.

Configuration

• Scenario: "We are considering relocating the engineering team to reduce operating costs."
• Answer: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
• maxUpdates: 1
• Configured model: qwen-claude:latest at http://192.168.1.111:11434
• Dev server: REUSED EXISTING (HTTP 200)

Run

Call Accounting

startCalls: 1 updateCalls: 1 totalCalls: 2

START

HTTP status: 200
stage: unknown
selected question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
node count: 8
edge count: 5

UPDATE 1

HTTP status: 422
stage: proposal_compatibility
First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
selected question: null
node count: 8 (unchanged)
edge count: 5 (unchanged)

Rejected Proposal Snapshot:

{
  "answerMeaning": {
    "userSupportedMeaning": "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
    "possibleInference": null
  },
  "updatedNodes": [],
  "resolvedUnknownNodeIds": [],
  "addedNodes": [
    {
      "id": "n_prereq_constraint",
      "kind": "assumption",
      "label": "Prerequisite condition for proceeding",
      "description": "Relocation decision requires verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
      "parentId": null,
      "dependsOn": ["nqylvkl"],
      "affects": [],
      "childIds": []
    }
  ],
  "addedEdges": [
    {
      "fromNodeId": "n_prereq_constraint",
      "toNodeId": "nqylvkl",
      "relationship": "depends_on"
    }
  ]
}

Analysis

Meaning Fidelity

Classification: STRENGTHENED

The model transformed the raw answer:

• Raw: "Before deciding, I need evidence that X and Y." (statement of information-need)
• Produced: "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that X and Y." (prescriptive constraint on the decision)

This is a non-trivial semantic strengthening. The model converted a neutral report of what it needs ("I need evidence...") into prescriptive language about what the decision requires ("the decision is explicitly conditional on..."). This introduces a conditional_qualification meaning category stronger than the raw answer supports.

Structural Mutation

updatedNodes: 0
resolvedUnknownNodeIds: 0
addedNodes: 1 (n_prereq_constraint, kind=assumption)
addedEdges: 1 (depends_on → nqylvkl)

The model did produce minimal structural mutation (1 new node + 1 edge). However, this mutation is built on the strengthened meaning, not a faithful translation of the raw answer. The added node's label ("Prerequisite condition for proceeding") and description directly reflect the prescriptive framing introduced by the strengthening, not the neutral information-need stated by the user.

Classification: C — CORRECT FIDELITY REJECTION

The model strengthened the raw answer beyond what it supports, and the existing semantic-fidelity validator correctly rejected this at proposal_compatibility. This is not a v0.17 semantic-to-mutation failure because the strengthening was caught at the semantic fidelity layer before reaching the mutation boundary.

Did v0.17 remove the faithful semantic-only no-op failure?

UNPROVEN

This run did not test v0.17's core question because the model never produced a faithful userSupportedMeaning. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised. A faithful semantic-only no-op is neither reproduced nor disproved here.

What this run establishes

1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in Experiments 57J.32, 57J.33.
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
3. When meaning is strengthened and rejected, the model's structural proposal reflects the strengthened framing rather than faithful translation.

What this does NOT prove

1. That the configured model produces faithful userSupportedMeaning on this scenario under repeated runs.
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
3. That strengthening avoidance would occur with different phrasing, domain, or model.
4. That v0.17 works in any case where the model does produce faithful meaning.

Production code changed: NO

Prompt changed during experiment: NO

Harness restored: YES

No-retry preserved: YES

Dev server disturbed: NO

Ollama calls beyond harness count: 0