134 lines
5.9 KiB
Markdown
134 lines
5.9 KiB
Markdown
# Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation
|
|
|
|
**Objective:** On one fresh live run, does the v0.17 prompt contract cause a faithful `userSupportedMeaning` to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?
|
|
|
|
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
|
**Starting HEAD:** 712c0c4 docs: experiment 57J.39 record and handoff update
|
|
|
|
## Pre-written expectation
|
|
|
|
The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful `userSupportedMeaning` should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.
|
|
|
|
A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17. The specific failure under test is faithful `userSupportedMeaning` plus zero structural mutation.
|
|
|
|
## Configuration
|
|
|
|
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
- **Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
|
- **maxUpdates:** 1
|
|
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
- **Dev server:** REUSED EXISTING (HTTP 200)
|
|
|
|
## Run
|
|
|
|
### Call Accounting
|
|
|
|
startCalls: 1
|
|
updateCalls: 1
|
|
totalCalls: 2
|
|
|
|
### START
|
|
|
|
```
|
|
HTTP status: 200
|
|
stage: unknown
|
|
selected question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
|
|
node count: 8
|
|
edge count: 5
|
|
```
|
|
|
|
### UPDATE 1
|
|
|
|
```
|
|
HTTP status: 422
|
|
stage: proposal_compatibility
|
|
First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
|
selected question: null
|
|
node count: 8 (unchanged)
|
|
edge count: 5 (unchanged)
|
|
```
|
|
|
|
**Rejected Proposal Snapshot:**
|
|
```json
|
|
{
|
|
"answerMeaning": {
|
|
"userSupportedMeaning": "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
|
"possibleInference": null
|
|
},
|
|
"updatedNodes": [],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [
|
|
{
|
|
"id": "n_prereq_constraint",
|
|
"kind": "assumption",
|
|
"label": "Prerequisite condition for proceeding",
|
|
"description": "Relocation decision requires verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
|
"parentId": null,
|
|
"dependsOn": ["nqylvkl"],
|
|
"affects": [],
|
|
"childIds": []
|
|
}
|
|
],
|
|
"addedEdges": [
|
|
{
|
|
"fromNodeId": "n_prereq_constraint",
|
|
"toNodeId": "nqylvkl",
|
|
"relationship": "depends_on"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Analysis
|
|
|
|
### Meaning Fidelity
|
|
|
|
**Classification: STRENGTHENED**
|
|
|
|
The model transformed the raw answer:
|
|
- **Raw:** "Before deciding, I need evidence that X and Y." (statement of information-need)
|
|
- **Produced:** "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that X and Y." (prescriptive constraint on the decision)
|
|
|
|
This is a non-trivial semantic strengthening. The model converted a neutral report of what it needs ("I need evidence...") into prescriptive language about what the *decision* requires ("the decision is explicitly conditional on..."). This introduces a `conditional_qualification` meaning category stronger than the raw answer supports.
|
|
|
|
### Structural Mutation
|
|
|
|
```
|
|
updatedNodes: 0
|
|
resolvedUnknownNodeIds: 0
|
|
addedNodes: 1 (n_prereq_constraint, kind=assumption)
|
|
addedEdges: 1 (depends_on → nqylvkl)
|
|
```
|
|
|
|
The model did produce minimal structural mutation (1 new node + 1 edge). However, this mutation is built on the strengthened meaning, not a faithful translation of the raw answer. The added node's label ("Prerequisite condition for proceeding") and description directly reflect the prescriptive framing introduced by the strengthening, not the neutral information-need stated by the user.
|
|
|
|
### Classification: C — CORRECT FIDELITY REJECTION
|
|
|
|
The model strengthened the raw answer beyond what it supports, and the existing semantic-fidelity validator correctly rejected this at `proposal_compatibility`. This is not a v0.17 semantic-to-mutation failure because the strengthening was caught at the semantic fidelity layer before reaching the mutation boundary.
|
|
|
|
## Did v0.17 remove the faithful semantic-only no-op failure?
|
|
|
|
**UNPROVEN**
|
|
|
|
This run did not test v0.17's core question because the model never produced a faithful `userSupportedMeaning`. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised. A faithful semantic-only no-op is neither reproduced nor disproved here.
|
|
|
|
## What this run establishes
|
|
|
|
1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in Experiments 57J.32, 57J.33.
|
|
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
|
|
3. When meaning is strengthened and rejected, the model's structural proposal reflects the strengthened framing rather than faithful translation.
|
|
|
|
## What this does NOT prove
|
|
|
|
1. That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs.
|
|
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
|
|
3. That strengthening avoidance would occur with different phrasing, domain, or model.
|
|
4. That v0.17 works in any case where the model does produce faithful meaning.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed during experiment: NO
|
|
## Harness restored: YES
|
|
## No-retry preserved: YES
|
|
## Dev server disturbed: NO
|
|
## Ollama calls beyond harness count: 0
|