experiment: validate semantic-to-mutation contract live
This commit is contained in:
@@ -0,0 +1,133 @@
|
||||
# Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation
|
||||
|
||||
**Objective:** On one fresh live run, does the v0.17 prompt contract cause a faithful `userSupportedMeaning` to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||||
**Starting HEAD:** 712c0c4 docs: experiment 57J.39 record and handoff update
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful `userSupportedMeaning` should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.
|
||||
|
||||
A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17. The specific failure under test is faithful `userSupportedMeaning` plus zero structural mutation.
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
- **Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
- **maxUpdates:** 1
|
||||
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
- **Dev server:** REUSED EXISTING (HTTP 200)
|
||||
|
||||
## Run
|
||||
|
||||
### Call Accounting
|
||||
|
||||
startCalls: 1
|
||||
updateCalls: 1
|
||||
totalCalls: 2
|
||||
|
||||
### START
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
stage: unknown
|
||||
selected question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
|
||||
node count: 8
|
||||
edge count: 5
|
||||
```
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
stage: proposal_compatibility
|
||||
First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
selected question: null
|
||||
node count: 8 (unchanged)
|
||||
edge count: 5 (unchanged)
|
||||
```
|
||||
|
||||
**Rejected Proposal Snapshot:**
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
||||
"possibleInference": null
|
||||
},
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [
|
||||
{
|
||||
"id": "n_prereq_constraint",
|
||||
"kind": "assumption",
|
||||
"label": "Prerequisite condition for proceeding",
|
||||
"description": "Relocation decision requires verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["nqylvkl"],
|
||||
"affects": [],
|
||||
"childIds": []
|
||||
}
|
||||
],
|
||||
"addedEdges": [
|
||||
{
|
||||
"fromNodeId": "n_prereq_constraint",
|
||||
"toNodeId": "nqylvkl",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Analysis
|
||||
|
||||
### Meaning Fidelity
|
||||
|
||||
**Classification: STRENGTHENED**
|
||||
|
||||
The model transformed the raw answer:
|
||||
- **Raw:** "Before deciding, I need evidence that X and Y." (statement of information-need)
|
||||
- **Produced:** "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that X and Y." (prescriptive constraint on the decision)
|
||||
|
||||
This is a non-trivial semantic strengthening. The model converted a neutral report of what it needs ("I need evidence...") into prescriptive language about what the *decision* requires ("the decision is explicitly conditional on..."). This introduces a `conditional_qualification` meaning category stronger than the raw answer supports.
|
||||
|
||||
### Structural Mutation
|
||||
|
||||
```
|
||||
updatedNodes: 0
|
||||
resolvedUnknownNodeIds: 0
|
||||
addedNodes: 1 (n_prereq_constraint, kind=assumption)
|
||||
addedEdges: 1 (depends_on → nqylvkl)
|
||||
```
|
||||
|
||||
The model did produce minimal structural mutation (1 new node + 1 edge). However, this mutation is built on the strengthened meaning, not a faithful translation of the raw answer. The added node's label ("Prerequisite condition for proceeding") and description directly reflect the prescriptive framing introduced by the strengthening, not the neutral information-need stated by the user.
|
||||
|
||||
### Classification: C — CORRECT FIDELITY REJECTION
|
||||
|
||||
The model strengthened the raw answer beyond what it supports, and the existing semantic-fidelity validator correctly rejected this at `proposal_compatibility`. This is not a v0.17 semantic-to-mutation failure because the strengthening was caught at the semantic fidelity layer before reaching the mutation boundary.
|
||||
|
||||
## Did v0.17 remove the faithful semantic-only no-op failure?
|
||||
|
||||
**UNPROVEN**
|
||||
|
||||
This run did not test v0.17's core question because the model never produced a faithful `userSupportedMeaning`. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised. A faithful semantic-only no-op is neither reproduced nor disproved here.
|
||||
|
||||
## What this run establishes
|
||||
|
||||
1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in Experiments 57J.32, 57J.33.
|
||||
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
|
||||
3. When meaning is strengthened and rejected, the model's structural proposal reflects the strengthened framing rather than faithful translation.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs.
|
||||
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
|
||||
3. That strengthening avoidance would occur with different phrasing, domain, or model.
|
||||
4. That v0.17 works in any case where the model does produce faithful meaning.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Harness restored: YES
|
||||
## No-retry preserved: YES
|
||||
## Dev server disturbed: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
Reference in New Issue
Block a user