experiment: validate semantic-to-mutation contract live
This commit is contained in:
@@ -564,3 +564,116 @@ Ollama calls: 0. Dev server disturbed: NO. Branch: feature/semantic-to-mutation-
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation
|
||||
|
||||
**Objective:** On one fresh live run, does the v0.17 prompt contract cause a faithful `userSupportedMeaning` to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?
|
||||
|
||||
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
**Fixed answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
|
||||
**Live-call count:** start: 1, update: 1, total: 2
|
||||
|
||||
**START:** HTTP 200 | stage: unknown | nodes: 8 | edges: 5 | question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
|
||||
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
|
||||
Rejected proposal snapshot: `userSupportedMeaning`: "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved." — this converts the user's information-need statement ("I need evidence that...") into prescriptive conditional language about what the *decision* requires, introducing a `conditional_qualification` meaning stronger than the raw answer. `possibleInference`: null. Structural proposal: 1 addedNode (`n_prereq_constraint`, kind=assumption) + 1 addedEdge (depends_on to state node).
|
||||
|
||||
**Meaning fidelity classification: STRENGTHENED.** The model transformed a neutral information-need ("Before deciding, I need evidence...") into prescriptive constraint language ("Proceeding...is explicitly conditional on obtaining verified evidence"). This introduces a stronger reasoning category not supported by the raw answer.
|
||||
|
||||
**Classification: C — CORRECT FIDELITY REJECTION.** The model strengthened the raw answer beyond what the user established, and the existing semantic-fidelity validator correctly rejected it at `proposal_compatibility`. This is not a v0.17 semantic-to-mutation failure — the strengthening was caught before any graph mutation could proceed.
|
||||
|
||||
**What this establishes:** (1) The configured model still maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario; (2) The existing semantic-fidelity guard correctly catches this strengthening at proposal_compatibility; (3) v0.17's MUST rule did not trigger in this run because the meaning was strengthened before reaching the mutation boundary.
|
||||
|
||||
**What it does NOT prove:** That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs; that v0.17's contract prevents the faithful semantic-only no-op when meaning is truly preserved; that strengthening avoidance would occur with different phrasing or in later turns.
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. Branch: `feature/semantic-to-mutation-contract-v0.17`.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation (Full Record)
|
||||
|
||||
**Objective:** Answer exactly: "On one fresh live run, does the v0.17 prompt contract cause a faithful `userSupportedMeaning` to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?"
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||||
**Starting HEAD:** 712c0c4 docs: experiment 57J.39 record and handoff update
|
||||
|
||||
**Pre-written expectation:**
|
||||
> The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful `userSupportedMeaning` should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.
|
||||
> A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17.
|
||||
|
||||
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
**Fixed answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
|
||||
**Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Dev server:** REUSED EXISTING (HTTP 200)
|
||||
|
||||
**CALL ACCOUNTING**
|
||||
startCalls: 1
|
||||
updateCalls: 1
|
||||
totalCalls: 2
|
||||
|
||||
Supplementary scripts used: NO
|
||||
Retries: 0
|
||||
|
||||
**START**
|
||||
Nodes: 8
|
||||
Edges: 5
|
||||
Selected question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
|
||||
|
||||
**UPDATE 1**
|
||||
HTTP: 422
|
||||
Stage: proposal_compatibility
|
||||
First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
Nodes: 8 (unchanged)
|
||||
Edges: 5 (unchanged)
|
||||
Selected question: null
|
||||
|
||||
**ANSWER MEANING**
|
||||
userSupportedMeaning: "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved."
|
||||
possibleInference: null
|
||||
|
||||
**Meaning fidelity:** STRENGTHENED
|
||||
The model transformed a neutral information-need ("Before deciding, I need evidence...") into prescriptive constraint language ("Proceeding...is explicitly conditional on obtaining verified evidence"). This introduces stronger reasoning category (conditional qualification) beyond what the raw answer supports. The transformation maps "I need evidence that X and Y" to "decision is conditionally dependent on X and Y" — a non-trivial semantic strengthening.
|
||||
|
||||
**STRUCTURAL PROPOSAL**
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: 1 (`n_prereq_constraint`, kind=assumption, label="Prerequisite condition for proceeding")
|
||||
addedEdges: 1 (depends_on from added node to state anchor `nqylvkl`)
|
||||
|
||||
updatedNodes count: 0
|
||||
resolvedUnknownNodeIds count: 0
|
||||
addedNodes count: 1
|
||||
addedEdges count: 1
|
||||
|
||||
**Structural mutation present:** YES
|
||||
Mutation type: ADD NODE
|
||||
|
||||
**Classification:** C — CORRECT FIDELITY REJECTION
|
||||
|
||||
**Why:** The model strengthened the raw answer beyond what it supports (information-need → prescriptive constraint), and the existing semantic-fidelity validator correctly rejected this at `proposal_compatibility`. The rejection was triggered by the strengthening detection, not by v0.17's semantic-to-mutation contract. The model did produce 1 added node and 1 edge, but these are downstream artifacts of the strengthened meaning — they do not represent faithful structural translation of the raw answer.
|
||||
|
||||
**Did v0.17 remove the faithful semantic-only no-op failure:** UNPROVEN
|
||||
This run did not test the v0.17 contract's core question because the model never produced a faithful `userSupportedMeaning` to begin with. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised in this run. A faithful semantic-only no-op (if it exists under v0.17) is neither reproduced nor disproved here.
|
||||
|
||||
**What this clean run establishes:**
|
||||
1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in earlier experiments (57J.32, 57J.33).
|
||||
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
|
||||
3. When meaning is strengthened and rejected, the model's structural proposal is built on the strengthened meaning (not a faithful translation), making mutation analysis irrelevant to v0.17.
|
||||
|
||||
**What it does NOT prove:**
|
||||
1. That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs.
|
||||
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
|
||||
3. That strengthening avoidance would occur with different phrasing, domain, or model.
|
||||
4. That v0.17 works in any case where the model does produce faithful meaning.
|
||||
|
||||
**Production code changed:** NO
|
||||
**Prompt changed during experiment:** NO
|
||||
**Canonical harness restored:** YES
|
||||
**Hardened no-retry behaviour preserved:** YES
|
||||
**Dev server disturbed:** NO
|
||||
**Ollama calls beyond harness count:** 0
|
||||
**Documentation updated:** YES
|
||||
|
||||
Now let me commit:
|
||||
@@ -0,0 +1,133 @@
|
||||
# Experiment 57J.40 — Semantic-to-Mutation Contract Live Validation
|
||||
|
||||
**Objective:** On one fresh live run, does the v0.17 prompt contract cause a faithful `userSupportedMeaning` to produce meaningful structural graph mutation instead of a semantic-only no-op proposal?
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||||
**Starting HEAD:** 712c0c4 docs: experiment 57J.39 record and handoff update
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful `userSupportedMeaning` should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.
|
||||
|
||||
A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17. The specific failure under test is faithful `userSupportedMeaning` plus zero structural mutation.
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
- **Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
- **maxUpdates:** 1
|
||||
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
- **Dev server:** REUSED EXISTING (HTTP 200)
|
||||
|
||||
## Run
|
||||
|
||||
### Call Accounting
|
||||
|
||||
startCalls: 1
|
||||
updateCalls: 1
|
||||
totalCalls: 2
|
||||
|
||||
### START
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
stage: unknown
|
||||
selected question: "What was the comparable state before current baseline costs vs. projected costs at target location?"
|
||||
node count: 8
|
||||
edge count: 5
|
||||
```
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
stage: proposal_compatibility
|
||||
First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
selected question: null
|
||||
node count: 8 (unchanged)
|
||||
edge count: 5 (unchanged)
|
||||
```
|
||||
|
||||
**Rejected Proposal Snapshot:**
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
||||
"possibleInference": null
|
||||
},
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [
|
||||
{
|
||||
"id": "n_prereq_constraint",
|
||||
"kind": "assumption",
|
||||
"label": "Prerequisite condition for proceeding",
|
||||
"description": "Relocation decision requires verified evidence that projected office savings are realistic and that key engineer retention is preserved.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["nqylvkl"],
|
||||
"affects": [],
|
||||
"childIds": []
|
||||
}
|
||||
],
|
||||
"addedEdges": [
|
||||
{
|
||||
"fromNodeId": "n_prereq_constraint",
|
||||
"toNodeId": "nqylvkl",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Analysis
|
||||
|
||||
### Meaning Fidelity
|
||||
|
||||
**Classification: STRENGTHENED**
|
||||
|
||||
The model transformed the raw answer:
|
||||
- **Raw:** "Before deciding, I need evidence that X and Y." (statement of information-need)
|
||||
- **Produced:** "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that X and Y." (prescriptive constraint on the decision)
|
||||
|
||||
This is a non-trivial semantic strengthening. The model converted a neutral report of what it needs ("I need evidence...") into prescriptive language about what the *decision* requires ("the decision is explicitly conditional on..."). This introduces a `conditional_qualification` meaning category stronger than the raw answer supports.
|
||||
|
||||
### Structural Mutation
|
||||
|
||||
```
|
||||
updatedNodes: 0
|
||||
resolvedUnknownNodeIds: 0
|
||||
addedNodes: 1 (n_prereq_constraint, kind=assumption)
|
||||
addedEdges: 1 (depends_on → nqylvkl)
|
||||
```
|
||||
|
||||
The model did produce minimal structural mutation (1 new node + 1 edge). However, this mutation is built on the strengthened meaning, not a faithful translation of the raw answer. The added node's label ("Prerequisite condition for proceeding") and description directly reflect the prescriptive framing introduced by the strengthening, not the neutral information-need stated by the user.
|
||||
|
||||
### Classification: C — CORRECT FIDELITY REJECTION
|
||||
|
||||
The model strengthened the raw answer beyond what it supports, and the existing semantic-fidelity validator correctly rejected this at `proposal_compatibility`. This is not a v0.17 semantic-to-mutation failure because the strengthening was caught at the semantic fidelity layer before reaching the mutation boundary.
|
||||
|
||||
## Did v0.17 remove the faithful semantic-only no-op failure?
|
||||
|
||||
**UNPROVEN**
|
||||
|
||||
This run did not test v0.17's core question because the model never produced a faithful `userSupportedMeaning`. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised. A faithful semantic-only no-op is neither reproduced nor disproved here.
|
||||
|
||||
## What this run establishes
|
||||
|
||||
1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in Experiments 57J.32, 57J.33.
|
||||
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
|
||||
3. When meaning is strengthened and rejected, the model's structural proposal reflects the strengthened framing rather than faithful translation.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs.
|
||||
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
|
||||
3. That strengthening avoidance would occur with different phrasing, domain, or model.
|
||||
4. That v0.17 works in any case where the model does produce faithful meaning.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Harness restored: YES
|
||||
## No-retry preserved: YES
|
||||
## Dev server disturbed: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
Reference in New Issue
Block a user