118 lines
5.7 KiB
Markdown
118 lines
5.7 KiB
Markdown
# Experiment 57J.41 — Semantic-to-Mutation Contract Live Validation: Faithful Meaning Only
|
|
|
|
**Objective:** When the user introduces one simple, explicit unresolved uncertainty with no conditional/constraint language, does v0.17 translate that faithful meaning into structural graph progress rather than a semantic-only no-op?
|
|
|
|
57J.40 could not test this because Qwen strengthened the original answer into a decision condition. This experiment deliberately removes that confound.
|
|
|
|
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
|
**Starting HEAD:** 39217b6 experiment: validate semantic-to-mutation contract live
|
|
|
|
## Pre-written expectation
|
|
|
|
The answer introduces one clear unresolved uncertainty: whether projected office savings are realistic. A faithful proposal should preserve that uncertainty and express its effect structurally, either by updating/refining equivalent existing graph structure or by adding justified new structure. `answerMeaning` alone with zero graph mutation is the specific failure under test.
|
|
|
|
## Configuration
|
|
|
|
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
- **Answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
|
- **maxUpdates:** 1
|
|
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
- **Dev server:** REUSED EXISTING (HTTP 200)
|
|
|
|
## Run
|
|
|
|
### Call Accounting
|
|
|
|
startCalls: 1
|
|
updateCalls: 1
|
|
totalCalls: 2
|
|
|
|
### START
|
|
|
|
```
|
|
HTTP status: 200
|
|
stage: unknown
|
|
selected question: "What would clarify current annual operating costs and cost structure of the engineering team in this situation?"
|
|
node count: 6
|
|
edge count: 3
|
|
```
|
|
|
|
### UPDATE 1
|
|
|
|
```
|
|
HTTP status: 422
|
|
stage: proposal_compatibility
|
|
First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
|
selected question: null
|
|
node count: 6 (unchanged)
|
|
edge count: 3 (unchanged)
|
|
```
|
|
|
|
**Rejected Proposal Snapshot:**
|
|
```json
|
|
{
|
|
"answerMeaning": {
|
|
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
|
|
"possibleInference": null
|
|
},
|
|
"updatedNodes": [],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [],
|
|
"addedEdges": []
|
|
}
|
|
```
|
|
|
|
## Analysis
|
|
|
|
### Meaning Fidelity
|
|
|
|
**Classification: FAITHFUL**
|
|
|
|
The `userSupportedMeaning` directly preserves the raw answer's uncertainty:
|
|
- **Raw:** "I am unsure whether the projected office savings from the relocation are realistic."
|
|
- **Produced:** "The user is unsure whether the projected office savings from the relocation are realistic."
|
|
|
|
No conditional language. No constraint language. No decision requirement. No priority statement. The model preserved the simple uncertainty about savings realism without strengthening or degradation.
|
|
|
|
`possibleInference` is null — appropriate for a direct, unambiguous single-dimension uncertainty.
|
|
|
|
### Structural Mutation
|
|
|
|
```
|
|
updatedNodes: 0
|
|
resolvedUnknownNodeIds: 0
|
|
addedNodes: 0
|
|
addedEdges: 0
|
|
```
|
|
|
|
Zero structural mutation across all fields. This is a semantic-only no-op at the proposal level.
|
|
|
|
The rejection occurred at `proposal_compatibility` because the v0.17 MUST rule triggers when `userSupportedMeaning` is populated with zero structural mutation. The rejection error exactly matches the new contract diagnostic: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation."
|
|
|
|
### Classification: B — SAME SEMANTIC-ONLY NO-OP
|
|
|
|
Meaning is FAITHFUL. All structural mutation fields are empty.
|
|
|
|
However, this is not a silent semantic-only no-op (which was the original 57J.36 problem). It is an **explicitly rejected** semantic-only no-op enforced by the v0.17 MUST rule + validator diagnostic. The model produced faithful meaning but zero structural progress, and the new contract boundary caught it before graph mutation could occur.
|
|
|
|
## Did v0.17 remove the faithful semantic-only no-op failure?
|
|
|
|
**UNPROVEN for positive outcome.** v0.17 successfully converts what would have been an accepted semantic-only no-op into a rejected proposal with a specific diagnostic error. This confirms the v0.17 contract fix (Option B) is working as designed — it blocks faithfulness-verified but structurally-empty proposals.
|
|
|
|
However, v0.17 does NOT prove that faithful meaning CAN produce graph progress. It proves the opposite direction: that v0.17 prevents a semantically faithful proposal with zero structure from passing through. The open question remains unanswered — is there any valid pathway where faithful meaning translates to structural mutation under v0.17, or does the new constraint universally block it?
|
|
|
|
## What this run establishes
|
|
|
|
1. The configured model preserves the explicit uncertainty about savings realism without strengthening (direct improvement over 57J.40).
|
|
2. The v0.17 MUST rule + validator diagnostic fires exactly as designed: faithful meaning with zero structural mutation → rejected at proposal_compatibility with specific error.
|
|
3. The original 57J.36 failure pattern (accepted semantic-only no-op) is now blocked — the rejection is explicit and diagnostic.
|
|
|
|
## What this does NOT prove
|
|
|
|
1. That faithful meaning CAN produce graph progress under v0.17.
|
|
2. Whether the model can simultaneously preserve faithfulness AND add justified structure for this or other scenarios.
|
|
3. Whether the MUST rule is too aggressive — it may block both no-ops and legitimate partial-progress proposals.
|
|
4. That cold-start quality (6 nodes) affects the outcome — but cold-start variance was not the variable under test here.
|
|
|
|
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. Harness restored to original scenario/answers. No-retry preserved. Dev server disturbed: NO.
|