experiment: isolate semantic-to-mutation contract live
This commit is contained in:
+25
-1
@@ -676,4 +676,28 @@ This run did not test the v0.17 contract's core question because the model never
|
||||
**Ollama calls beyond harness count:** 0
|
||||
**Documentation updated:** YES
|
||||
|
||||
Now let me commit:
|
||||
---
|
||||
|
||||
### Experiment 57J.41 — Semantic-to-Mutation Contract Live Validation: Faithful Meaning Only
|
||||
|
||||
**Objective:** When the user introduces one simple, explicit unresolved uncertainty with no conditional/constraint language, does v0.17 translate that faithful meaning into structural graph progress rather than a semantic-only no-op?
|
||||
|
||||
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
**Live-call count:** start: 1, update: 1, total: 2
|
||||
|
||||
**START:** HTTP 200 | stage: unknown | nodes: 6 | edges: 3 | question: "What would clarify current annual operating costs and cost structure of the engineering team in this situation?"
|
||||
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
|
||||
**Rejected Proposal Snapshot:**
|
||||
- `userSupportedMeaning`: "The user is unsure whether the projected office savings from the relocation are realistic." — **FAITHFUL** (preserves uncertainty, no conditional/constraint language)
|
||||
- `possibleInference`: null
|
||||
- All structural fields: empty (zero updatedNodes, zero resolvedUnknownNodeIds, zero addedNodes, zero addedEdges)
|
||||
|
||||
**Classification: B — SAME SEMANTIC-ONLY NO-OP.** Meaning is FAITHFUL. All structural mutation fields are empty. The v0.17 MUST rule + validator diagnostic fires as designed, converting the original 57J.36 silent accepted no-op into an explicit rejected proposal with specific error. However, this does NOT prove that faithful meaning CAN produce graph progress — only that it cannot do so without structural mutation under v0.17's new contract.
|
||||
|
||||
**Key finding:** v0.17 successfully blocks the original failure (accepted semantic-only no-op) but does not establish a positive path for faithful meaning to produce graph progress. The open question remains: is there any valid pathway where faithful meaning translates to structural mutation, or does the MUST rule universally block it?
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Harness restored. No-retry preserved. Dev server disturbed: NO.
|
||||
@@ -0,0 +1,117 @@
|
||||
# Experiment 57J.41 — Semantic-to-Mutation Contract Live Validation: Faithful Meaning Only
|
||||
|
||||
**Objective:** When the user introduces one simple, explicit unresolved uncertainty with no conditional/constraint language, does v0.17 translate that faithful meaning into structural graph progress rather than a semantic-only no-op?
|
||||
|
||||
57J.40 could not test this because Qwen strengthened the original answer into a decision condition. This experiment deliberately removes that confound.
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||||
**Starting HEAD:** 39217b6 experiment: validate semantic-to-mutation contract live
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
The answer introduces one clear unresolved uncertainty: whether projected office savings are realistic. A faithful proposal should preserve that uncertainty and express its effect structurally, either by updating/refining equivalent existing graph structure or by adding justified new structure. `answerMeaning` alone with zero graph mutation is the specific failure under test.
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
- **Answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **maxUpdates:** 1
|
||||
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
- **Dev server:** REUSED EXISTING (HTTP 200)
|
||||
|
||||
## Run
|
||||
|
||||
### Call Accounting
|
||||
|
||||
startCalls: 1
|
||||
updateCalls: 1
|
||||
totalCalls: 2
|
||||
|
||||
### START
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
stage: unknown
|
||||
selected question: "What would clarify current annual operating costs and cost structure of the engineering team in this situation?"
|
||||
node count: 6
|
||||
edge count: 3
|
||||
```
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
stage: proposal_compatibility
|
||||
First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
selected question: null
|
||||
node count: 6 (unchanged)
|
||||
edge count: 3 (unchanged)
|
||||
```
|
||||
|
||||
**Rejected Proposal Snapshot:**
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
|
||||
"possibleInference": null
|
||||
},
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [],
|
||||
"addedEdges": []
|
||||
}
|
||||
```
|
||||
|
||||
## Analysis
|
||||
|
||||
### Meaning Fidelity
|
||||
|
||||
**Classification: FAITHFUL**
|
||||
|
||||
The `userSupportedMeaning` directly preserves the raw answer's uncertainty:
|
||||
- **Raw:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **Produced:** "The user is unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
No conditional language. No constraint language. No decision requirement. No priority statement. The model preserved the simple uncertainty about savings realism without strengthening or degradation.
|
||||
|
||||
`possibleInference` is null — appropriate for a direct, unambiguous single-dimension uncertainty.
|
||||
|
||||
### Structural Mutation
|
||||
|
||||
```
|
||||
updatedNodes: 0
|
||||
resolvedUnknownNodeIds: 0
|
||||
addedNodes: 0
|
||||
addedEdges: 0
|
||||
```
|
||||
|
||||
Zero structural mutation across all fields. This is a semantic-only no-op at the proposal level.
|
||||
|
||||
The rejection occurred at `proposal_compatibility` because the v0.17 MUST rule triggers when `userSupportedMeaning` is populated with zero structural mutation. The rejection error exactly matches the new contract diagnostic: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation."
|
||||
|
||||
### Classification: B — SAME SEMANTIC-ONLY NO-OP
|
||||
|
||||
Meaning is FAITHFUL. All structural mutation fields are empty.
|
||||
|
||||
However, this is not a silent semantic-only no-op (which was the original 57J.36 problem). It is an **explicitly rejected** semantic-only no-op enforced by the v0.17 MUST rule + validator diagnostic. The model produced faithful meaning but zero structural progress, and the new contract boundary caught it before graph mutation could occur.
|
||||
|
||||
## Did v0.17 remove the faithful semantic-only no-op failure?
|
||||
|
||||
**UNPROVEN for positive outcome.** v0.17 successfully converts what would have been an accepted semantic-only no-op into a rejected proposal with a specific diagnostic error. This confirms the v0.17 contract fix (Option B) is working as designed — it blocks faithfulness-verified but structurally-empty proposals.
|
||||
|
||||
However, v0.17 does NOT prove that faithful meaning CAN produce graph progress. It proves the opposite direction: that v0.17 prevents a semantically faithful proposal with zero structure from passing through. The open question remains unanswered — is there any valid pathway where faithful meaning translates to structural mutation under v0.17, or does the new constraint universally block it?
|
||||
|
||||
## What this run establishes
|
||||
|
||||
1. The configured model preserves the explicit uncertainty about savings realism without strengthening (direct improvement over 57J.40).
|
||||
2. The v0.17 MUST rule + validator diagnostic fires exactly as designed: faithful meaning with zero structural mutation → rejected at proposal_compatibility with specific error.
|
||||
3. The original 57J.36 failure pattern (accepted semantic-only no-op) is now blocked — the rejection is explicit and diagnostic.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. That faithful meaning CAN produce graph progress under v0.17.
|
||||
2. Whether the model can simultaneously preserve faithfulness AND add justified structure for this or other scenarios.
|
||||
3. Whether the MUST rule is too aggressive — it may block both no-ops and legitimate partial-progress proposals.
|
||||
4. That cold-start quality (6 nodes) affects the outcome — but cold-start variance was not the variable under test here.
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. Harness restored to original scenario/answers. No-retry preserved. Dev server disturbed: NO.
|
||||
Reference in New Issue
Block a user