experiment: isolate semantic-to-mutation contract live
This commit is contained in:
+25
-1
@@ -676,4 +676,28 @@ This run did not test the v0.17 contract's core question because the model never
|
|||||||
**Ollama calls beyond harness count:** 0
|
**Ollama calls beyond harness count:** 0
|
||||||
**Documentation updated:** YES
|
**Documentation updated:** YES
|
||||||
|
|
||||||
Now let me commit:
|
---
|
||||||
|
|
||||||
|
### Experiment 57J.41 — Semantic-to-Mutation Contract Live Validation: Faithful Meaning Only
|
||||||
|
|
||||||
|
**Objective:** When the user introduces one simple, explicit unresolved uncertainty with no conditional/constraint language, does v0.17 translate that faithful meaning into structural graph progress rather than a semantic-only no-op?
|
||||||
|
|
||||||
|
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||||
|
**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||||
|
|
||||||
|
**Live-call count:** start: 1, update: 1, total: 2
|
||||||
|
|
||||||
|
**START:** HTTP 200 | stage: unknown | nodes: 6 | edges: 3 | question: "What would clarify current annual operating costs and cost structure of the engineering team in this situation?"
|
||||||
|
|
||||||
|
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||||
|
|
||||||
|
**Rejected Proposal Snapshot:**
|
||||||
|
- `userSupportedMeaning`: "The user is unsure whether the projected office savings from the relocation are realistic." — **FAITHFUL** (preserves uncertainty, no conditional/constraint language)
|
||||||
|
- `possibleInference`: null
|
||||||
|
- All structural fields: empty (zero updatedNodes, zero resolvedUnknownNodeIds, zero addedNodes, zero addedEdges)
|
||||||
|
|
||||||
|
**Classification: B — SAME SEMANTIC-ONLY NO-OP.** Meaning is FAITHFUL. All structural mutation fields are empty. The v0.17 MUST rule + validator diagnostic fires as designed, converting the original 57J.36 silent accepted no-op into an explicit rejected proposal with specific error. However, this does NOT prove that faithful meaning CAN produce graph progress — only that it cannot do so without structural mutation under v0.17's new contract.
|
||||||
|
|
||||||
|
**Key finding:** v0.17 successfully blocks the original failure (accepted semantic-only no-op) but does not establish a positive path for faithful meaning to produce graph progress. The open question remains: is there any valid pathway where faithful meaning translates to structural mutation, or does the MUST rule universally block it?
|
||||||
|
|
||||||
|
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Harness restored. No-retry preserved. Dev server disturbed: NO.
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
# Experiment 57J.41 — Semantic-to-Mutation Contract Live Validation: Faithful Meaning Only
|
||||||
|
|
||||||
|
**Objective:** When the user introduces one simple, explicit unresolved uncertainty with no conditional/constraint language, does v0.17 translate that faithful meaning into structural graph progress rather than a semantic-only no-op?
|
||||||
|
|
||||||
|
57J.40 could not test this because Qwen strengthened the original answer into a decision condition. This experiment deliberately removes that confound.
|
||||||
|
|
||||||
|
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||||||
|
**Starting HEAD:** 39217b6 experiment: validate semantic-to-mutation contract live
|
||||||
|
|
||||||
|
## Pre-written expectation
|
||||||
|
|
||||||
|
The answer introduces one clear unresolved uncertainty: whether projected office savings are realistic. A faithful proposal should preserve that uncertainty and express its effect structurally, either by updating/refining equivalent existing graph structure or by adding justified new structure. `answerMeaning` alone with zero graph mutation is the specific failure under test.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||||
|
- **Answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||||
|
- **maxUpdates:** 1
|
||||||
|
- **Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||||
|
- **Dev server:** REUSED EXISTING (HTTP 200)
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
### Call Accounting
|
||||||
|
|
||||||
|
startCalls: 1
|
||||||
|
updateCalls: 1
|
||||||
|
totalCalls: 2
|
||||||
|
|
||||||
|
### START
|
||||||
|
|
||||||
|
```
|
||||||
|
HTTP status: 200
|
||||||
|
stage: unknown
|
||||||
|
selected question: "What would clarify current annual operating costs and cost structure of the engineering team in this situation?"
|
||||||
|
node count: 6
|
||||||
|
edge count: 3
|
||||||
|
```
|
||||||
|
|
||||||
|
### UPDATE 1
|
||||||
|
|
||||||
|
```
|
||||||
|
HTTP status: 422
|
||||||
|
stage: proposal_compatibility
|
||||||
|
First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||||
|
selected question: null
|
||||||
|
node count: 6 (unchanged)
|
||||||
|
edge count: 3 (unchanged)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rejected Proposal Snapshot:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"answerMeaning": {
|
||||||
|
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
|
||||||
|
"possibleInference": null
|
||||||
|
},
|
||||||
|
"updatedNodes": [],
|
||||||
|
"resolvedUnknownNodeIds": [],
|
||||||
|
"addedNodes": [],
|
||||||
|
"addedEdges": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Analysis
|
||||||
|
|
||||||
|
### Meaning Fidelity
|
||||||
|
|
||||||
|
**Classification: FAITHFUL**
|
||||||
|
|
||||||
|
The `userSupportedMeaning` directly preserves the raw answer's uncertainty:
|
||||||
|
- **Raw:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||||
|
- **Produced:** "The user is unsure whether the projected office savings from the relocation are realistic."
|
||||||
|
|
||||||
|
No conditional language. No constraint language. No decision requirement. No priority statement. The model preserved the simple uncertainty about savings realism without strengthening or degradation.
|
||||||
|
|
||||||
|
`possibleInference` is null — appropriate for a direct, unambiguous single-dimension uncertainty.
|
||||||
|
|
||||||
|
### Structural Mutation
|
||||||
|
|
||||||
|
```
|
||||||
|
updatedNodes: 0
|
||||||
|
resolvedUnknownNodeIds: 0
|
||||||
|
addedNodes: 0
|
||||||
|
addedEdges: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Zero structural mutation across all fields. This is a semantic-only no-op at the proposal level.
|
||||||
|
|
||||||
|
The rejection occurred at `proposal_compatibility` because the v0.17 MUST rule triggers when `userSupportedMeaning` is populated with zero structural mutation. The rejection error exactly matches the new contract diagnostic: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation."
|
||||||
|
|
||||||
|
### Classification: B — SAME SEMANTIC-ONLY NO-OP
|
||||||
|
|
||||||
|
Meaning is FAITHFUL. All structural mutation fields are empty.
|
||||||
|
|
||||||
|
However, this is not a silent semantic-only no-op (which was the original 57J.36 problem). It is an **explicitly rejected** semantic-only no-op enforced by the v0.17 MUST rule + validator diagnostic. The model produced faithful meaning but zero structural progress, and the new contract boundary caught it before graph mutation could occur.
|
||||||
|
|
||||||
|
## Did v0.17 remove the faithful semantic-only no-op failure?
|
||||||
|
|
||||||
|
**UNPROVEN for positive outcome.** v0.17 successfully converts what would have been an accepted semantic-only no-op into a rejected proposal with a specific diagnostic error. This confirms the v0.17 contract fix (Option B) is working as designed — it blocks faithfulness-verified but structurally-empty proposals.
|
||||||
|
|
||||||
|
However, v0.17 does NOT prove that faithful meaning CAN produce graph progress. It proves the opposite direction: that v0.17 prevents a semantically faithful proposal with zero structure from passing through. The open question remains unanswered — is there any valid pathway where faithful meaning translates to structural mutation under v0.17, or does the new constraint universally block it?
|
||||||
|
|
||||||
|
## What this run establishes
|
||||||
|
|
||||||
|
1. The configured model preserves the explicit uncertainty about savings realism without strengthening (direct improvement over 57J.40).
|
||||||
|
2. The v0.17 MUST rule + validator diagnostic fires exactly as designed: faithful meaning with zero structural mutation → rejected at proposal_compatibility with specific error.
|
||||||
|
3. The original 57J.36 failure pattern (accepted semantic-only no-op) is now blocked — the rejection is explicit and diagnostic.
|
||||||
|
|
||||||
|
## What this does NOT prove
|
||||||
|
|
||||||
|
1. That faithful meaning CAN produce graph progress under v0.17.
|
||||||
|
2. Whether the model can simultaneously preserve faithfulness AND add justified structure for this or other scenarios.
|
||||||
|
3. Whether the MUST rule is too aggressive — it may block both no-ops and legitimate partial-progress proposals.
|
||||||
|
4. That cold-start quality (6 nodes) affects the outcome — but cold-start variance was not the variable under test here.
|
||||||
|
|
||||||
|
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. Harness restored to original scenario/answers. No-retry preserved. Dev server disturbed: NO.
|
||||||
Reference in New Issue
Block a user