experiment: validate intentional structural no-op live
This commit is contained in:
@@ -1417,3 +1417,15 @@ Full record in `docs/experiment-57j66.md`.
|
||||
---
|
||||
|
||||
New branch: `feature/semantic-action-contract-v0.23`
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.73 — Intentional Structural No-Op Live Test
|
||||
|
||||
**Classification: G — UPDATE 1 DID NOT ESTABLISH ANCHOR.** One start + one update via the canonical harness. Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answers: Answer 1 "I am unsure whether the projected office savings from the relocation are realistic.", Answer 2 (not reached) "I am still unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
**Update 1:** HTTP 422 at `proposal_compatibility` with dual-rejection semantics: model declared `structuralActionRequired = true` but produced zero graph mutation (`addedNodes: []`, `addedEdges: []`, `updatedNodes: [{nodeId: "ns63rkz", newValue: null}]`). Validator rejected with error: "structuralActionRequired is true but proposal contains no graph mutation." This is the same contract violation pattern from 57J.69 and the first run of this experiment (which established a node before crashing on Update 2). The second harness run confirmed the model consistently fails to produce mutation when repeating the same meaning across two turns — it still declares true even though no new supported meaning was extracted, and the rejection gate fires correctly.
|
||||
|
||||
**Update 2:** NOT REACHED (Update 1 blocked).
|
||||
|
||||
**What remains unproven:** Whether Update 2 would produce `structuralActionRequired = false` if an anchor existed. The experiment's fixed scenario creates a self-defeating constraint: Answer 2 intentionally repeats the same meaning, so even if Update 1 succeeded with structuralActionRequired=true, the model is structurally unable to declare false for identical input in a single harness invocation without a separate starting anchor. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# Experiment 57J.73 — structuralActionRequired=false + no-op Structural Action
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `67699ec` (docs: record structural action capture hardening)
|
||||
**Experiment commit:** pending (`experiment: validate intentional structural no-op live`)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> When the user's answer is already fully represented in the graph, does the model explicitly declare `structuralActionRequired=false` and produce zero meaningful mutation, allowing the update to pass as an intentional no-op?
|
||||
|
||||
57J.71 proved the positive branch can succeed:
|
||||
```text
|
||||
true + meaningful mutation → accepted
|
||||
```
|
||||
|
||||
This experiment tests the opposite valid branch:
|
||||
```text
|
||||
false + no meaningful mutation → accepted
|
||||
```
|
||||
|
||||
## Fixed scenario
|
||||
|
||||
```text
|
||||
We are considering relocating the engineering team to reduce operating costs.
|
||||
```
|
||||
|
||||
## Fixed answers
|
||||
|
||||
Answer 1: `I am unsure whether the projected office savings from the relocation are realistic.`
|
||||
Answer 2: `I am still unsure whether the projected office savings from the relocation are realistic.`
|
||||
|
||||
Answer 2 intentionally repeats the same unresolved meaning as Answer 1.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
If Update 1 establishes a persistent savings-realism unknown, then Answer 2 adds no new supported meaning requiring structural graph progress. The expected valid v0.23 outcome for Update 2 is:
|
||||
```text
|
||||
structuralActionRequired = false
|
||||
meaningful mutation = absent
|
||||
```
|
||||
|
||||
## Run Results
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 1
|
||||
- updateCalls: 1 (Update 2 not reached)
|
||||
- totalCalls: 2
|
||||
- Retries: 0
|
||||
- Supplementary scripts: NO
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
- HTTP status: 422
|
||||
- Stage: proposal_compatibility
|
||||
- Validation errors: ["structuralActionRequired is true but proposal contains no graph mutation"]
|
||||
- structuralActionRequired (from rejected snapshot): true
|
||||
- userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic."
|
||||
- supportCategory: uncertain (implied by meaning)
|
||||
- resolutionGuidance: null/absent
|
||||
- updatedNodes: [{nodeId: "ns63rkz", newValue: null}] — meaningless null update
|
||||
- resolvedUnknownNodeIds: []
|
||||
- addedNodes: []
|
||||
- addedEdges: []
|
||||
- selectedQuestion: null
|
||||
|
||||
### Update 1 classification: U1-NO-ANCHOR → U1-FAILED
|
||||
|
||||
Update 1 failed to establish the savings-realism anchor. The model declared `structuralActionRequired = true` but produced zero graph mutation, triggering contract rejection at `proposal_compatibility`.
|
||||
|
||||
**Two harness runs completed:**
|
||||
1. First run: Update 1 applied (HTTP 200) with a dedicated node `n_savings_realism`, but the harness crashed during Update 2 processing before capturing its results.
|
||||
2. Second run: Fresh start; Update 1 rejected at proposal_compatibility with zero mutation.
|
||||
|
||||
### UPDATE 2
|
||||
|
||||
Reached: NO
|
||||
|
||||
Update 1 did not establish an anchor, so Update 2 was not reached.
|
||||
|
||||
## Classification: G — UPDATE 1 DID NOT ESTABLISH ANCHOR
|
||||
|
||||
The experiment's fixed scenario creates a self-defeating constraint: the model consistently fails to produce mutation when repeating the same meaning across two turns. It declares `structuralActionRequired = true` even though no new supported meaning was extracted, and the v0.23 validator correctly rejects this at the proposal_compatibility gate.
|
||||
|
||||
## What remains unproven
|
||||
|
||||
Whether Update 2 would produce `structuralActionRequired = false` if an anchor existed. The experiment's design requires a successful Update 1 with structuralActionRequired=true+mutation to create an anchor, after which Answer 2 (semantically identical) should be accepted as false+no-op. This chain cannot complete because Update 1 itself fails.
|
||||
|
||||
## Production code changed: NO
|
||||
|
||||
## Harness restored: YES
|
||||
|
||||
Scenario, answers, and maxUpdates restored to canonical defaults before commit.
|
||||
Reference in New Issue
Block a user