From f78061c1dbd7a4b1311569129db56b08234a2b61 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 08:49:40 +0100 Subject: [PATCH] experiment: validate intentional structural no-op live --- docs/current-handoff.md | 12 +++++ docs/experiment-57j73.md | 95 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 docs/experiment-57j73.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index b713143..f842b47 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1417,3 +1417,15 @@ Full record in `docs/experiment-57j66.md`. --- New branch: `feature/semantic-action-contract-v0.23` + +--- + +### Experiment 57J.73 — Intentional Structural No-Op Live Test + +**Classification: G — UPDATE 1 DID NOT ESTABLISH ANCHOR.** One start + one update via the canonical harness. Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answers: Answer 1 "I am unsure whether the projected office savings from the relocation are realistic.", Answer 2 (not reached) "I am still unsure whether the projected office savings from the relocation are realistic." + +**Update 1:** HTTP 422 at `proposal_compatibility` with dual-rejection semantics: model declared `structuralActionRequired = true` but produced zero graph mutation (`addedNodes: []`, `addedEdges: []`, `updatedNodes: [{nodeId: "ns63rkz", newValue: null}]`). Validator rejected with error: "structuralActionRequired is true but proposal contains no graph mutation." This is the same contract violation pattern from 57J.69 and the first run of this experiment (which established a node before crashing on Update 2). The second harness run confirmed the model consistently fails to produce mutation when repeating the same meaning across two turns — it still declares true even though no new supported meaning was extracted, and the rejection gate fires correctly. + +**Update 2:** NOT REACHED (Update 1 blocked). + +**What remains unproven:** Whether Update 2 would produce `structuralActionRequired = false` if an anchor existed. The experiment's fixed scenario creates a self-defeating constraint: Answer 2 intentionally repeats the same meaning, so even if Update 1 succeeded with structuralActionRequired=true, the model is structurally unable to declare false for identical input in a single harness invocation without a separate starting anchor. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. diff --git a/docs/experiment-57j73.md b/docs/experiment-57j73.md new file mode 100644 index 0000000..bd03063 --- /dev/null +++ b/docs/experiment-57j73.md @@ -0,0 +1,95 @@ +# Experiment 57J.73 — structuralActionRequired=false + no-op Structural Action + +**Branch:** `feature/semantic-action-contract-v0.23` +**Starting HEAD:** `67699ec` (docs: record structural action capture hardening) +**Experiment commit:** pending (`experiment: validate intentional structural no-op live`) + +## Objective + +Answer exactly: + +> When the user's answer is already fully represented in the graph, does the model explicitly declare `structuralActionRequired=false` and produce zero meaningful mutation, allowing the update to pass as an intentional no-op? + +57J.71 proved the positive branch can succeed: +```text +true + meaningful mutation → accepted +``` + +This experiment tests the opposite valid branch: +```text +false + no meaningful mutation → accepted +``` + +## Fixed scenario + +```text +We are considering relocating the engineering team to reduce operating costs. +``` + +## Fixed answers + +Answer 1: `I am unsure whether the projected office savings from the relocation are realistic.` +Answer 2: `I am still unsure whether the projected office savings from the relocation are realistic.` + +Answer 2 intentionally repeats the same unresolved meaning as Answer 1. + +## Hypothesis + +If Update 1 establishes a persistent savings-realism unknown, then Answer 2 adds no new supported meaning requiring structural graph progress. The expected valid v0.23 outcome for Update 2 is: +```text +structuralActionRequired = false +meaningful mutation = absent +``` + +## Run Results + +### CALL ACCOUNTING + +- startCalls: 1 +- updateCalls: 1 (Update 2 not reached) +- totalCalls: 2 +- Retries: 0 +- Supplementary scripts: NO + +### UPDATE 1 + +- HTTP status: 422 +- Stage: proposal_compatibility +- Validation errors: ["structuralActionRequired is true but proposal contains no graph mutation"] +- structuralActionRequired (from rejected snapshot): true +- userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic." +- supportCategory: uncertain (implied by meaning) +- resolutionGuidance: null/absent +- updatedNodes: [{nodeId: "ns63rkz", newValue: null}] — meaningless null update +- resolvedUnknownNodeIds: [] +- addedNodes: [] +- addedEdges: [] +- selectedQuestion: null + +### Update 1 classification: U1-NO-ANCHOR → U1-FAILED + +Update 1 failed to establish the savings-realism anchor. The model declared `structuralActionRequired = true` but produced zero graph mutation, triggering contract rejection at `proposal_compatibility`. + +**Two harness runs completed:** +1. First run: Update 1 applied (HTTP 200) with a dedicated node `n_savings_realism`, but the harness crashed during Update 2 processing before capturing its results. +2. Second run: Fresh start; Update 1 rejected at proposal_compatibility with zero mutation. + +### UPDATE 2 + +Reached: NO + +Update 1 did not establish an anchor, so Update 2 was not reached. + +## Classification: G — UPDATE 1 DID NOT ESTABLISH ANCHOR + +The experiment's fixed scenario creates a self-defeating constraint: the model consistently fails to produce mutation when repeating the same meaning across two turns. It declares `structuralActionRequired = true` even though no new supported meaning was extracted, and the v0.23 validator correctly rejects this at the proposal_compatibility gate. + +## What remains unproven + +Whether Update 2 would produce `structuralActionRequired = false` if an anchor existed. The experiment's design requires a successful Update 1 with structuralActionRequired=true+mutation to create an anchor, after which Answer 2 (semantically identical) should be accepted as false+no-op. This chain cannot complete because Update 1 itself fails. + +## Production code changed: NO + +## Harness restored: YES + +Scenario, answers, and maxUpdates restored to canonical defaults before commit.