From c899ad620ca031a6dc794a45d4c95ca68c07eac5 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 08:04:41 +0100 Subject: [PATCH] experiment: validate structural action contract live --- docs/current-handoff.md | 6 ++ docs/experiment-57j69.md | 150 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 docs/experiment-57j69.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index d4ed91a..6b137d7 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -233,6 +233,12 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult --- +### Experiment 57J.69 — structuralActionRequired Live Population and Contract Enforcement + +**Classification: C — TRUE/NO-MUTATION CONTRADICTION.** One start + one update (fixed scenario: "We are considering relocating the engineering team to reduce operating costs." / fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."). Start HTTP 200 (5 nodes, 3 edges). Update 1 returned HTTP 422 at `proposal_compatibility`. Model populated `structuralActionRequired = true` and extracted `userSupportedMeaning` ("User is unsure...") with supportCategory="uncertain". Proposal contained only `[updatedNodes: [{nodeId: nqx00rq, newValue: null}]]` — no added nodes, no added edges, meaningfully zero mutation. Validator rejected with dual errors: new contract rule ("structuralActionRequired is true but proposal contains no graph mutation") + legacy guard ("answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"). **What this establishes:** (1) `structuralActionRequired` IS populated for savings-realism uncertainty; (2) v0.23 validator ENFORCES the strict declaration-to-mutation contract; (3) rejection fires at correct stage before any mutation. **What remains unproven:** whether model can produce true+mutation in a single attempt, stability across runs, prompt enforcement adequacy. The remaining gap: model knows it should act structurally but fails to produce the mutation itself — same prompt-enforcement gap from 57J.64. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j69.md`. + +--- + ### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test **Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`. diff --git a/docs/experiment-57j69.md b/docs/experiment-57j69.md new file mode 100644 index 0000000..8bfe4e9 --- /dev/null +++ b/docs/experiment-57j69.md @@ -0,0 +1,150 @@ +# Experiment 57J.69 — `structuralActionRequired` Live Population and Contract Enforcement + +**Branch:** `feature/semantic-action-contract-v0.23` +**Starting HEAD:** `1b3bbd5` (experiment: record structural action contract implementation) +**Parent design experiment:** 57J.68 (bounded implementation complete — schema, validator, prompt, tests) + +## Objective + +Answer exactly: + +> For the fixed savings-realism uncertainty case, does the live model populate `structuralActionRequired`, and does the resulting proposal satisfy the new strict declaration-to-mutation contract? + +## Fixed scenario + +```text +We are considering relocating the engineering team to reduce operating costs. +``` + +## Fixed answer + +```text +I am unsure whether the projected office savings from the relocation are realistic. +``` + +## Hypothesis + +For this answer, the model should explicitly declare `structuralActionRequired = true` if the savings-realism uncertainty is not already fully represented in the graph. If true, the proposal must contain meaningful graph mutation. + +## Call accounting + +| Metric | Value | +|---|---| +| startCalls | 1 | +| updateCalls | 1 | +| totalCalls | 2 | + +Retries: 0 +Supplementary scripts: NO + +## START + +- **HTTP:** 200 +- **Stage:** unknown +- **Nodes:** 5 +- **Edges:** 3 +- **Selected question:** "What evidence would confirm or rule out current location, target location, team size, collaboration dependencies, and productivity impact?" +- **Relevant unresolved unknowns:** (graph contains state/uncertainty nodes only from fresh start — no pre-existing savings-realism node) + +## UPDATE 1 + +- **HTTP:** 422 +- **Stage:** proposal_compatibility +- **First error:** "structuralActionRequired is true but proposal contains no graph mutation" + "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress." + +### Answer meaning fields + +| Field | Value | +|---|---| +| userSupportedMeaning | "User is unsure whether the projected office savings from the relocation are realistic." | +| possibleInference | "If the savings projections are overestimated, the net financial benefit of relocating the engineering team may be negligible or negative." | +| supportCategory | "uncertain" | +| resolutionGuidance | null | + +### structuralActionRequired + +`true` (declared by model) + +### Proposal content (rejected snapshot) + +```json +{ + "updatedNodes": [{ "nodeId": "nqx00rq", "newValue": null }], + "resolvedUnknownNodeIds": [], + "addedNodes": [], + "addedEdges": [] +} +``` + +- **selectedQuestion:** null (rejected before question selection) + +### Meaningful mutation check + +`updatedNodes` contains only `[{nodeId: "nqx00rq", newValue: null}]` — a null assignment to an existing node. `addedNodes` and `addedEdges` are empty. `resolvedUnknownNodeIds` is empty. + +Using production `hasMeaningfulChange` semantics, this evaluates to **NO MEANINGFUL MUTATION** (the only structural change is a null set on an existing node, which does not create or alter graph topology). + +## Contract state + +| structuralActionRequired | meaningful mutation | Contract classification | +|---|---|---| +| true | absent | **CONTRACT TRUE + NO MUTATION** | + +## Structural identity + +**UNAVAILABLE** — no mutation occurred. + +## Classification: C — TRUE/NO-MUTATION CONTRADICTION + +The model declared `structuralActionRequired = true` but produced a proposal with no meaningful graph mutation. The validator correctly rejected this at `proposal_compatibility` stage (HTTP 422). + +### Why + +The configured model (`qwen-claude:latest`) recognized that the savings-realism uncertainty warranted structural action and set `structuralActionRequired = true`. However, instead of creating a dedicated unknown node for the savings-realism concern, it produced only a null-set on an existing node — structurally inert. This is the same class of proposal failure observed in Experiment 57J.61 (meaning extracted but zero mutation proposed) and Experiment 57J.63 (same rejection pattern). + +The validator's new `structuralActionRequired` contract check fired first (it appears before the legacy `userSupportedMeaning` guard in evaluation order), producing the dual rejection message: +1. "structuralActionRequired is true but proposal contains no graph mutation" — new v0.23 contract rule +2. "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation" — legacy guard + +Both errors express the same fundamental violation: model declared action needed but didn't deliver structural change. + +### Did model populate structuralActionRequired: YES + +The field was present and set to `true`. + +### Did declaration match proposal shape: NO + +`structuralActionRequired = true` contradicts the zero-mutation proposal content. + +### Did validator enforce the strict contract: YES + +The validator rejected at `proposal_compatibility` with specific dual error messages covering both the new contract rule and the legacy guard, preventing any graph mutation from being applied. + +### What this establishes: + +1. The `structuralActionRequired` field IS populated by the live model for savings-realism uncertainty. +2. The v0.23 validator ENFORCES the strict declaration-to-mutation contract — a true declaration with zero mutation is rejected. +3. The new contract rejection fires at the correct stage (`proposal_compatibility`) before any graph mutation occurs. +4. The dual-error output (new + legacy) works correctly: both guards agree on the violation. + +### What this does NOT prove: + +1. Whether `structuralActionRequired = true` is semantically correct for this answer — the validator tests contract consistency, not semantic truth of the boolean choice. +2. Whether the model could produce a correct true+mutation proposal in a subsequent retry (retries are forbidden). +3. Stability across repeated runs with this scenario/answer pair. +4. That Update 2 would proceed differently if Update 1 had succeeded. +5. Whether cold-start node count variance (5 nodes) affects the model's ability to commit to structural action. + +### What this reveals about the remaining gap: + +The model knows it should act structurally (`structuralActionRequired = true`) but fails to produce the actual graph mutation in a single attempt. This is the same prompt-enforcement gap identified in 57J.64 — the model owns the structural action decision, and when it chooses true, code rejects the no-op without providing a bounded repair path. The production-only path (no regeneration/retry) means this remains an unresolved capability gap. + +--- + +**Ollama calls beyond harness count:** 0 +**Dev server disturbed:** NO +**Production code changed:** NO +**Prompt changed during experiment:** NO +**Canonical harness restored:** YES +**57J.62 capture hardening preserved:** YES (rejectedProposalSnapshot captured correctly) +**Hardened no-retry behaviour preserved:** YES