experiment: validate controlled structural no-op live

This commit is contained in:
2026-08-12 09:35:05 +01:00
parent 8184e050c8
commit 85fb2b4256
2 changed files with 109 additions and 0 deletions
+8
View File
@@ -1441,3 +1441,11 @@ New branch: `feature/semantic-action-contract-v0.23`
**What this fixes:** The self-defeating constraint from 57J.73 where Answer 2 was unreachable because Update 1 failed at proposal_compatibility when declaring `structuralActionRequired=true` with zero mutation. Now the harness can inject a pre-anchored graph as the Update input, enabling direct testing of whether the live model produces `false + no meaningful mutation → accepted` on a graph that already contains the savings-realism anchor.
**What remains unproven:** Whether the live model will actually produce `false + no meaningful mutation` for Answer 2 when given this pre-anchored graph (a separate live experiment is required).
### Experiment 57J.75 — Pre-Anchored No-Op Update Live Test
**Objective:** Run a single controlled live test: inject the pre-anchored savings-realism fixture directly as the Update input with an answer preserving existing uncertainty, and verify the model produces `false + zero mutation → accepted`. **Classification: A — INTENTIONAL NO-OP WORKS.**
**Answer:** The one controlled run shows that with an equivalent savings-realism uncertainty already present, the model directly emitted `structuralActionRequired = false`, produced zero meaningful mutation (no added/updated nodes or edges), and the production validator accepted the proposal. All four required conditions met: structuralActionRequired=false observed directly, zero meaningful mutation, HTTP 200 / update_applied, exactly one equivalent savings-realism uncertainty remained after Update.
**What remains unproven:** Whether `false + zero mutation` would also be accepted for non-anchored graphs; cold-start comparison via normal Start→Update; stability across repeated runs (only one run); generalization beyond savings-realism anchors.
+101
View File
@@ -0,0 +1,101 @@
# Experiment 57J.75 — Pre-Anchored No-Op Update Live Test
**Branch:** `feature/semantic-action-contract-v0.23`
**Starting HEAD:** `8184e05` (docs: record pre-anchored update apparatus)
**Experiment commit:** pending (`experiment: validate controlled structural no-op live`)
## Objective
Answer exactly:
> Given a deterministic pre-existing graph fixture with a savings-realism anchor already present, does the live model produce `structuralActionRequired = false` + zero meaningful mutation when updating with an answer that preserves the existing uncertainty, and is that accepted?
This is the direct follow-up to **57J.73** (Classification G), where the `false + no meaningful mutation → accepted` branch was unreachable because Update 1 failed at `proposal_compatibility`. The apparatus from **57J.74** enables this test via a pre-anchored fixture injected directly into the Update request's `situationGraph`, bypassing Start entirely.
## Hypothesis
If the model receives a pre-anchored graph where the savings-realism unknown already exists, and updates with an answer that preserves (rather than resolves) that uncertainty, it will:
1. Declare `structuralActionRequired = false` (no new structure needed)
2. Produce zero meaningful mutation (graph unchanged)
3. Be **accepted** (not rejected by the contract validator, because the v0.23 contract validates `false + zero mutation` as a valid intentional no-op)
## Tooling
The pre-anchored apparatus from **57J.74** consists of:
1. JSON fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` — deterministic graph with one unresolved savings-realism unknown
2. Harness tests: `scripts/reproduce-multi-turn-investigation.harness.test.js` — 10 deterministic tests covering anchor count, schema validity, relationship integrity, exact graph injection
The live test sends the fixture's graph directly as the Update request's `situationGraph`, with an answer that preserves (not resolves) the existing uncertainty.
## Test Config
- **Fixture:** Pre-existing graph with savings-realism unknown (`n_savings_realism`, kind=unknown, status=unknown)
- **Answer:** "I am still unsure whether the projected office savings from the relocation are realistic."
- **Model:** qwen-claude:latest at http://192.168.1.111:11434
- **Previous Question:** "Are the projected office savings from relocation realistic?" (from fixture)
## Results
**HTTP status: 200 — update_applied**
```json
{
"success": true,
"stage": "update_applied",
"proposal": {
"structuralActionRequired": false,
"answerMeaning": {
"userSupportedMeaning": "The user remains unsure whether the projected office savings from relocation are realistic.",
"possibleInference": "Proceeding with relocation without validated savings projections carries unquantified financial risk.",
"supportCategory": "uncertain",
"resolutionGuidance": "may_resolve"
},
"addedNodes": [],
"updatedNodes": [],
"addedEdges": [],
"resolvedUnknownNodeIds": []
}
}
```
**Graph after Update:** Unchanged — still exactly 2 nodes (1 state, 1 unknown) and 1 edge. The savings-realism anchor persisted without modification.
**Next Question generated:** "What would clarify are the projected office savings from relocation realistic in this situation?"
## Evidence Summary
This one controlled run shows that, with an equivalent savings-realism uncertainty already present in the pre-anchored graph, the model directly emitted `structuralActionRequired = false`, produced zero meaningful mutation (no added/updated nodes or edges), and the production validator accepted the proposal.
## Classification: A — INTENTIONAL NO-OP WORKS
All four conditions met:
1. `structuralActionRequired = false` observed directly in the Update response
2. Zero meaningful mutation (addedNodes=[], updatedNodes=[], addedEdges=[])
3. HTTP 200 / update_applied
4. Exactly one equivalent savings-realism uncertainty remained after Update
## What this establishes
This controlled run demonstrates that when the model receives a pre-anchored graph with an existing savings-realism unknown and produces an answer preserving that uncertainty, it correctly declares no structural action needed and is accepted by the v0.23 contract validator. The `false + zero mutation → accepted` path does exist in the contract for pre-anchored inputs.
## What this does NOT prove
1. Whether `false + zero mutation` would also be accepted for non-anchored graphs (where the model might legitimately need to create structure)
2. Whether the same answer would produce different results starting from a clean Start (i.e., whether cold-start dynamics change the outcome)
3. Stability across repeated runs — only one test run was performed
4. Whether this generalizes to other types of anchors beyond savings-realism
## What remains unproven
1. **Cold-start comparison:** Run the same answer through normal Start→Update to compare whether the starting state changes the model's structural-action judgment.
2. **Different anchors:** Test with different pre-anchored graphs (e.g., risk-constraint anchor, timeline anchor) to verify generalization.
## Production code changed: NO
No production logic changed. Only deterministic harness tests from 57J.74 were used as the test apparatus.
## Harness restored: YES
Scenario, answers, and maxUpdates in `scripts/reproduce-multi-turn-investigation.mjs` remain at canonical defaults. No fixture mode was committed to the harness.
## Dev server disturbed: NO