experiment: validate structural action contract live
This commit is contained in:
@@ -233,6 +233,12 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.69 — structuralActionRequired Live Population and Contract Enforcement
|
||||
|
||||
**Classification: C — TRUE/NO-MUTATION CONTRADICTION.** One start + one update (fixed scenario: "We are considering relocating the engineering team to reduce operating costs." / fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."). Start HTTP 200 (5 nodes, 3 edges). Update 1 returned HTTP 422 at `proposal_compatibility`. Model populated `structuralActionRequired = true` and extracted `userSupportedMeaning` ("User is unsure...") with supportCategory="uncertain". Proposal contained only `[updatedNodes: [{nodeId: nqx00rq, newValue: null}]]` — no added nodes, no added edges, meaningfully zero mutation. Validator rejected with dual errors: new contract rule ("structuralActionRequired is true but proposal contains no graph mutation") + legacy guard ("answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation"). **What this establishes:** (1) `structuralActionRequired` IS populated for savings-realism uncertainty; (2) v0.23 validator ENFORCES the strict declaration-to-mutation contract; (3) rejection fires at correct stage before any mutation. **What remains unproven:** whether model can produce true+mutation in a single attempt, stability across runs, prompt enforcement adequacy. The remaining gap: model knows it should act structurally but fails to produce the mutation itself — same prompt-enforcement gap from 57J.64. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j69.md`.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
|
||||
|
||||
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# Experiment 57J.69 — `structuralActionRequired` Live Population and Contract Enforcement
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `1b3bbd5` (experiment: record structural action contract implementation)
|
||||
**Parent design experiment:** 57J.68 (bounded implementation complete — schema, validator, prompt, tests)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> For the fixed savings-realism uncertainty case, does the live model populate `structuralActionRequired`, and does the resulting proposal satisfy the new strict declaration-to-mutation contract?
|
||||
|
||||
## Fixed scenario
|
||||
|
||||
```text
|
||||
We are considering relocating the engineering team to reduce operating costs.
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```text
|
||||
I am unsure whether the projected office savings from the relocation are realistic.
|
||||
```
|
||||
|
||||
## Hypothesis
|
||||
|
||||
For this answer, the model should explicitly declare `structuralActionRequired = true` if the savings-realism uncertainty is not already fully represented in the graph. If true, the proposal must contain meaningful graph mutation.
|
||||
|
||||
## Call accounting
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| startCalls | 1 |
|
||||
| updateCalls | 1 |
|
||||
| totalCalls | 2 |
|
||||
|
||||
Retries: 0
|
||||
Supplementary scripts: NO
|
||||
|
||||
## START
|
||||
|
||||
- **HTTP:** 200
|
||||
- **Stage:** unknown
|
||||
- **Nodes:** 5
|
||||
- **Edges:** 3
|
||||
- **Selected question:** "What evidence would confirm or rule out current location, target location, team size, collaboration dependencies, and productivity impact?"
|
||||
- **Relevant unresolved unknowns:** (graph contains state/uncertainty nodes only from fresh start — no pre-existing savings-realism node)
|
||||
|
||||
## UPDATE 1
|
||||
|
||||
- **HTTP:** 422
|
||||
- **Stage:** proposal_compatibility
|
||||
- **First error:** "structuralActionRequired is true but proposal contains no graph mutation" + "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
|
||||
### Answer meaning fields
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| userSupportedMeaning | "User is unsure whether the projected office savings from the relocation are realistic." |
|
||||
| possibleInference | "If the savings projections are overestimated, the net financial benefit of relocating the engineering team may be negligible or negative." |
|
||||
| supportCategory | "uncertain" |
|
||||
| resolutionGuidance | null |
|
||||
|
||||
### structuralActionRequired
|
||||
|
||||
`true` (declared by model)
|
||||
|
||||
### Proposal content (rejected snapshot)
|
||||
|
||||
```json
|
||||
{
|
||||
"updatedNodes": [{ "nodeId": "nqx00rq", "newValue": null }],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [],
|
||||
"addedEdges": []
|
||||
}
|
||||
```
|
||||
|
||||
- **selectedQuestion:** null (rejected before question selection)
|
||||
|
||||
### Meaningful mutation check
|
||||
|
||||
`updatedNodes` contains only `[{nodeId: "nqx00rq", newValue: null}]` — a null assignment to an existing node. `addedNodes` and `addedEdges` are empty. `resolvedUnknownNodeIds` is empty.
|
||||
|
||||
Using production `hasMeaningfulChange` semantics, this evaluates to **NO MEANINGFUL MUTATION** (the only structural change is a null set on an existing node, which does not create or alter graph topology).
|
||||
|
||||
## Contract state
|
||||
|
||||
| structuralActionRequired | meaningful mutation | Contract classification |
|
||||
|---|---|---|
|
||||
| true | absent | **CONTRACT TRUE + NO MUTATION** |
|
||||
|
||||
## Structural identity
|
||||
|
||||
**UNAVAILABLE** — no mutation occurred.
|
||||
|
||||
## Classification: C — TRUE/NO-MUTATION CONTRADICTION
|
||||
|
||||
The model declared `structuralActionRequired = true` but produced a proposal with no meaningful graph mutation. The validator correctly rejected this at `proposal_compatibility` stage (HTTP 422).
|
||||
|
||||
### Why
|
||||
|
||||
The configured model (`qwen-claude:latest`) recognized that the savings-realism uncertainty warranted structural action and set `structuralActionRequired = true`. However, instead of creating a dedicated unknown node for the savings-realism concern, it produced only a null-set on an existing node — structurally inert. This is the same class of proposal failure observed in Experiment 57J.61 (meaning extracted but zero mutation proposed) and Experiment 57J.63 (same rejection pattern).
|
||||
|
||||
The validator's new `structuralActionRequired` contract check fired first (it appears before the legacy `userSupportedMeaning` guard in evaluation order), producing the dual rejection message:
|
||||
1. "structuralActionRequired is true but proposal contains no graph mutation" — new v0.23 contract rule
|
||||
2. "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation" — legacy guard
|
||||
|
||||
Both errors express the same fundamental violation: model declared action needed but didn't deliver structural change.
|
||||
|
||||
### Did model populate structuralActionRequired: YES
|
||||
|
||||
The field was present and set to `true`.
|
||||
|
||||
### Did declaration match proposal shape: NO
|
||||
|
||||
`structuralActionRequired = true` contradicts the zero-mutation proposal content.
|
||||
|
||||
### Did validator enforce the strict contract: YES
|
||||
|
||||
The validator rejected at `proposal_compatibility` with specific dual error messages covering both the new contract rule and the legacy guard, preventing any graph mutation from being applied.
|
||||
|
||||
### What this establishes:
|
||||
|
||||
1. The `structuralActionRequired` field IS populated by the live model for savings-realism uncertainty.
|
||||
2. The v0.23 validator ENFORCES the strict declaration-to-mutation contract — a true declaration with zero mutation is rejected.
|
||||
3. The new contract rejection fires at the correct stage (`proposal_compatibility`) before any graph mutation occurs.
|
||||
4. The dual-error output (new + legacy) works correctly: both guards agree on the violation.
|
||||
|
||||
### What this does NOT prove:
|
||||
|
||||
1. Whether `structuralActionRequired = true` is semantically correct for this answer — the validator tests contract consistency, not semantic truth of the boolean choice.
|
||||
2. Whether the model could produce a correct true+mutation proposal in a subsequent retry (retries are forbidden).
|
||||
3. Stability across repeated runs with this scenario/answer pair.
|
||||
4. That Update 2 would proceed differently if Update 1 had succeeded.
|
||||
5. Whether cold-start node count variance (5 nodes) affects the model's ability to commit to structural action.
|
||||
|
||||
### What this reveals about the remaining gap:
|
||||
|
||||
The model knows it should act structurally (`structuralActionRequired = true`) but fails to produce the actual graph mutation in a single attempt. This is the same prompt-enforcement gap identified in 57J.64 — the model owns the structural action decision, and when it chooses true, code rejects the no-op without providing a bounded repair path. The production-only path (no regeneration/retry) means this remains an unresolved capability gap.
|
||||
|
||||
---
|
||||
|
||||
**Ollama calls beyond harness count:** 0
|
||||
**Dev server disturbed:** NO
|
||||
**Production code changed:** NO
|
||||
**Prompt changed during experiment:** NO
|
||||
**Canonical harness restored:** YES
|
||||
**57J.62 capture hardening preserved:** YES (rejectedProposalSnapshot captured correctly)
|
||||
**Hardened no-retry behaviour preserved:** YES
|
||||
Reference in New Issue
Block a user