151 lines
6.7 KiB
Markdown
151 lines
6.7 KiB
Markdown
# Experiment 57J.69 — `structuralActionRequired` Live Population and Contract Enforcement
|
|
|
|
**Branch:** `feature/semantic-action-contract-v0.23`
|
|
**Starting HEAD:** `1b3bbd5` (experiment: record structural action contract implementation)
|
|
**Parent design experiment:** 57J.68 (bounded implementation complete — schema, validator, prompt, tests)
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> For the fixed savings-realism uncertainty case, does the live model populate `structuralActionRequired`, and does the resulting proposal satisfy the new strict declaration-to-mutation contract?
|
|
|
|
## Fixed scenario
|
|
|
|
```text
|
|
We are considering relocating the engineering team to reduce operating costs.
|
|
```
|
|
|
|
## Fixed answer
|
|
|
|
```text
|
|
I am unsure whether the projected office savings from the relocation are realistic.
|
|
```
|
|
|
|
## Hypothesis
|
|
|
|
For this answer, the model should explicitly declare `structuralActionRequired = true` if the savings-realism uncertainty is not already fully represented in the graph. If true, the proposal must contain meaningful graph mutation.
|
|
|
|
## Call accounting
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| startCalls | 1 |
|
|
| updateCalls | 1 |
|
|
| totalCalls | 2 |
|
|
|
|
Retries: 0
|
|
Supplementary scripts: NO
|
|
|
|
## START
|
|
|
|
- **HTTP:** 200
|
|
- **Stage:** unknown
|
|
- **Nodes:** 5
|
|
- **Edges:** 3
|
|
- **Selected question:** "What evidence would confirm or rule out current location, target location, team size, collaboration dependencies, and productivity impact?"
|
|
- **Relevant unresolved unknowns:** (graph contains state/uncertainty nodes only from fresh start — no pre-existing savings-realism node)
|
|
|
|
## UPDATE 1
|
|
|
|
- **HTTP:** 422
|
|
- **Stage:** proposal_compatibility
|
|
- **First error:** "structuralActionRequired is true but proposal contains no graph mutation" + "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
|
|
|
### Answer meaning fields
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| userSupportedMeaning | "User is unsure whether the projected office savings from the relocation are realistic." |
|
|
| possibleInference | "If the savings projections are overestimated, the net financial benefit of relocating the engineering team may be negligible or negative." |
|
|
| supportCategory | "uncertain" |
|
|
| resolutionGuidance | null |
|
|
|
|
### structuralActionRequired
|
|
|
|
`true` (declared by model)
|
|
|
|
### Proposal content (rejected snapshot)
|
|
|
|
```json
|
|
{
|
|
"updatedNodes": [{ "nodeId": "nqx00rq", "newValue": null }],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [],
|
|
"addedEdges": []
|
|
}
|
|
```
|
|
|
|
- **selectedQuestion:** null (rejected before question selection)
|
|
|
|
### Meaningful mutation check
|
|
|
|
`updatedNodes` contains only `[{nodeId: "nqx00rq", newValue: null}]` — a null assignment to an existing node. `addedNodes` and `addedEdges` are empty. `resolvedUnknownNodeIds` is empty.
|
|
|
|
Using production `hasMeaningfulChange` semantics, this evaluates to **NO MEANINGFUL MUTATION** (the only structural change is a null set on an existing node, which does not create or alter graph topology).
|
|
|
|
## Contract state
|
|
|
|
| structuralActionRequired | meaningful mutation | Contract classification |
|
|
|---|---|---|
|
|
| true | absent | **CONTRACT TRUE + NO MUTATION** |
|
|
|
|
## Structural identity
|
|
|
|
**UNAVAILABLE** — no mutation occurred.
|
|
|
|
## Classification: C — TRUE/NO-MUTATION CONTRADICTION
|
|
|
|
The model declared `structuralActionRequired = true` but produced a proposal with no meaningful graph mutation. The validator correctly rejected this at `proposal_compatibility` stage (HTTP 422).
|
|
|
|
### Why
|
|
|
|
The configured model (`qwen-claude:latest`) recognized that the savings-realism uncertainty warranted structural action and set `structuralActionRequired = true`. However, instead of creating a dedicated unknown node for the savings-realism concern, it produced only a null-set on an existing node — structurally inert. This is the same class of proposal failure observed in Experiment 57J.61 (meaning extracted but zero mutation proposed) and Experiment 57J.63 (same rejection pattern).
|
|
|
|
The validator's new `structuralActionRequired` contract check fired first (it appears before the legacy `userSupportedMeaning` guard in evaluation order), producing the dual rejection message:
|
|
1. "structuralActionRequired is true but proposal contains no graph mutation" — new v0.23 contract rule
|
|
2. "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation" — legacy guard
|
|
|
|
Both errors express the same fundamental violation: model declared action needed but didn't deliver structural change.
|
|
|
|
### Did model populate structuralActionRequired: YES
|
|
|
|
The field was present and set to `true`.
|
|
|
|
### Did declaration match proposal shape: NO
|
|
|
|
`structuralActionRequired = true` contradicts the zero-mutation proposal content.
|
|
|
|
### Did validator enforce the strict contract: YES
|
|
|
|
The validator rejected at `proposal_compatibility` with specific dual error messages covering both the new contract rule and the legacy guard, preventing any graph mutation from being applied.
|
|
|
|
### What this establishes:
|
|
|
|
1. The `structuralActionRequired` field IS populated by the live model for savings-realism uncertainty.
|
|
2. The v0.23 validator ENFORCES the strict declaration-to-mutation contract — a true declaration with zero mutation is rejected.
|
|
3. The new contract rejection fires at the correct stage (`proposal_compatibility`) before any graph mutation occurs.
|
|
4. The dual-error output (new + legacy) works correctly: both guards agree on the violation.
|
|
|
|
### What this does NOT prove:
|
|
|
|
1. Whether `structuralActionRequired = true` is semantically correct for this answer — the validator tests contract consistency, not semantic truth of the boolean choice.
|
|
2. Whether the model could produce a correct true+mutation proposal in a subsequent retry (retries are forbidden).
|
|
3. Stability across repeated runs with this scenario/answer pair.
|
|
4. That Update 2 would proceed differently if Update 1 had succeeded.
|
|
5. Whether cold-start node count variance (5 nodes) affects the model's ability to commit to structural action.
|
|
|
|
### What this reveals about the remaining gap:
|
|
|
|
The model knows it should act structurally (`structuralActionRequired = true`) but fails to produce the actual graph mutation in a single attempt. This is the same prompt-enforcement gap identified in 57J.64 — the model owns the structural action decision, and when it chooses true, code rejects the no-op without providing a bounded repair path. The production-only path (no regeneration/retry) means this remains an unresolved capability gap.
|
|
|
|
---
|
|
|
|
**Ollama calls beyond harness count:** 0
|
|
**Dev server disturbed:** NO
|
|
**Production code changed:** NO
|
|
**Prompt changed during experiment:** NO
|
|
**Canonical harness restored:** YES
|
|
**57J.62 capture hardening preserved:** YES (rejectedProposalSnapshot captured correctly)
|
|
**Hardened no-retry behaviour preserved:** YES
|