experiment: rerun equivalent uncertainty identity with hardened capture
This commit is contained in:
@@ -221,6 +221,12 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.63 — Equivalent Uncertainty Identity Rerun with Hardened Capture
|
||||
|
||||
**Objective:** Once Update 1 establishes a persistent savings-realism uncertainty, does a semantically equivalent Answer 2 reuse that same unresolved node? **Classification: D — UPDATE 1 DID NOT ESTABLISH ANCHOR.** One start + one update. Start HTTP 200 (8 nodes). Update 1 returned HTTP 422 at `proposal_compatibility` with error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." The model extracted userSupportedMeaning ("I am unsure whether the projected office savings from the relocation are realistic.") with possibleInference but proposed zero graph mutations (addedNodes=[], addedEdges=[], updatedNodes=[{nodeId: nz4k4ep, newValue: null}]). No persistent savings-realism unknown was established. Update 2 not reached. Confirms same blocking class as 57J.61 but with full diagnostic evidence captured (57J.62 hardening works). Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
|
||||
|
||||
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Experiment 57J.63 — Equivalent Uncertainty Identity Rerun with Hardened Capture
|
||||
|
||||
**Branch:** `feature/selected-question-contract-v0.22`
|
||||
**Starting HEAD:** `47509d3` (docs: record accepted-update capture hardening)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> Once Update 1 establishes a persistent savings-realism uncertainty, does a semantically equivalent Answer 2 reuse that same unresolved node without creating duplicate graph state?
|
||||
|
||||
57J.61 was inconclusive because accepted Update 1 state was not captured reliably.
|
||||
57J.62 fixed that apparatus.
|
||||
|
||||
Do not change the reasoning fixture.
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
- **Answer 1:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **Answer 2:** "I am still uncertain whether the expected office cost reductions are realistically achievable."
|
||||
- **maxUpdates:** 2
|
||||
- **Ollama model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
> Answer 1 should establish one persistent savings-realism uncertainty. Answer 2 expresses the same unresolved resolution question and should therefore reuse/refine that existing identity or leave it as the sole representation. It must not create a second equivalent unresolved unknown.
|
||||
|
||||
## Run results
|
||||
|
||||
### Start
|
||||
|
||||
- **HTTP:** 200
|
||||
- **Stage:** unknown
|
||||
- **Selected question:** "What would clarify projected savings or minimum cost reduction threshold required to justify relocation in this situation?"
|
||||
- **Node count:** 8
|
||||
- **Edge count:** 5
|
||||
- **Relevant unresolved cost/savings unknowns:** None established by start alone
|
||||
|
||||
### Update 1
|
||||
|
||||
- **HTTP:** 422
|
||||
- **Stage:** proposal_compatibility
|
||||
- **First error:** "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
|
||||
#### RejectedProposalSnapshot
|
||||
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "I am unsure whether the projected office savings from the relocation are realistic.",
|
||||
"possibleInference": "If the savings are not realistic, the relocation may fail to meet the goal of reducing operating costs."
|
||||
},
|
||||
"updatedNodes": [
|
||||
{
|
||||
"nodeId": "nz4k4ep",
|
||||
"newValue": null
|
||||
}
|
||||
],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [],
|
||||
"addedEdges": []
|
||||
}
|
||||
```
|
||||
|
||||
- **userSupportedMeaning:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **supportCategory:** null (not populated by model)
|
||||
- **resolutionGuidance:** null (not populated by model)
|
||||
- **updatedNodes:** [{nodeId: "nz4k4ep", newValue: null}]
|
||||
- **resolvedUnknownNodeIds:** []
|
||||
- **addedNodes:** []
|
||||
- **addedEdges:** []
|
||||
- **selectedQuestion:** null
|
||||
|
||||
#### Persistent savings-realism node: NONE
|
||||
|
||||
The model extracted userSupportedMeaning but proposed zero graph mutations. The gateway rejected the proposal at proposal_compatibility. No persistent savings-realism unknown was established.
|
||||
|
||||
#### Update 1 classification: U1-NO-PERSISTENT-UNCERTAINTY
|
||||
|
||||
### Update 2
|
||||
|
||||
**Reached:** NO
|
||||
|
||||
Update 1 did not establish a persistent savings-realism anchor. Per protocol, Update 2 is not executed.
|
||||
|
||||
## Call accounting
|
||||
|
||||
- startCalls: 1
|
||||
- updateCalls: 1
|
||||
- totalCalls: 2
|
||||
|
||||
## Identity result
|
||||
|
||||
N/A — no anchor was established by Update 1.
|
||||
|
||||
## Classification: D — UPDATE 1 DID NOT ESTABLISH ANCHOR
|
||||
|
||||
The same blocking class as 57J.61. The model correctly extracts userSupportedMeaning for savings-realism uncertainty but does not propose a graph mutation (no new unknown node, no edge). The proposal_compatibility gateway rejects this with HTTP 422. Without an anchor, the identity invariant cannot be tested.
|
||||
|
||||
### What this establishes:
|
||||
- The harness captured rejected proposal detail correctly (57J.62 hardening works).
|
||||
- When the model produces userSupportedMeaning for savings-realism uncertainty without adding a dedicated unknown node, the gateway rejects at proposal_compatibility with the expected error message.
|
||||
- Same failure class as 57J.61 but with full diagnostics visible.
|
||||
|
||||
### What this does NOT prove:
|
||||
- Whether equivalent paraphrase creates duplicate state (identity invariant untestable without an anchor).
|
||||
- Whether a dedicated savings-realism unknown node can be created at all in the current production path for this answer class.
|
||||
- Stability across different answers or scenarios that do establish anchors.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed during experiment: NO
|
||||
## Canonical harness restored: YES
|
||||
## 57J.62 capture hardening preserved: YES
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user