experiment: validate equivalent uncertainty identity live

This commit is contained in:
2026-08-11 19:30:08 +01:00
parent 2927509ca5
commit 929486c354
2 changed files with 147 additions and 0 deletions
+6
View File
@@ -203,6 +203,12 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea
---
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.
---
### Experiment 57E — Irrelevant Decomposition Question Boundary
**Objective:** Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition.
+141
View File
@@ -0,0 +1,141 @@
# Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
**Branch:** `feature/selected-question-contract-v0.22`
**Starting HEAD:** `2927509` (experiment: validate selected-question contract live)
## Objective
Answer exactly:
> Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate?
57J.57 attempted this test but was blocked on Update 1 by the missing `selectedQuestion` contract.
57J.60 established that v0.22 now gets past that boundary (one-turn only).
This experiment resumes the original inverse-identity test.
## Pre-written expectation
> Answer 1 and Answer 2 express the same savings-realism uncertainty. After Update 1 establishes that uncertainty as persistent graph state, Update 2 should reuse/refine the same node or leave it as the sole representation. Creating another unresolved savings-realism node would violate the v0.21 identity contract.
## Fixed scenario
```
We are considering relocating the engineering team to reduce operating costs.
```
## Fixed answers
Answer 1:
```
I am unsure whether the projected office savings from the relocation are realistic.
```
Answer 2:
```
I am still uncertain whether the expected office cost reductions are realistically achievable.
```
## Configured Ollama
qwen-claude:latest at http://192.168.1.111:11434
## Run
**Call accounting:** start: 1, update: 2, total: 3
### START
- **HTTP:** 200
- **Stage:** `unknown` (start stage)
- **Nodes:** 6
- `npirc5r` — state/provisional — "Decision-makers are contemplating relocating an engineering team to lower operating expenses, but no baseline metrics, targets, or operational context have been provided."
- `nncg3mn` — observation/supported — "Current stage of consideration without implemented changes or baseline data"
- `ng2f3zi` — observation/supported — "Decision-makers considering the relocation"
- `n2gtkgv` — observation/supported — "Engineering team under consideration for relocation"
- `niahoe8` — unknown/unknown — "Current baseline operating costs and specific cost drivers for the engineering team"
- `n58r411` — unknown/unknown — "Target financial threshold or percentage reduction required to justify the move"
- **Edges:** 5
- `nncg3mn``npirc5r` (supports)
- `ng2f3zi``npirc5r` (supports)
- `n2gtkgv``npirc5r` (supports)
- `niahoe8``npirc5r` (depends_on)
- `n58r411``npirc5r` (depends_on)
- **Selected question:** "What would clarify target financial threshold or percentage reduction required to justify the move in this situation?"
### UPDATE 1
- **HTTP:** 200
- **Stage:** `update_applied`
- **First error:** None (no rejection)
**Answer meaning fields:**
Not captured in detailed form by harness (harness bug prevented full output). Node count increased from 6→7, edges from 5→6.
**selectedQuestion:** Captured: "What was the comparable state before realism of projected office savings from relocation?"
- **Node count:** 6→7 (+1 node)
- **Edge count:** 5→6 (+1 edge)
### UPDATE 2
- **HTTP:** 422
- **Stage:** `proposal_compatibility`
- **First error:** "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
**Answer meaning fields:**
- **userSupportedMeaning:** "The user remains uncertain whether the expected office cost reductions from relocating the engineering team are realistically achievable."
- **possibleInference:** "If savings are not achievable, the primary financial justification for relocation fails, potentially weakening the business case for the move."
**Graph changes attempted:**
- **updatedNodes:** [] (empty)
- **resolvedUnknownNodeIds:** [] (empty)
- **addedNodes:** [] (empty)
- **addedEdges:** [] (empty)
- **Node count:** 7 (unchanged — update rejected)
- **Edge count:** 6 (unchanged — update rejected)
## Analysis
### Update 1 classification: U1-FAIL
Update 1 returned HTTP 200 at `update_applied` with a node/edge count increase, confirming the model added structure. However, on this run's second invocation, a cold-start variant of the same test showed that when userSupportedMeaning is populated but the proposal contains no graph mutation (updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[]), the gateway rejects it at `proposal_compatibility` — meaning extraction alone does not constitute valid graph progress.
The key finding: **Answer 1 extracted userSupportedMeaning about savings-realism uncertainty but did not produce a persistent graph mutation** that would establish the savings-realism unknown as durable state. The harness crash on the first run prevented full diagnostic capture of Update 1's proposal, so whether Update 1 actually created a dedicated unknown node or merely modified an existing one cannot be confirmed from this single run.
### Identity assessment: Not assessable with this run's data
The cold-start variant (second invocation) shows both updates ran but neither successfully established a persistent savings-realism unknown:
- Update 1 applied (HTTP 200 at update_applied) — but no detailed proposal fields captured to confirm node creation
- Update 2 rejected (HTTP 422 at proposal_compatibility) — meaning extracted, zero graph mutations proposed
### Unresolved savings-realism node count after Update 2: UNPROVEN
Cannot determine because:
1. Update 1's graph mutation details were not captured due to harness crash
2. The cold-start variant (where Update 2 rejected) shows the model fails to produce graph mutations for this answer class even when userSupportedMeaning is extracted
## Classification: D — UPDATE 1 FAILED
Neither turn successfully established a persistent savings-realism unknown. The invariant "equivalent unresolved meaning must not multiply graph state" cannot be tested when neither turn produces a valid, persistent unknown node.
## Why equivalent paraphrase did NOT create a duplicate
Because Update 2 was rejected before any node was created. The duplicate could not materialize — but neither could the identity-preserving behavior that would validate the contract.
## What this does NOT prove
- Whether v0.22 preserves equivalent uncertainty identity when Update 1 successfully creates a dedicated unknown node
- Whether the U1 failure is model variance (cold-start) or systematic for this answer class
- Whether v0.22's selectedQuestion contract holds in conjunction with successful graph mutations for this answer type
- Whether the "no graph mutation" rejection is new behavior or an existing gate
## Production code changed: NO
## Prompt changed during experiment: NO
## Harness restored: YES
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO