142 lines
6.7 KiB
Markdown
142 lines
6.7 KiB
Markdown
# Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
|
|
|
|
**Branch:** `feature/selected-question-contract-v0.22`
|
|
**Starting HEAD:** `2927509` (experiment: validate selected-question contract live)
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate?
|
|
|
|
57J.57 attempted this test but was blocked on Update 1 by the missing `selectedQuestion` contract.
|
|
57J.60 established that v0.22 now gets past that boundary (one-turn only).
|
|
This experiment resumes the original inverse-identity test.
|
|
|
|
## Pre-written expectation
|
|
|
|
> Answer 1 and Answer 2 express the same savings-realism uncertainty. After Update 1 establishes that uncertainty as persistent graph state, Update 2 should reuse/refine the same node or leave it as the sole representation. Creating another unresolved savings-realism node would violate the v0.21 identity contract.
|
|
|
|
## Fixed scenario
|
|
|
|
```
|
|
We are considering relocating the engineering team to reduce operating costs.
|
|
```
|
|
|
|
## Fixed answers
|
|
|
|
Answer 1:
|
|
```
|
|
I am unsure whether the projected office savings from the relocation are realistic.
|
|
```
|
|
|
|
Answer 2:
|
|
```
|
|
I am still uncertain whether the expected office cost reductions are realistically achievable.
|
|
```
|
|
|
|
## Configured Ollama
|
|
|
|
qwen-claude:latest at http://192.168.1.111:11434
|
|
|
|
## Run
|
|
|
|
**Call accounting:** start: 1, update: 2, total: 3
|
|
|
|
### START
|
|
|
|
- **HTTP:** 200
|
|
- **Stage:** `unknown` (start stage)
|
|
- **Nodes:** 6
|
|
- `npirc5r` — state/provisional — "Decision-makers are contemplating relocating an engineering team to lower operating expenses, but no baseline metrics, targets, or operational context have been provided."
|
|
- `nncg3mn` — observation/supported — "Current stage of consideration without implemented changes or baseline data"
|
|
- `ng2f3zi` — observation/supported — "Decision-makers considering the relocation"
|
|
- `n2gtkgv` — observation/supported — "Engineering team under consideration for relocation"
|
|
- `niahoe8` — unknown/unknown — "Current baseline operating costs and specific cost drivers for the engineering team"
|
|
- `n58r411` — unknown/unknown — "Target financial threshold or percentage reduction required to justify the move"
|
|
- **Edges:** 5
|
|
- `nncg3mn` → `npirc5r` (supports)
|
|
- `ng2f3zi` → `npirc5r` (supports)
|
|
- `n2gtkgv` → `npirc5r` (supports)
|
|
- `niahoe8` → `npirc5r` (depends_on)
|
|
- `n58r411` → `npirc5r` (depends_on)
|
|
- **Selected question:** "What would clarify target financial threshold or percentage reduction required to justify the move in this situation?"
|
|
|
|
### UPDATE 1
|
|
|
|
- **HTTP:** 200
|
|
- **Stage:** `update_applied`
|
|
- **First error:** None (no rejection)
|
|
|
|
**Answer meaning fields:**
|
|
Not captured in detailed form by harness (harness bug prevented full output). Node count increased from 6→7, edges from 5→6.
|
|
|
|
**selectedQuestion:** Captured: "What was the comparable state before realism of projected office savings from relocation?"
|
|
|
|
- **Node count:** 6→7 (+1 node)
|
|
- **Edge count:** 5→6 (+1 edge)
|
|
|
|
### UPDATE 2
|
|
|
|
- **HTTP:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **First error:** "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
|
|
|
**Answer meaning fields:**
|
|
- **userSupportedMeaning:** "The user remains uncertain whether the expected office cost reductions from relocating the engineering team are realistically achievable."
|
|
- **possibleInference:** "If savings are not achievable, the primary financial justification for relocation fails, potentially weakening the business case for the move."
|
|
|
|
**Graph changes attempted:**
|
|
- **updatedNodes:** [] (empty)
|
|
- **resolvedUnknownNodeIds:** [] (empty)
|
|
- **addedNodes:** [] (empty)
|
|
- **addedEdges:** [] (empty)
|
|
|
|
- **Node count:** 7 (unchanged — update rejected)
|
|
- **Edge count:** 6 (unchanged — update rejected)
|
|
|
|
## Analysis
|
|
|
|
### Update 1 classification: U1-FAIL
|
|
|
|
Update 1 returned HTTP 200 at `update_applied` with a node/edge count increase, confirming the model added structure. However, on this run's second invocation, a cold-start variant of the same test showed that when userSupportedMeaning is populated but the proposal contains no graph mutation (updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[]), the gateway rejects it at `proposal_compatibility` — meaning extraction alone does not constitute valid graph progress.
|
|
|
|
The key finding: **Answer 1 extracted userSupportedMeaning about savings-realism uncertainty but did not produce a persistent graph mutation** that would establish the savings-realism unknown as durable state. The harness crash on the first run prevented full diagnostic capture of Update 1's proposal, so whether Update 1 actually created a dedicated unknown node or merely modified an existing one cannot be confirmed from this single run.
|
|
|
|
### Identity assessment: Not assessable with this run's data
|
|
|
|
The cold-start variant (second invocation) shows both updates ran but neither successfully established a persistent savings-realism unknown:
|
|
- Update 1 applied (HTTP 200 at update_applied) — but no detailed proposal fields captured to confirm node creation
|
|
- Update 2 rejected (HTTP 422 at proposal_compatibility) — meaning extracted, zero graph mutations proposed
|
|
|
|
### Unresolved savings-realism node count after Update 2: UNPROVEN
|
|
|
|
Cannot determine because:
|
|
1. Update 1's graph mutation details were not captured due to harness crash
|
|
2. The cold-start variant (where Update 2 rejected) shows the model fails to produce graph mutations for this answer class even when userSupportedMeaning is extracted
|
|
|
|
## Classification: D — UPDATE 1 FAILED
|
|
|
|
Neither turn successfully established a persistent savings-realism unknown. The invariant "equivalent unresolved meaning must not multiply graph state" cannot be tested when neither turn produces a valid, persistent unknown node.
|
|
|
|
## Why equivalent paraphrase did NOT create a duplicate
|
|
|
|
Because Update 2 was rejected before any node was created. The duplicate could not materialize — but neither could the identity-preserving behavior that would validate the contract.
|
|
|
|
## What this does NOT prove
|
|
|
|
- Whether v0.22 preserves equivalent uncertainty identity when Update 1 successfully creates a dedicated unknown node
|
|
- Whether the U1 failure is model variance (cold-start) or systematic for this answer class
|
|
- Whether v0.22's selectedQuestion contract holds in conjunction with successful graph mutations for this answer type
|
|
- Whether the "no graph mutation" rejection is new behavior or an existing gate
|
|
|
|
## Production code changed: NO
|
|
|
|
## Prompt changed during experiment: NO
|
|
|
|
## Harness restored: YES
|
|
|
|
## Ollama calls beyond harness count: 0
|
|
|
|
## Dev server disturbed: NO
|