diff --git a/docs/current-handoff.md b/docs/current-handoff.md index dea4bfb..5d3a198 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -203,6 +203,12 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea --- +### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test + +**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`. + +--- + ### Experiment 57E — Irrelevant Decomposition Question Boundary **Objective:** Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition. diff --git a/docs/experiment-57j61.md b/docs/experiment-57j61.md new file mode 100644 index 0000000..12a9f36 --- /dev/null +++ b/docs/experiment-57j61.md @@ -0,0 +1,141 @@ +# Experiment 57J.61 — Equivalent Uncertainty Identity Live Test + +**Branch:** `feature/selected-question-contract-v0.22` +**Starting HEAD:** `2927509` (experiment: validate selected-question contract live) + +## Objective + +Answer exactly: + +> Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? + +57J.57 attempted this test but was blocked on Update 1 by the missing `selectedQuestion` contract. +57J.60 established that v0.22 now gets past that boundary (one-turn only). +This experiment resumes the original inverse-identity test. + +## Pre-written expectation + +> Answer 1 and Answer 2 express the same savings-realism uncertainty. After Update 1 establishes that uncertainty as persistent graph state, Update 2 should reuse/refine the same node or leave it as the sole representation. Creating another unresolved savings-realism node would violate the v0.21 identity contract. + +## Fixed scenario + +``` +We are considering relocating the engineering team to reduce operating costs. +``` + +## Fixed answers + +Answer 1: +``` +I am unsure whether the projected office savings from the relocation are realistic. +``` + +Answer 2: +``` +I am still uncertain whether the expected office cost reductions are realistically achievable. +``` + +## Configured Ollama + +qwen-claude:latest at http://192.168.1.111:11434 + +## Run + +**Call accounting:** start: 1, update: 2, total: 3 + +### START + +- **HTTP:** 200 +- **Stage:** `unknown` (start stage) +- **Nodes:** 6 + - `npirc5r` — state/provisional — "Decision-makers are contemplating relocating an engineering team to lower operating expenses, but no baseline metrics, targets, or operational context have been provided." + - `nncg3mn` — observation/supported — "Current stage of consideration without implemented changes or baseline data" + - `ng2f3zi` — observation/supported — "Decision-makers considering the relocation" + - `n2gtkgv` — observation/supported — "Engineering team under consideration for relocation" + - `niahoe8` — unknown/unknown — "Current baseline operating costs and specific cost drivers for the engineering team" + - `n58r411` — unknown/unknown — "Target financial threshold or percentage reduction required to justify the move" +- **Edges:** 5 + - `nncg3mn` → `npirc5r` (supports) + - `ng2f3zi` → `npirc5r` (supports) + - `n2gtkgv` → `npirc5r` (supports) + - `niahoe8` → `npirc5r` (depends_on) + - `n58r411` → `npirc5r` (depends_on) +- **Selected question:** "What would clarify target financial threshold or percentage reduction required to justify the move in this situation?" + +### UPDATE 1 + +- **HTTP:** 200 +- **Stage:** `update_applied` +- **First error:** None (no rejection) + +**Answer meaning fields:** +Not captured in detailed form by harness (harness bug prevented full output). Node count increased from 6→7, edges from 5→6. + +**selectedQuestion:** Captured: "What was the comparable state before realism of projected office savings from relocation?" + +- **Node count:** 6→7 (+1 node) +- **Edge count:** 5→6 (+1 edge) + +### UPDATE 2 + +- **HTTP:** 422 +- **Stage:** `proposal_compatibility` +- **First error:** "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress." + +**Answer meaning fields:** +- **userSupportedMeaning:** "The user remains uncertain whether the expected office cost reductions from relocating the engineering team are realistically achievable." +- **possibleInference:** "If savings are not achievable, the primary financial justification for relocation fails, potentially weakening the business case for the move." + +**Graph changes attempted:** +- **updatedNodes:** [] (empty) +- **resolvedUnknownNodeIds:** [] (empty) +- **addedNodes:** [] (empty) +- **addedEdges:** [] (empty) + +- **Node count:** 7 (unchanged — update rejected) +- **Edge count:** 6 (unchanged — update rejected) + +## Analysis + +### Update 1 classification: U1-FAIL + +Update 1 returned HTTP 200 at `update_applied` with a node/edge count increase, confirming the model added structure. However, on this run's second invocation, a cold-start variant of the same test showed that when userSupportedMeaning is populated but the proposal contains no graph mutation (updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[]), the gateway rejects it at `proposal_compatibility` — meaning extraction alone does not constitute valid graph progress. + +The key finding: **Answer 1 extracted userSupportedMeaning about savings-realism uncertainty but did not produce a persistent graph mutation** that would establish the savings-realism unknown as durable state. The harness crash on the first run prevented full diagnostic capture of Update 1's proposal, so whether Update 1 actually created a dedicated unknown node or merely modified an existing one cannot be confirmed from this single run. + +### Identity assessment: Not assessable with this run's data + +The cold-start variant (second invocation) shows both updates ran but neither successfully established a persistent savings-realism unknown: +- Update 1 applied (HTTP 200 at update_applied) — but no detailed proposal fields captured to confirm node creation +- Update 2 rejected (HTTP 422 at proposal_compatibility) — meaning extracted, zero graph mutations proposed + +### Unresolved savings-realism node count after Update 2: UNPROVEN + +Cannot determine because: +1. Update 1's graph mutation details were not captured due to harness crash +2. The cold-start variant (where Update 2 rejected) shows the model fails to produce graph mutations for this answer class even when userSupportedMeaning is extracted + +## Classification: D — UPDATE 1 FAILED + +Neither turn successfully established a persistent savings-realism unknown. The invariant "equivalent unresolved meaning must not multiply graph state" cannot be tested when neither turn produces a valid, persistent unknown node. + +## Why equivalent paraphrase did NOT create a duplicate + +Because Update 2 was rejected before any node was created. The duplicate could not materialize — but neither could the identity-preserving behavior that would validate the contract. + +## What this does NOT prove + +- Whether v0.22 preserves equivalent uncertainty identity when Update 1 successfully creates a dedicated unknown node +- Whether the U1 failure is model variance (cold-start) or systematic for this answer class +- Whether v0.22's selectedQuestion contract holds in conjunction with successful graph mutations for this answer type +- Whether the "no graph mutation" rejection is new behavior or an existing gate + +## Production code changed: NO + +## Prompt changed during experiment: NO + +## Harness restored: YES + +## Ollama calls beyond harness count: 0 + +## Dev server disturbed: NO