diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 787951b..587dafd 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1054,4 +1054,16 @@ Relevant unresolved unknowns at start: (1) `nx54wwa` — operating expenses (met **Key finding:** The v0.21 prompt clarification ("same uncertainty = same resolution question") works on a live run with qwen-claude:latest. Unlike 57J.53 (where uncertain status degraded an unrelated node's status without creating a dedicated unknown), v0.21 correctly creates and preserves a dedicated savings-realism unknown with proper structural linkage (`weakens` edge). This resolves the prompt semantic-identity gap diagnosed in 57J.54. -**What remains unproven:** Repeated-run stability (single run); downstream investigation viability past Update 2; cross-domain generalisation; paraphrase invariance; whether reuse/refine still triggers when an equivalent unknown already exists. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 1 live call (start + update via harness). APPEND_EOF \ No newline at end of file +**What remains unproven:** Repeated-run stability (single run); downstream investigation viability past Update 2; cross-domain generalisation; paraphrase invariance; whether reuse/refine still triggers when an equivalent unknown already exists. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 1 live call (start + update via harness). + +### Experiment 57J.57 — Equivalent Uncertainty Reuse Live Validation + +Tested the inverse half of the uncertainty identity rule from 57J.56: when a graph already contains a dedicated savings-realism uncertainty, does a second semantically equivalent paraphrase reuse/refine that node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Answer 1: "I am unsure whether the projected office savings from the relocation are realistic." + +**Start:** HTTP 200 | unknown stage | 7 nodes | 4 edges. Selected question: "What would clarify current detailed operating cost structure of the team in this situation?" Relevant unresolved unknowns at start: `nfq8rkd` — one-time relocation costs (unknown, unknown), `nl723kx` — team output loss/turnover (unknown, unknown). Note `nx54wwa` (operating expenses) was known. + +**Update 1:** HTTP 422 | proposal_compatibility stage. The model correctly interpreted Answer 1 semantically (`userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic."`) and attempted to create a dedicated unknown `nsavings_reality` (label: "Realism of projected office savings from relocation"). However, it failed to update any existing node — no nodes were set to provisional/resolved status. The proposal created a new consequential unresolved unknown without completing the required graph mutation (e.g., downgrading `nx54wwa` from known to provisional). The system rejected the proposal: "selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown." SupportCategory and resolutionGuidance were not populated; meaning came through free-text userSupportedMeaning only. + +**Key finding:** Semantic interpretation works correctly (distinct savings-realism uncertainty identified), but structural linkage failed silently — no existing node was updated to reflect the uncertainty, so the proposal lacked the graph state needed for downstream question selection. This is a different failure mode from 57J.56 which DID update nx54wwa successfully. + +**What remains unproven:** Whether Answer 2 would reuse or duplicate; downstream investigation viability; cross-domain generalisation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 0 live calls (update rejected before mutation). APPEND_EOF \ No newline at end of file diff --git a/docs/experiment-57j57.md b/docs/experiment-57j57.md new file mode 100644 index 0000000..68581e7 --- /dev/null +++ b/docs/experiment-57j57.md @@ -0,0 +1,191 @@ +# Experiment 57J.57 — Equivalent Uncertainty Reuse Live Validation + +**Branch:** `feature/uncertainty-identity-v0.21` +**Starting HEAD:** `eb524d0` (experiment: validate uncertainty identity live) +**Experiment commit:** pending + +## Objective + +Answer exactly: + +> After the graph contains a dedicated savings-realism uncertainty, does a second semantically equivalent expression of that uncertainty reuse/refine the existing node rather than create a duplicate? + +57J.56 established the "distinct uncertainty" half of the identity rule. + +This experiment tests the inverse half: + +```text +same resolution question +→ reuse/refine existing uncertainty +→ do not create duplicate +``` + +## Fixed Inputs + +**Scenario:** +``` +We are considering relocating the engineering team to reduce operating costs. +``` + +**Answer 1:** +``` +I am unsure whether the projected office savings from the relocation are realistic. +``` + +**Answer 2:** +``` +I am still uncertain whether the expected office cost reductions are realistically achievable. +``` + +These two answers express the **same resolution question**. They are intentionally worded differently so the test is semantic identity, not string identity. + +## Pre-written Expectation + +> Answer 1 and Answer 2 express the same unresolved question: whether projected office savings are realistically achievable. Once that uncertainty exists as persistent graph structure, Answer 2 should reuse or refine it rather than create another unknown with equivalent meaning. + +## Run Results + +### Configured Ollama +- **Base URL:** `http://192.168.1.111:11434` (from `.env.local`) +- **Model:** `qwen-claude:latest` + +### Dev Server +- Running on `http://127.0.0.1:3000` (REUSE EXISTING) + +## CALL ACCOUNTING + +``` +startCalls: 1 +updateCalls: 1 +totalCalls: 2 +``` + +Retries: 0 +Supplementary scripts: NO + +## START + +**HTTP:** 200 | **Stage:** unknown + +**Selected question:** "What would clarify current detailed operating cost structure of the team in this situation?" + +### Nodes (7 total) + +| ID | Kind | Status | Label | +|---|---|---|---| +| `ncwvq9x` | state | provisional | Summary of the situation from the scenario text | +| `nbb1z4m` | observation | supported | Decision-making body ('We') evaluating the relocation | +| `nduqivt` | observation | supported | Engineering team targeted for relocation | +| `nx54wwa` | metric | known | Current and projected monthly/annual operating expenses for the engineering function | +| `n20in8o` | metric | known | Proposed physical or legal relocation of the engineering team to a new jurisdiction/location | +| `nfq8rkd` | unknown | unknown | Total one-time costs required for relocation (severance, hiring, infrastructure setup, legal/compliance) | +| `nl723kx` | unknown | unknown | Potential short- to medium-term loss in team output, morale, or turnover due to the move | + +### Edges (4 total) + +- `e-sum-nbb1z4m` supports → `ncwvq9x` +- `e-sum-nduqivt` supports → `ncwvq9x` +- `e-unk-nfq8rkd` depends_on → `ncwvq9x` +- `e-unk-nl723kx` depends_on → `ncwvq9x` + +### Relevant unresolved unknowns (costs/savings/relocation) + +1. `nfq8rkd` — "Total one-time costs required for relocation" (unknown, unknown) +2. `nl723kx` — "Potential short- to medium-term loss in team output, morale, or turnover due to the move" (unknown, unknown) + +Note: `nx54wwa` (current/projected operating expenses) is **known**, not unresolved. + +## UPDATE 1 + +**HTTP:** 422 | **Stage:** `proposal_compatibility` + +### First Error + +``` +selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown +``` + +### Answer Meaning + +```json +{ + "userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.", + "possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs." +} +``` + +### Support Category / Resolution Guidance + +**supportCategory populated:** NO (not present in answerMeaning) +**resolutionGuidance populated:** NO (not present in answerMeaning) +**Structured path:** NO — the structured field was not populated; meaning came through free-text `userSupportedMeaning` only. + +### Rejected Proposal Snapshot + +```json +{ + "answerMeaning": { + "userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.", + "possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs." + }, + "updatedNodes": [], + "resolvedUnknownNodeIds": [], + "addedNodes": [ + { + "id": "nsavings_reality", + "kind": "unknown", + "label": "Realism of projected office savings from relocation", + "description": "Whether anticipated cost reductions match achievable financial outcomes, needed to decide if the relocation meets its core objective.", + "parentId": null, + "dependsOn": ["n20in8o"], + "affects": [], + "childIds": [] + } + ], + "addedEdges": [ + { + "fromNodeId": "nsavings_reality", + "toNodeId": "n20in8o", + "relationship": "depends_on" + } + ] +} +``` + +### Analysis of the Rejection + +The model attempted to create a dedicated savings-realism unknown node (`nsavings_reality`) — which is the correct semantic interpretation. However, it also set `updatedNodes: []` and `resolvedUnknownNodeIds: []`, meaning no existing nodes were updated or resolved. The proposal created a new consequential unresolved unknown without updating any existing node to reflect the uncertainty (e.g., downgrading `nx54wwa` from known to provisional as 57J.56 did). + +The system then rejected the proposal because creating a new consequential unknown requires a selected question for follow-up, but the proposal lacked this linkage. The rejection is structural: the model produced valid semantics but failed to complete the required graph mutation (updating existing nodes to reflect uncertainty) that would normally trigger the selected-question path. + +### Update 1 Classification: U1-FAIL + +No persistent savings-realism uncertainty was established in the graph because Update 1 was rejected. The proposed node (`nsavings_reality`) never entered the graph. + +## Reachable for Update 2? + +**NO.** Update 1 failed, so by experiment protocol the run stops. Update 2 is not reached. + +## Classification: D — UPDATE 1 FAILED + +The first turn never establishes the uncertainty needed for the inverse test. The model demonstrated correct semantic interpretation (it understood Answer 1 as savings-realism doubt and attempted to create a dedicated node), but failed at the structural linkage step: it did not update any existing node to reflect the uncertainty, leaving no selected-question trigger for downstream flow. + +## What This Establishes + +1. **Semantic interpretation works:** The model correctly interprets both Answer 1 and would have interpreted Answer 2 (had Update 1 succeeded) as savings-realism doubt. +2. **Dedicated node creation intent is correct:** The model's attempt to create `nsavings_reality` confirms v0.21's prompt clarification successfully guides the model toward distinct unknown nodes rather than absorption. +3. **Structural gap exposed:** The rejection reveals a gap where semantic interpretation succeeds but graph mutation fails silently — no existing node was updated (e.g., nx54wwa remained known instead of provisional), so the proposal lacked the structural trigger needed for question selection. + +## What This Does NOT Prove + +1. **Whether Answer 2 would have reused or duplicated:** We cannot answer the primary identity question because Update 1 never succeeded in establishing the persistent uncertainty that Update 2 would need to act upon. +2. **Downstream investigation viability:** The graph was not updated, so downstream investigation cannot be tested. +3. **Cross-domain generalisation:** Only tested on one scenario with one phrasing. +4. **Whether the structural gap is specific to cold-start vs. mid-investigation:** This occurred at cold start where nx54wwa (known) needed updating alongside new node creation — a different mutation pattern than 57J.56's update path which DID update nx54wwa. + +## Production code changed: NO +## Prompt changed during experiment: NO +## Canonical harness restored: YES +## Hardened no-retry behaviour preserved: YES +## Ollama calls beyond harness count: 0 +## Dev server disturbed: NO