experiment: validate equivalent uncertainty reuse live

This commit is contained in:
2026-08-11 18:29:02 +01:00
parent eb524d08e1
commit f25b1f550e
2 changed files with 204 additions and 1 deletions
+13 -1
View File
@@ -1054,4 +1054,16 @@ Relevant unresolved unknowns at start: (1) `nx54wwa` — operating expenses (met
**Key finding:** The v0.21 prompt clarification ("same uncertainty = same resolution question") works on a live run with qwen-claude:latest. Unlike 57J.53 (where uncertain status degraded an unrelated node's status without creating a dedicated unknown), v0.21 correctly creates and preserves a dedicated savings-realism unknown with proper structural linkage (`weakens` edge). This resolves the prompt semantic-identity gap diagnosed in 57J.54. **Key finding:** The v0.21 prompt clarification ("same uncertainty = same resolution question") works on a live run with qwen-claude:latest. Unlike 57J.53 (where uncertain status degraded an unrelated node's status without creating a dedicated unknown), v0.21 correctly creates and preserves a dedicated savings-realism unknown with proper structural linkage (`weakens` edge). This resolves the prompt semantic-identity gap diagnosed in 57J.54.
**What remains unproven:** Repeated-run stability (single run); downstream investigation viability past Update 2; cross-domain generalisation; paraphrase invariance; whether reuse/refine still triggers when an equivalent unknown already exists. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 1 live call (start + update via harness). APPEND_EOF **What remains unproven:** Repeated-run stability (single run); downstream investigation viability past Update 2; cross-domain generalisation; paraphrase invariance; whether reuse/refine still triggers when an equivalent unknown already exists. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 1 live call (start + update via harness).
### Experiment 57J.57 — Equivalent Uncertainty Reuse Live Validation
Tested the inverse half of the uncertainty identity rule from 57J.56: when a graph already contains a dedicated savings-realism uncertainty, does a second semantically equivalent paraphrase reuse/refine that node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Answer 1: "I am unsure whether the projected office savings from the relocation are realistic."
**Start:** HTTP 200 | unknown stage | 7 nodes | 4 edges. Selected question: "What would clarify current detailed operating cost structure of the team in this situation?" Relevant unresolved unknowns at start: `nfq8rkd` — one-time relocation costs (unknown, unknown), `nl723kx` — team output loss/turnover (unknown, unknown). Note `nx54wwa` (operating expenses) was known.
**Update 1:** HTTP 422 | proposal_compatibility stage. The model correctly interpreted Answer 1 semantically (`userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic."`) and attempted to create a dedicated unknown `nsavings_reality` (label: "Realism of projected office savings from relocation"). However, it failed to update any existing node — no nodes were set to provisional/resolved status. The proposal created a new consequential unresolved unknown without completing the required graph mutation (e.g., downgrading `nx54wwa` from known to provisional). The system rejected the proposal: "selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown." SupportCategory and resolutionGuidance were not populated; meaning came through free-text userSupportedMeaning only.
**Key finding:** Semantic interpretation works correctly (distinct savings-realism uncertainty identified), but structural linkage failed silently — no existing node was updated to reflect the uncertainty, so the proposal lacked the graph state needed for downstream question selection. This is a different failure mode from 57J.56 which DID update nx54wwa successfully.
**What remains unproven:** Whether Answer 2 would reuse or duplicate; downstream investigation viability; cross-domain generalisation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 0 live calls (update rejected before mutation). APPEND_EOF
+191
View File
@@ -0,0 +1,191 @@
# Experiment 57J.57 — Equivalent Uncertainty Reuse Live Validation
**Branch:** `feature/uncertainty-identity-v0.21`
**Starting HEAD:** `eb524d0` (experiment: validate uncertainty identity live)
**Experiment commit:** pending
## Objective
Answer exactly:
> After the graph contains a dedicated savings-realism uncertainty, does a second semantically equivalent expression of that uncertainty reuse/refine the existing node rather than create a duplicate?
57J.56 established the "distinct uncertainty" half of the identity rule.
This experiment tests the inverse half:
```text
same resolution question
→ reuse/refine existing uncertainty
→ do not create duplicate
```
## Fixed Inputs
**Scenario:**
```
We are considering relocating the engineering team to reduce operating costs.
```
**Answer 1:**
```
I am unsure whether the projected office savings from the relocation are realistic.
```
**Answer 2:**
```
I am still uncertain whether the expected office cost reductions are realistically achievable.
```
These two answers express the **same resolution question**. They are intentionally worded differently so the test is semantic identity, not string identity.
## Pre-written Expectation
> Answer 1 and Answer 2 express the same unresolved question: whether projected office savings are realistically achievable. Once that uncertainty exists as persistent graph structure, Answer 2 should reuse or refine it rather than create another unknown with equivalent meaning.
## Run Results
### Configured Ollama
- **Base URL:** `http://192.168.1.111:11434` (from `.env.local`)
- **Model:** `qwen-claude:latest`
### Dev Server
- Running on `http://127.0.0.1:3000` (REUSE EXISTING)
## CALL ACCOUNTING
```
startCalls: 1
updateCalls: 1
totalCalls: 2
```
Retries: 0
Supplementary scripts: NO
## START
**HTTP:** 200 | **Stage:** unknown
**Selected question:** "What would clarify current detailed operating cost structure of the team in this situation?"
### Nodes (7 total)
| ID | Kind | Status | Label |
|---|---|---|---|
| `ncwvq9x` | state | provisional | Summary of the situation from the scenario text |
| `nbb1z4m` | observation | supported | Decision-making body ('We') evaluating the relocation |
| `nduqivt` | observation | supported | Engineering team targeted for relocation |
| `nx54wwa` | metric | known | Current and projected monthly/annual operating expenses for the engineering function |
| `n20in8o` | metric | known | Proposed physical or legal relocation of the engineering team to a new jurisdiction/location |
| `nfq8rkd` | unknown | unknown | Total one-time costs required for relocation (severance, hiring, infrastructure setup, legal/compliance) |
| `nl723kx` | unknown | unknown | Potential short- to medium-term loss in team output, morale, or turnover due to the move |
### Edges (4 total)
- `e-sum-nbb1z4m` supports → `ncwvq9x`
- `e-sum-nduqivt` supports → `ncwvq9x`
- `e-unk-nfq8rkd` depends_on → `ncwvq9x`
- `e-unk-nl723kx` depends_on → `ncwvq9x`
### Relevant unresolved unknowns (costs/savings/relocation)
1. `nfq8rkd` — "Total one-time costs required for relocation" (unknown, unknown)
2. `nl723kx` — "Potential short- to medium-term loss in team output, morale, or turnover due to the move" (unknown, unknown)
Note: `nx54wwa` (current/projected operating expenses) is **known**, not unresolved.
## UPDATE 1
**HTTP:** 422 | **Stage:** `proposal_compatibility`
### First Error
```
selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown
```
### Answer Meaning
```json
{
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
"possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs."
}
```
### Support Category / Resolution Guidance
**supportCategory populated:** NO (not present in answerMeaning)
**resolutionGuidance populated:** NO (not present in answerMeaning)
**Structured path:** NO — the structured field was not populated; meaning came through free-text `userSupportedMeaning` only.
### Rejected Proposal Snapshot
```json
{
"answerMeaning": {
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
"possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs."
},
"updatedNodes": [],
"resolvedUnknownNodeIds": [],
"addedNodes": [
{
"id": "nsavings_reality",
"kind": "unknown",
"label": "Realism of projected office savings from relocation",
"description": "Whether anticipated cost reductions match achievable financial outcomes, needed to decide if the relocation meets its core objective.",
"parentId": null,
"dependsOn": ["n20in8o"],
"affects": [],
"childIds": []
}
],
"addedEdges": [
{
"fromNodeId": "nsavings_reality",
"toNodeId": "n20in8o",
"relationship": "depends_on"
}
]
}
```
### Analysis of the Rejection
The model attempted to create a dedicated savings-realism unknown node (`nsavings_reality`) — which is the correct semantic interpretation. However, it also set `updatedNodes: []` and `resolvedUnknownNodeIds: []`, meaning no existing nodes were updated or resolved. The proposal created a new consequential unresolved unknown without updating any existing node to reflect the uncertainty (e.g., downgrading `nx54wwa` from known to provisional as 57J.56 did).
The system then rejected the proposal because creating a new consequential unknown requires a selected question for follow-up, but the proposal lacked this linkage. The rejection is structural: the model produced valid semantics but failed to complete the required graph mutation (updating existing nodes to reflect uncertainty) that would normally trigger the selected-question path.
### Update 1 Classification: U1-FAIL
No persistent savings-realism uncertainty was established in the graph because Update 1 was rejected. The proposed node (`nsavings_reality`) never entered the graph.
## Reachable for Update 2?
**NO.** Update 1 failed, so by experiment protocol the run stops. Update 2 is not reached.
## Classification: D — UPDATE 1 FAILED
The first turn never establishes the uncertainty needed for the inverse test. The model demonstrated correct semantic interpretation (it understood Answer 1 as savings-realism doubt and attempted to create a dedicated node), but failed at the structural linkage step: it did not update any existing node to reflect the uncertainty, leaving no selected-question trigger for downstream flow.
## What This Establishes
1. **Semantic interpretation works:** The model correctly interprets both Answer 1 and would have interpreted Answer 2 (had Update 1 succeeded) as savings-realism doubt.
2. **Dedicated node creation intent is correct:** The model's attempt to create `nsavings_reality` confirms v0.21's prompt clarification successfully guides the model toward distinct unknown nodes rather than absorption.
3. **Structural gap exposed:** The rejection reveals a gap where semantic interpretation succeeds but graph mutation fails silently — no existing node was updated (e.g., nx54wwa remained known instead of provisional), so the proposal lacked the structural trigger needed for question selection.
## What This Does NOT Prove
1. **Whether Answer 2 would have reused or duplicated:** We cannot answer the primary identity question because Update 1 never succeeded in establishing the persistent uncertainty that Update 2 would need to act upon.
2. **Downstream investigation viability:** The graph was not updated, so downstream investigation cannot be tested.
3. **Cross-domain generalisation:** Only tested on one scenario with one phrasing.
4. **Whether the structural gap is specific to cold-start vs. mid-investigation:** This occurred at cold start where nx54wwa (known) needed updating alongside new node creation — a different mutation pattern than 57J.56's update path which DID update nx54wwa.
## Production code changed: NO
## Prompt changed during experiment: NO
## Canonical harness restored: YES
## Hardened no-retry behaviour preserved: YES
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO