192 lines
8.3 KiB
Markdown
192 lines
8.3 KiB
Markdown
# Experiment 57J.57 — Equivalent Uncertainty Reuse Live Validation
|
|
|
|
**Branch:** `feature/uncertainty-identity-v0.21`
|
|
**Starting HEAD:** `eb524d0` (experiment: validate uncertainty identity live)
|
|
**Experiment commit:** pending
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> After the graph contains a dedicated savings-realism uncertainty, does a second semantically equivalent expression of that uncertainty reuse/refine the existing node rather than create a duplicate?
|
|
|
|
57J.56 established the "distinct uncertainty" half of the identity rule.
|
|
|
|
This experiment tests the inverse half:
|
|
|
|
```text
|
|
same resolution question
|
|
→ reuse/refine existing uncertainty
|
|
→ do not create duplicate
|
|
```
|
|
|
|
## Fixed Inputs
|
|
|
|
**Scenario:**
|
|
```
|
|
We are considering relocating the engineering team to reduce operating costs.
|
|
```
|
|
|
|
**Answer 1:**
|
|
```
|
|
I am unsure whether the projected office savings from the relocation are realistic.
|
|
```
|
|
|
|
**Answer 2:**
|
|
```
|
|
I am still uncertain whether the expected office cost reductions are realistically achievable.
|
|
```
|
|
|
|
These two answers express the **same resolution question**. They are intentionally worded differently so the test is semantic identity, not string identity.
|
|
|
|
## Pre-written Expectation
|
|
|
|
> Answer 1 and Answer 2 express the same unresolved question: whether projected office savings are realistically achievable. Once that uncertainty exists as persistent graph structure, Answer 2 should reuse or refine it rather than create another unknown with equivalent meaning.
|
|
|
|
## Run Results
|
|
|
|
### Configured Ollama
|
|
- **Base URL:** `http://192.168.1.111:11434` (from `.env.local`)
|
|
- **Model:** `qwen-claude:latest`
|
|
|
|
### Dev Server
|
|
- Running on `http://127.0.0.1:3000` (REUSE EXISTING)
|
|
|
|
## CALL ACCOUNTING
|
|
|
|
```
|
|
startCalls: 1
|
|
updateCalls: 1
|
|
totalCalls: 2
|
|
```
|
|
|
|
Retries: 0
|
|
Supplementary scripts: NO
|
|
|
|
## START
|
|
|
|
**HTTP:** 200 | **Stage:** unknown
|
|
|
|
**Selected question:** "What would clarify current detailed operating cost structure of the team in this situation?"
|
|
|
|
### Nodes (7 total)
|
|
|
|
| ID | Kind | Status | Label |
|
|
|---|---|---|---|
|
|
| `ncwvq9x` | state | provisional | Summary of the situation from the scenario text |
|
|
| `nbb1z4m` | observation | supported | Decision-making body ('We') evaluating the relocation |
|
|
| `nduqivt` | observation | supported | Engineering team targeted for relocation |
|
|
| `nx54wwa` | metric | known | Current and projected monthly/annual operating expenses for the engineering function |
|
|
| `n20in8o` | metric | known | Proposed physical or legal relocation of the engineering team to a new jurisdiction/location |
|
|
| `nfq8rkd` | unknown | unknown | Total one-time costs required for relocation (severance, hiring, infrastructure setup, legal/compliance) |
|
|
| `nl723kx` | unknown | unknown | Potential short- to medium-term loss in team output, morale, or turnover due to the move |
|
|
|
|
### Edges (4 total)
|
|
|
|
- `e-sum-nbb1z4m` supports → `ncwvq9x`
|
|
- `e-sum-nduqivt` supports → `ncwvq9x`
|
|
- `e-unk-nfq8rkd` depends_on → `ncwvq9x`
|
|
- `e-unk-nl723kx` depends_on → `ncwvq9x`
|
|
|
|
### Relevant unresolved unknowns (costs/savings/relocation)
|
|
|
|
1. `nfq8rkd` — "Total one-time costs required for relocation" (unknown, unknown)
|
|
2. `nl723kx` — "Potential short- to medium-term loss in team output, morale, or turnover due to the move" (unknown, unknown)
|
|
|
|
Note: `nx54wwa` (current/projected operating expenses) is **known**, not unresolved.
|
|
|
|
## UPDATE 1
|
|
|
|
**HTTP:** 422 | **Stage:** `proposal_compatibility`
|
|
|
|
### First Error
|
|
|
|
```
|
|
selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown
|
|
```
|
|
|
|
### Answer Meaning
|
|
|
|
```json
|
|
{
|
|
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
|
|
"possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs."
|
|
}
|
|
```
|
|
|
|
### Support Category / Resolution Guidance
|
|
|
|
**supportCategory populated:** NO (not present in answerMeaning)
|
|
**resolutionGuidance populated:** NO (not present in answerMeaning)
|
|
**Structured path:** NO — the structured field was not populated; meaning came through free-text `userSupportedMeaning` only.
|
|
|
|
### Rejected Proposal Snapshot
|
|
|
|
```json
|
|
{
|
|
"answerMeaning": {
|
|
"userSupportedMeaning": "The user is unsure whether the projected office savings from the relocation are realistic.",
|
|
"possibleInference": "If projections are unrealistic, the financial justification for relocating may be flawed, potentially leading to increased or unchanged operating costs."
|
|
},
|
|
"updatedNodes": [],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [
|
|
{
|
|
"id": "nsavings_reality",
|
|
"kind": "unknown",
|
|
"label": "Realism of projected office savings from relocation",
|
|
"description": "Whether anticipated cost reductions match achievable financial outcomes, needed to decide if the relocation meets its core objective.",
|
|
"parentId": null,
|
|
"dependsOn": ["n20in8o"],
|
|
"affects": [],
|
|
"childIds": []
|
|
}
|
|
],
|
|
"addedEdges": [
|
|
{
|
|
"fromNodeId": "nsavings_reality",
|
|
"toNodeId": "n20in8o",
|
|
"relationship": "depends_on"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Analysis of the Rejection
|
|
|
|
The model attempted to create a dedicated savings-realism unknown node (`nsavings_reality`) — which is the correct semantic interpretation. However, it also set `updatedNodes: []` and `resolvedUnknownNodeIds: []`, meaning no existing nodes were updated or resolved. The proposal created a new consequential unresolved unknown without updating any existing node to reflect the uncertainty (e.g., downgrading `nx54wwa` from known to provisional as 57J.56 did).
|
|
|
|
The system then rejected the proposal because creating a new consequential unknown requires a selected question for follow-up, but the proposal lacked this linkage. The rejection is structural: the model produced valid semantics but failed to complete the required graph mutation (updating existing nodes to reflect uncertainty) that would normally trigger the selected-question path.
|
|
|
|
### Update 1 Classification: U1-FAIL
|
|
|
|
No persistent savings-realism uncertainty was established in the graph because Update 1 was rejected. The proposed node (`nsavings_reality`) never entered the graph.
|
|
|
|
## Reachable for Update 2?
|
|
|
|
**NO.** Update 1 failed, so by experiment protocol the run stops. Update 2 is not reached.
|
|
|
|
## Classification: D — UPDATE 1 FAILED
|
|
|
|
The first turn never establishes the uncertainty needed for the inverse test. The model demonstrated correct semantic interpretation (it understood Answer 1 as savings-realism doubt and attempted to create a dedicated node), but failed at the structural linkage step: it did not update any existing node to reflect the uncertainty, leaving no selected-question trigger for downstream flow.
|
|
|
|
## What This Establishes
|
|
|
|
1. **Semantic interpretation works:** The model correctly interprets both Answer 1 and would have interpreted Answer 2 (had Update 1 succeeded) as savings-realism doubt.
|
|
2. **Dedicated node creation intent is correct:** The model's attempt to create `nsavings_reality` confirms v0.21's prompt clarification successfully guides the model toward distinct unknown nodes rather than absorption.
|
|
3. **Structural gap exposed:** The rejection reveals a gap where semantic interpretation succeeds but graph mutation fails silently — no existing node was updated (e.g., nx54wwa remained known instead of provisional), so the proposal lacked the structural trigger needed for question selection.
|
|
|
|
## What This Does NOT Prove
|
|
|
|
1. **Whether Answer 2 would have reused or duplicated:** We cannot answer the primary identity question because Update 1 never succeeded in establishing the persistent uncertainty that Update 2 would need to act upon.
|
|
2. **Downstream investigation viability:** The graph was not updated, so downstream investigation cannot be tested.
|
|
3. **Cross-domain generalisation:** Only tested on one scenario with one phrasing.
|
|
4. **Whether the structural gap is specific to cold-start vs. mid-investigation:** This occurred at cold start where nx54wwa (known) needed updating alongside new node creation — a different mutation pattern than 57J.56's update path which DID update nx54wwa.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed during experiment: NO
|
|
## Canonical harness restored: YES
|
|
## Hardened no-retry behaviour preserved: YES
|
|
## Ollama calls beyond harness count: 0
|
|
## Dev server disturbed: NO
|