120 lines
6.0 KiB
Markdown
120 lines
6.0 KiB
Markdown
# Experiment 58A.1 — Qualified Answer Reasoning
|
|
|
|
**Branch:** `feature/semantic-action-contract-v0.23`
|
|
**Starting HEAD:** `a78f3edb1013c2205948cac3fee33042ab367ddd`
|
|
**Experiment commit:** pending
|
|
|
|
## Objective
|
|
|
|
When the user gives a mixed answer containing useful evidence, doubt, and a new assumption, does the engine preserve all three without over-resolving the existing uncertainty, and does it ask the right next question?
|
|
|
|
## Fixed user answer
|
|
|
|
> The £2 million saving looks attractive, but I don't really trust it yet. It assumes we can get out of the existing lease without a significant penalty, and it also doesn't include the disruption cost of moving the team.
|
|
|
|
## Configured model: qwen-claude:latest at http://192.168.1.111:11434
|
|
|
|
### CALL ACCOUNTING
|
|
|
|
- startCalls: 0
|
|
- updateCalls: 1
|
|
- totalCalls: 1
|
|
- Retries: 0
|
|
|
|
### UPDATE RESULT
|
|
|
|
- HTTP status: 200
|
|
- Stage: update_applied
|
|
- Validation errors: none
|
|
- structuralActionRequired: null (known gap)
|
|
|
|
#### Proposal mutations
|
|
|
|
```
|
|
updatedNodes: []
|
|
resolvedUnknownNodeIds: []
|
|
addedNodes: [
|
|
{id: n_lease_penalty, label: "What is the lease exit penalty?", kind: unknown, status: unknown},
|
|
{id: n_disruption_cost, label: "What is the disruption cost?", kind: unknown, status: unknown}
|
|
]
|
|
addedEdges: [
|
|
{from: n_lease_penalty, to: n_relocation_state, relationship: depends_on},
|
|
{from: n_disruption_cost, to: n_relocation_state, relationship: depends_on}
|
|
]
|
|
selectedQuestion.nodeId: "n_disruption_cost"
|
|
```
|
|
|
|
#### Selected question
|
|
|
|
> "What would clarify what is the disruption cost in this situation?"
|
|
> nodeId: n_disruption_cost
|
|
|
|
### Resulting persistent graph (4 nodes, 3 edges)
|
|
|
|
```
|
|
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
|
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown
|
|
node: id=n_lease_penalty, kind=unknown, label=What is the lease exit penalty?, status=unknown
|
|
node: id=n_disruption_cost, kind=unknown, label=What is the disruption cost?, status=unknown
|
|
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
|
edge: from=n_lease_penalty, to=n_relocation_state, relationship=depends_on
|
|
edge: from=n_disruption_cost, to=n_relocation_state, relationship=depends_on
|
|
```
|
|
|
|
### Reasoning assessment
|
|
|
|
**£2m figure:** LOST — not captured as newValue on any node, not represented in addedNodes or updatedNodes. The `updatedNodes` list is empty. Neither answerMeaning userSupportedMeaning nor supportCategory were printed by the harness.
|
|
|
|
**User doubt:** UNAVAILABLE — harness does not print answerMeaning fields for updateOnly mode's accepted path. Cannot verify from captured output whether doubt survived in userSupportedMeaning or was implicitly preserved through structural separation of assumptions.
|
|
|
|
**Original savings-realism uncertainty:** REMAINS UNRESOLVED — n_savings_realism persists with status=unknown, value=null. Not duplicated (no second savings-realism node).
|
|
|
|
**Lease-exit assumption:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_lease_penalty` with kind=unknown, status=unknown, parentId linked to source state.
|
|
|
|
**Disruption-cost assumption:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_disruption_cost` with kind=unknown, status=unknown, parentId linked to source state.
|
|
|
|
**Next question quality:** ACCEPTABLE — asks about disruption cost (the stronger of the two newly exposed uncertainties). Relevant and material, but asking lease penalty would have been equally or more direct since the user's core trust problem is about the £2m figure's validity, which directly depends on lease penalty. Disruption cost is a valid next step but less discriminative.
|
|
|
|
### Classification: B — MOSTLY GOOD, INFORMATION LOSS
|
|
|
|
Core reasoning direction is right (preserves original uncertainty, creates structural nodes for new assumptions) but the £2m figure is lost — not captured as newValue, not attached to any node, and `updatedNodes` is empty. The engine understood what needed structurally but did not preserve the user's specific evidence in the graph.
|
|
|
|
### What the engine understood correctly:
|
|
|
|
1. The original savings-realism uncertainty should remain unresolved
|
|
2. Two new material assumptions were exposed by the answer (lease penalty, disruption cost)
|
|
3. These assumptions warrant dedicated unknown nodes rather than prose embedding
|
|
4. A follow-up question should target one of these newly exposed uncertainties
|
|
5. No duplicate savings-realism uncertainty was created
|
|
|
|
### What information, if any, it lost:
|
|
|
|
The specific £2m figure and the user's trust qualification were not preserved in the graph state. With empty `updatedNodes`, no node carries the numerical claim that motivated the answer. This is meaningful evidence loss for an investigation engine — the anchor fact disappears from the graph.
|
|
|
|
### What uncertainty it chose to pursue next:
|
|
|
|
Disruption cost (n_disruption_cost).
|
|
|
|
### Was that the best available next uncertainty:
|
|
|
|
DEBATABLE — both lease penalty and disruption cost are equally valid next steps. Lease penalty may be slightly more discriminative because it directly attacks whether the £2m saving exists at all, while disruption cost is a subtractive factor on top of an assumed £2m baseline.
|
|
|
|
### What this establishes:
|
|
|
|
1. The engine can structurally represent multiple newly exposed assumptions as separate unknowns
|
|
2. Original uncertainty identity is preserved without duplication
|
|
3. A next question targeting a new structural node works correctly
|
|
4. The updateOnly harness path for accepted updates does not print answerMeaning fields
|
|
|
|
### What this does NOT prove:
|
|
|
|
- Whether the £2m figure survives through the answerMeaning pathway (harness gap)
|
|
- Whether the model can simultaneously create structural nodes AND propagate a newValue on existing nodes
|
|
- Stability across repeated runs
|
|
|
|
### Production code changed: NO
|
|
### Prompt changed during experiment: NO
|
|
### Harness/tooling changed: NO
|
|
### Ollama calls beyond harness count: 0
|
|
### Dev server disturbed: NO
|