6.0 KiB
Experiment 58A.1 — Qualified Answer Reasoning
Branch: feature/semantic-action-contract-v0.23
Starting HEAD: a78f3edb1013c2205948cac3fee33042ab367ddd
Experiment commit: pending
Objective
When the user gives a mixed answer containing useful evidence, doubt, and a new assumption, does the engine preserve all three without over-resolving the existing uncertainty, and does it ask the right next question?
Fixed user answer
The £2 million saving looks attractive, but I don't really trust it yet. It assumes we can get out of the existing lease without a significant penalty, and it also doesn't include the disruption cost of moving the team.
Configured model: qwen-claude:latest at http://192.168.1.111:11434
CALL ACCOUNTING
- startCalls: 0
- updateCalls: 1
- totalCalls: 1
- Retries: 0
UPDATE RESULT
- HTTP status: 200
- Stage: update_applied
- Validation errors: none
- structuralActionRequired: null (known gap)
Proposal mutations
updatedNodes: []
resolvedUnknownNodeIds: []
addedNodes: [
{id: n_lease_penalty, label: "What is the lease exit penalty?", kind: unknown, status: unknown},
{id: n_disruption_cost, label: "What is the disruption cost?", kind: unknown, status: unknown}
]
addedEdges: [
{from: n_lease_penalty, to: n_relocation_state, relationship: depends_on},
{from: n_disruption_cost, to: n_relocation_state, relationship: depends_on}
]
selectedQuestion.nodeId: "n_disruption_cost"
Selected question
"What would clarify what is the disruption cost in this situation?" nodeId: n_disruption_cost
Resulting persistent graph (4 nodes, 3 edges)
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown
node: id=n_lease_penalty, kind=unknown, label=What is the lease exit penalty?, status=unknown
node: id=n_disruption_cost, kind=unknown, label=What is the disruption cost?, status=unknown
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
edge: from=n_lease_penalty, to=n_relocation_state, relationship=depends_on
edge: from=n_disruption_cost, to=n_relocation_state, relationship=depends_on
Reasoning assessment
£2m figure: LOST — not captured as newValue on any node, not represented in addedNodes or updatedNodes. The updatedNodes list is empty. Neither answerMeaning userSupportedMeaning nor supportCategory were printed by the harness.
User doubt: UNAVAILABLE — harness does not print answerMeaning fields for updateOnly mode's accepted path. Cannot verify from captured output whether doubt survived in userSupportedMeaning or was implicitly preserved through structural separation of assumptions.
Original savings-realism uncertainty: REMAINS UNRESOLVED — n_savings_realism persists with status=unknown, value=null. Not duplicated (no second savings-realism node).
Lease-exit assumption: STRUCTURALLY REPRESENTED — dedicated unknown node n_lease_penalty with kind=unknown, status=unknown, parentId linked to source state.
Disruption-cost assumption: STRUCTURALLY REPRESENTED — dedicated unknown node n_disruption_cost with kind=unknown, status=unknown, parentId linked to source state.
Next question quality: ACCEPTABLE — asks about disruption cost (the stronger of the two newly exposed uncertainties). Relevant and material, but asking lease penalty would have been equally or more direct since the user's core trust problem is about the £2m figure's validity, which directly depends on lease penalty. Disruption cost is a valid next step but less discriminative.
Classification: B — MOSTLY GOOD, INFORMATION LOSS
Core reasoning direction is right (preserves original uncertainty, creates structural nodes for new assumptions) but the £2m figure is lost — not captured as newValue, not attached to any node, and updatedNodes is empty. The engine understood what needed structurally but did not preserve the user's specific evidence in the graph.
What the engine understood correctly:
- The original savings-realism uncertainty should remain unresolved
- Two new material assumptions were exposed by the answer (lease penalty, disruption cost)
- These assumptions warrant dedicated unknown nodes rather than prose embedding
- A follow-up question should target one of these newly exposed uncertainties
- No duplicate savings-realism uncertainty was created
What information, if any, it lost:
The specific £2m figure and the user's trust qualification were not preserved in the graph state. With empty updatedNodes, no node carries the numerical claim that motivated the answer. This is meaningful evidence loss for an investigation engine — the anchor fact disappears from the graph.
What uncertainty it chose to pursue next:
Disruption cost (n_disruption_cost).
Was that the best available next uncertainty:
DEBATABLE — both lease penalty and disruption cost are equally valid next steps. Lease penalty may be slightly more discriminative because it directly attacks whether the £2m saving exists at all, while disruption cost is a subtractive factor on top of an assumed £2m baseline.
What this establishes:
- The engine can structurally represent multiple newly exposed assumptions as separate unknowns
- Original uncertainty identity is preserved without duplication
- A next question targeting a new structural node works correctly
- The updateOnly harness path for accepted updates does not print answerMeaning fields
What this does NOT prove:
- Whether the £2m figure survives through the answerMeaning pathway (harness gap)
- Whether the model can simultaneously create structural nodes AND propagate a newValue on existing nodes
- Stability across repeated runs