confidence-engine/docs/experiment-58a3.md

Experiment 58A.3 — Next-Question Formulation Quality on Correct Target

Branch: feature/semantic-action-contract-v0.23 Starting HEAD: 20e4b58d7619bceee42a182f84c5265f0eab5841 Experiment commit: pending

Objective

Can the engine ask a coherent next question when the right target is already known?

58A.2 showed that the engine could preserve the £2m figure, preserve the lease-exit uncertainty, and choose the correct investigation target — but the selected question text was malformed (template injection failure).

This experiment tests that one thing only: question formulation quality when n_savings_realism is the selected target.

Fixed scenario (from fixture)

"We are considering relocating the engineering team to reduce operating costs."

Existing unresolved uncertainty: "Are the projected office savings from relocation realistic?"

Fixed answer

The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease.

Configured model: qwen-claude:latest at http://192.168.1.111:11434

CALL ACCOUNTING

• startCalls: 0
• updateCalls: 1
• totalCalls: 1
• Retries: 0

UPDATE RESULT

• HTTP status: 200
• Stage: update_applied
• Validation errors: none
• structuralActionRequired: null (known gap from 57J.69)

Proposal mutations

updatedNodes: [
  {nodeId: n_savings_realism, previousStatus: unknown, newStatus: weakened,
   previousValue: null, newValue: null,
   reason: "The answer explicitly withholds trust in the £2m projection until the lease exit cost is known, weakening current confidence."}
]
resolvedUnknownNodeIds: []
addedNodes: [
  {id: n_lease_penalty, label: "Office lease exit penalty amount",
   description: "Total cost to break the current London office lease, needed to decide if the £2m/year relocation savings are realistic.",
   kind: unknown, status: unknown, confidence: low}
]
addedEdges: [
  {fromNodeId: n_savings_realism, toNodeId: n_lease_penalty, relationship: depends_on, confidence: medium,
   description: "Assessing savings realism depends on determining the lease exit penalty."}
]
selectedQuestion.nodeId: "n_savings_realism"

Selected question

"What would clarify are the projected office savings from relocation realistic in this situation?" nodeId: n_savings_realism

(Note: question text is grammatically broken — same template injection failure as 58A.2.)

Resulting persistent graph (3 nodes, 2 edges)

node: id=n_relocation_state,       kind=state,     label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism,        kind=unknown,   label=Are the projected office savings from relocation realistic?, status=weakened
node: id=n_lease_penalty,          kind=unknown,   label=Office lease exit penalty amount, status=unknown
edge: from=n_savings_realism,      to=n_relocation_state,     relationship=depends_on
edge: from=n_savings_realism,      to=n_lease_penalty,        relationship=depends_on

Reasoning assessment

Target selection: n_savings_realism is a GOOD TARGET. It is the existing savings-realism uncertainty that was already present and remains unresolved. The engine correctly chose it as the next investigation focus.

Question text quality: MALFORMED. The sentence "What would clarify are the projected office savings from relocation realistic in this situation?" fuses the template frame "What would clarify [X] in this situation?" with the full unknown label "are the projected office savings from relocation realistic", producing an ungrammatical sentence. A grammatically correct version would read: "What would clarify whether the projected office savings from relocation are realistic in this situation?" or simply "Are the projected office savings from relocation realistic in this situation?"

£2m figure preservation: LOST (relative to 58A.2). The updatedNode for n_savings_realism has newValue=null — the £2m figure was not carried forward at all in this run. In 58A.2, it survived as "~£2,000,000/year (pending lease exit cost)". The status changed to weakened instead of 58A.2's provisional, which signals a different reasoning pattern but equally loses the evidence.

Lease-exit uncertainty: STRUCTURALLY REPRESENTED — dedicated unknown node n_lease_penalty with clear description referencing the £2m/year savings context, plus a depends_on edge from n_savings_realism to it. This matches 58A.2's pattern.

Original savings-realism identity status: CHANGED from unknown → weakened. The status weakened (rather than 58A.2's provisional) signals that the model interpreted the user's doubt about the £2m figure as a reason to downgrade confidence in the uncertainty itself, rather than preserving it as an active investigation target. This is arguably correct reasoning (the user expressed distrust) but structurally the node no longer functions as "unresolved — needs evidence" since weakened has different downstream semantics than unknown.

Target assessment: GOOD TARGET

n_savings_realism is the correct next investigation target given the existing state. It was already unresolved, and the user's answer directly qualified its supporting evidence.

Question text assessment: MALFORMED

The question fuses a template frame with an unknown label into ungrammatical output. This is the same class of defect as 58A.2.

Classification: C — TARGET GOOD, QUESTION MALFORMED

Correct target selection, broken question text. The root cause remains in the question-formulation pipeline's handling of existing-node targets (not newly-created ones).

Why:

The model correctly identified n_savings_realism as the next investigation focus. However, the question formulation template system produced a grammatically broken sentence by inserting the full unknown label directly into a template without respecting English grammar boundaries.

Was the selected target sensible: YES

Was the question grammatically coherent: NO

Would answering it materially reduce the selected uncertainty: DEBATABLE — The question is broken, so no answer could be reliably given to it. A corrected version ("What would clarify whether the projected office savings from relocation are realistic?") would directly address the uncertainty.

Did the malformed-template failure from 58A.2 recur: YES

The exact same class of template injection failure persisted across two independent runs. The question text is nearly identical between 58A.2 and 58A.3:

• 58A.2: "What was the comparable state before are the projected office savings from relocation realistic?"
• 58A.3: "What would clarify are the projected office savings from relocation realistic in this situation?"

Both fuse a template frame with the unknown label, producing ungrammatical output. The specific template variant differs but the defect mechanism is identical.

What this establishes:

1. The malformed-template question-formulation defect is reproducible — it is not a cold-start anomaly from 58A.2
2. The defect persists across different model call contexts (subsequent to a graph update)
3. Target selection logic correctly chose n_savings_realism in this run
4. A new regression appeared: the £2m figure was not preserved in newValue (null), unlike 58A.2 where it survived as qualified text
5. The status change unknown → weakened (this run) vs unknown → provisional (58A.2) shows run-to-run variance in the model's assessment semantics

What this does NOT prove:

• Whether a question-formulation fix would work for newly-created unknown nodes (only existing-node targeting was tested)
• Whether different target nodes produce different template behavior
• Cross-domain generalisation
• Whether the £2m-loss regression is stable or 58A.2 was the lucky run

Production code changed: NO

Prompt changed during experiment: NO

Harness/tooling changed: NO

Ollama calls beyond harness count: 0

Dev server disturbed: NO