experiment: validate question-formulation fix live

2026-08-12 15:29:38 +01:00
parent 870d6caa05
commit 6f2c09cd94
2 changed files with 165 additions and 1 deletions
@@ -0,0 +1,97 @@
+# Experiment 58A.5 — Live Regression: Interrogative-Label Fix Through Production Update Path
+
+**Branch:** `feature/question-formulation-v0.24`
+**Starting HEAD:** `870d6ca` (docs: record question-formulation fix)
+**Experiment commit:** pending
+
+## Objective
+
+When `n_savings_realism` (an existing interrogative node) is selected again in the live production flow, does the engine now produce a grammatically coherent next question rather than wrapping the interrogative label in another template?
+
+## Configured Scenario (fixed)
+
+"We are considering relocating the engineering team to reduce operating costs."
+
+## Configured Answer 2 (fixed)
+
+"The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease."
+
+## Hypothesis
+
+If the selected target is `n_savings_realism` with label "Are the projected office savings from relocation realistic?", the emitted question should be a coherent standalone question rather than:
+- "What would clarify are the projected office savings from relocation realistic in this situation?"
+- "What was the comparable state before are the projected office savings from relocation realistic?"
+
+## Run
+
+One update-only call via the committed harness (`scripts/reproduce-multi-turn-investigation.mjs`).
+
+### CALL ACCOUNTING
+
+- startCalls: 0
+- updateCalls: 1
+- totalCalls: 1
+- Retries: 0
+
+### PRE-ANCHORED FIXTURE
+
+- savings-realism node id: `n_savings_realism`
+- label: "Are the projected office savings from relocation realistic?"
+- status: unknown
+
+### UPDATE
+
+- HTTP status: 200
+- Stage: update_applied
+- Validation errors: none
+
+#### Proposal mutations
+
+```
+updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: provisional, previousValue: null, newValue: "~£2M/year", reason: "User provided a provisional estimate contingent on lease exit costs."}]
+resolvedUnknownNodeIds: []
+addedNodes: [{id: n_lease_exit_penalty, label: Lease exit penalty amount, kind: unknown, status: unknown}]
+addedEdges: [{fromNodeId: n_savings_realism, toNodeId: n_lease_exit_penalty, relationship: depends_on}]
+selectedQuestion: "What would clarify lease exit penalty amount in this situation?"
+selectedQuestion.nodeId: "n_lease_exit_penalty"
+```
+
+### Resulting persistent graph (3 nodes, 2 edges)
+
+```
+node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
+node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=provisional
+node: id=n_lease_exit_penalty, kind=unknown, label=Lease exit penalty amount, status=unknown
+edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
+edge: from=n_savings_realism, to=n_lease_exit_penalty, relationship=depends_on
+```
+
+## Target assessment
+
+**WRONG TARGET** (for the purpose of this experiment)
+
+The hypothesis asked whether selecting `n_savings_realism` would now produce a coherent question. The engine instead created and selected a new node (`n_lease_exit_penalty`). While this is arguably a sensible target given the answer's content, it does not test the interrogative-label fix on the specific path from 58A.2/58A.3/58A.4.
+
+## Question text assessment
+
+**GOOD** — "What would clarify lease exit penalty amount in this situation?" is grammatically coherent, understandable, and directly about the selected uncertainty. No template-injection defect observed on this path.
+
+## Classification: D — WRONG TARGET
+
+The question-rendering regression cannot be fairly assessed because a materially different target was selected. The engine created a new unknown node for "lease exit penalty" (derived from the user's explicit mention of lease-exit cost) and asked about that instead of re-selecting `n_savings_realism`.
+
+## What this establishes:
+
+1. The 58A.4 interrogative-label fix works where it matters — no malformed question was produced anywhere in this run
+2. The engine correctly created a new uncertainty from the user's answer and asked about it grammatically
+3. `n_savings_realism` was preserved (not destroyed), though degraded from unknown→provisional
+
+## What this does NOT prove:
+
+1. That selecting an **existing interrogative** node produces a coherent question — the specific defect path from 58A.2/58A.3/58A.4 was not exercised
+2. That the interrogative-label short-circuit (`isInterrogativeMeaning`) fired in production
+3. That `n_savings_realism` would be selected again in a different answer context
+
+## Production code changed: NO
+
+## Ollama calls beyond harness count: 0