98 lines
4.5 KiB
Markdown
98 lines
4.5 KiB
Markdown
# Experiment 58A.5 — Live Regression: Interrogative-Label Fix Through Production Update Path
|
|
|
|
**Branch:** `feature/question-formulation-v0.24`
|
|
**Starting HEAD:** `870d6ca` (docs: record question-formulation fix)
|
|
**Experiment commit:** pending
|
|
|
|
## Objective
|
|
|
|
When `n_savings_realism` (an existing interrogative node) is selected again in the live production flow, does the engine now produce a grammatically coherent next question rather than wrapping the interrogative label in another template?
|
|
|
|
## Configured Scenario (fixed)
|
|
|
|
"We are considering relocating the engineering team to reduce operating costs."
|
|
|
|
## Configured Answer 2 (fixed)
|
|
|
|
"The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease."
|
|
|
|
## Hypothesis
|
|
|
|
If the selected target is `n_savings_realism` with label "Are the projected office savings from relocation realistic?", the emitted question should be a coherent standalone question rather than:
|
|
- "What would clarify are the projected office savings from relocation realistic in this situation?"
|
|
- "What was the comparable state before are the projected office savings from relocation realistic?"
|
|
|
|
## Run
|
|
|
|
One update-only call via the committed harness (`scripts/reproduce-multi-turn-investigation.mjs`).
|
|
|
|
### CALL ACCOUNTING
|
|
|
|
- startCalls: 0
|
|
- updateCalls: 1
|
|
- totalCalls: 1
|
|
- Retries: 0
|
|
|
|
### PRE-ANCHORED FIXTURE
|
|
|
|
- savings-realism node id: `n_savings_realism`
|
|
- label: "Are the projected office savings from relocation realistic?"
|
|
- status: unknown
|
|
|
|
### UPDATE
|
|
|
|
- HTTP status: 200
|
|
- Stage: update_applied
|
|
- Validation errors: none
|
|
|
|
#### Proposal mutations
|
|
|
|
```
|
|
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: provisional, previousValue: null, newValue: "~£2M/year", reason: "User provided a provisional estimate contingent on lease exit costs."}]
|
|
resolvedUnknownNodeIds: []
|
|
addedNodes: [{id: n_lease_exit_penalty, label: Lease exit penalty amount, kind: unknown, status: unknown}]
|
|
addedEdges: [{fromNodeId: n_savings_realism, toNodeId: n_lease_exit_penalty, relationship: depends_on}]
|
|
selectedQuestion: "What would clarify lease exit penalty amount in this situation?"
|
|
selectedQuestion.nodeId: "n_lease_exit_penalty"
|
|
```
|
|
|
|
### Resulting persistent graph (3 nodes, 2 edges)
|
|
|
|
```
|
|
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
|
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=provisional
|
|
node: id=n_lease_exit_penalty, kind=unknown, label=Lease exit penalty amount, status=unknown
|
|
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
|
edge: from=n_savings_realism, to=n_lease_exit_penalty, relationship=depends_on
|
|
```
|
|
|
|
## Target assessment
|
|
|
|
**WRONG TARGET** (for the purpose of this experiment)
|
|
|
|
The hypothesis asked whether selecting `n_savings_realism` would now produce a coherent question. The engine instead created and selected a new node (`n_lease_exit_penalty`). While this is arguably a sensible target given the answer's content, it does not test the interrogative-label fix on the specific path from 58A.2/58A.3/58A.4.
|
|
|
|
## Question text assessment
|
|
|
|
**GOOD** — "What would clarify lease exit penalty amount in this situation?" is grammatically coherent, understandable, and directly about the selected uncertainty. No template-injection defect observed on this path.
|
|
|
|
## Classification: D — WRONG TARGET
|
|
|
|
The question-rendering regression cannot be fairly assessed because a materially different target was selected. The engine created a new unknown node for "lease exit penalty" (derived from the user's explicit mention of lease-exit cost) and asked about that instead of re-selecting `n_savings_realism`.
|
|
|
|
## What this establishes:
|
|
|
|
1. The 58A.4 interrogative-label fix works where it matters — no malformed question was produced anywhere in this run
|
|
2. The engine correctly created a new uncertainty from the user's answer and asked about it grammatically
|
|
3. `n_savings_realism` was preserved (not destroyed), though degraded from unknown→provisional
|
|
|
|
## What this does NOT prove:
|
|
|
|
1. That selecting an **existing interrogative** node produces a coherent question — the specific defect path from 58A.2/58A.3/58A.4 was not exercised
|
|
2. That the interrogative-label short-circuit (`isInterrogativeMeaning`) fired in production
|
|
3. That `n_savings_realism` would be selected again in a different answer context
|
|
|
|
## Production code changed: NO
|
|
|
|
## Ollama calls beyond harness count: 0
|