docs: archive historical Confidence Engine evidence
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
# Experiment 58A.1 — Qualified Answer Reasoning
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `a78f3edb1013c2205948cac3fee33042ab367ddd`
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
When the user gives a mixed answer containing useful evidence, doubt, and a new assumption, does the engine preserve all three without over-resolving the existing uncertainty, and does it ask the right next question?
|
||||
|
||||
## Fixed user answer
|
||||
|
||||
> The £2 million saving looks attractive, but I don't really trust it yet. It assumes we can get out of the existing lease without a significant penalty, and it also doesn't include the disruption cost of moving the team.
|
||||
|
||||
## Configured model: qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 0
|
||||
- updateCalls: 1
|
||||
- totalCalls: 1
|
||||
- Retries: 0
|
||||
|
||||
### UPDATE RESULT
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
- structuralActionRequired: null (known gap)
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [
|
||||
{id: n_lease_penalty, label: "What is the lease exit penalty?", kind: unknown, status: unknown},
|
||||
{id: n_disruption_cost, label: "What is the disruption cost?", kind: unknown, status: unknown}
|
||||
]
|
||||
addedEdges: [
|
||||
{from: n_lease_penalty, to: n_relocation_state, relationship: depends_on},
|
||||
{from: n_disruption_cost, to: n_relocation_state, relationship: depends_on}
|
||||
]
|
||||
selectedQuestion.nodeId: "n_disruption_cost"
|
||||
```
|
||||
|
||||
#### Selected question
|
||||
|
||||
> "What would clarify what is the disruption cost in this situation?"
|
||||
> nodeId: n_disruption_cost
|
||||
|
||||
### Resulting persistent graph (4 nodes, 3 edges)
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown
|
||||
node: id=n_lease_penalty, kind=unknown, label=What is the lease exit penalty?, status=unknown
|
||||
node: id=n_disruption_cost, kind=unknown, label=What is the disruption cost?, status=unknown
|
||||
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
||||
edge: from=n_lease_penalty, to=n_relocation_state, relationship=depends_on
|
||||
edge: from=n_disruption_cost, to=n_relocation_state, relationship=depends_on
|
||||
```
|
||||
|
||||
### Reasoning assessment
|
||||
|
||||
**£2m figure:** LOST — not captured as newValue on any node, not represented in addedNodes or updatedNodes. The `updatedNodes` list is empty. Neither answerMeaning userSupportedMeaning nor supportCategory were printed by the harness.
|
||||
|
||||
**User doubt:** UNAVAILABLE — harness does not print answerMeaning fields for updateOnly mode's accepted path. Cannot verify from captured output whether doubt survived in userSupportedMeaning or was implicitly preserved through structural separation of assumptions.
|
||||
|
||||
**Original savings-realism uncertainty:** REMAINS UNRESOLVED — n_savings_realism persists with status=unknown, value=null. Not duplicated (no second savings-realism node).
|
||||
|
||||
**Lease-exit assumption:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_lease_penalty` with kind=unknown, status=unknown, parentId linked to source state.
|
||||
|
||||
**Disruption-cost assumption:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_disruption_cost` with kind=unknown, status=unknown, parentId linked to source state.
|
||||
|
||||
**Next question quality:** ACCEPTABLE — asks about disruption cost (the stronger of the two newly exposed uncertainties). Relevant and material, but asking lease penalty would have been equally or more direct since the user's core trust problem is about the £2m figure's validity, which directly depends on lease penalty. Disruption cost is a valid next step but less discriminative.
|
||||
|
||||
### Classification: B — MOSTLY GOOD, INFORMATION LOSS
|
||||
|
||||
Core reasoning direction is right (preserves original uncertainty, creates structural nodes for new assumptions) but the £2m figure is lost — not captured as newValue, not attached to any node, and `updatedNodes` is empty. The engine understood what needed structurally but did not preserve the user's specific evidence in the graph.
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. The original savings-realism uncertainty should remain unresolved
|
||||
2. Two new material assumptions were exposed by the answer (lease penalty, disruption cost)
|
||||
3. These assumptions warrant dedicated unknown nodes rather than prose embedding
|
||||
4. A follow-up question should target one of these newly exposed uncertainties
|
||||
5. No duplicate savings-realism uncertainty was created
|
||||
|
||||
### What information, if any, it lost:
|
||||
|
||||
The specific £2m figure and the user's trust qualification were not preserved in the graph state. With empty `updatedNodes`, no node carries the numerical claim that motivated the answer. This is meaningful evidence loss for an investigation engine — the anchor fact disappears from the graph.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
Disruption cost (n_disruption_cost).
|
||||
|
||||
### Was that the best available next uncertainty:
|
||||
|
||||
DEBATABLE — both lease penalty and disruption cost are equally valid next steps. Lease penalty may be slightly more discriminative because it directly attacks whether the £2m saving exists at all, while disruption cost is a subtractive factor on top of an assumed £2m baseline.
|
||||
|
||||
### What this establishes:
|
||||
|
||||
1. The engine can structurally represent multiple newly exposed assumptions as separate unknowns
|
||||
2. Original uncertainty identity is preserved without duplication
|
||||
3. A next question targeting a new structural node works correctly
|
||||
4. The updateOnly harness path for accepted updates does not print answerMeaning fields
|
||||
|
||||
### What this does NOT prove:
|
||||
|
||||
- Whether the £2m figure survives through the answerMeaning pathway (harness gap)
|
||||
- Whether the model can simultaneously create structural nodes AND propagate a newValue on existing nodes
|
||||
- Stability across repeated runs
|
||||
|
||||
### Production code changed: NO
|
||||
### Prompt changed during experiment: NO
|
||||
### Harness/tooling changed: NO
|
||||
### Ollama calls beyond harness count: 0
|
||||
### Dev server disturbed: NO
|
||||
@@ -0,0 +1,142 @@
|
||||
# Experiment 58A.2 — Evidence + Uncertainty with One Competing Demand
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `3218469cfb081d5f9e7e3ed1a763482fdd2a241f`
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
Can the engine preserve a concrete figure while also creating one new uncertainty?
|
||||
|
||||
58A.1 showed good uncertainty reasoning but lost the central £2m figure while creating two new unknowns. This experiment reduces the answer to **one concrete figure + one qualification + one new assumption** to test whether information loss was caused by competing structural demands rather than a fundamental inability to represent qualified evidence.
|
||||
|
||||
## Fixed scenario (from fixture)
|
||||
|
||||
"We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
Existing unresolved uncertainty: "Are the projected office savings from relocation realistic?"
|
||||
|
||||
## Fixed answer
|
||||
|
||||
> The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease.
|
||||
|
||||
## Configured model: qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 0
|
||||
- updateCalls: 1
|
||||
- totalCalls: 1
|
||||
- Retries: 0
|
||||
|
||||
### UPDATE RESULT
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
- structuralActionRequired: null (known gap from 57J.69)
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [
|
||||
{nodeId: n_savings_realism, previousStatus: unknown, newStatus: provisional,
|
||||
previousValue: null, newValue: "~£2,000,000/year (pending lease exit cost)",
|
||||
reason: "User provided a provisional savings projection that is contingent on unresolved lease termination costs."}
|
||||
]
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [
|
||||
{id: n_lease_penalty_uncertainty, label: "Lease exit penalty amount",
|
||||
description: "Financial penalty required to break the current London office lease before relocation,
|
||||
because it directly determines whether the ~£2M/year savings projection will actually materialize.",
|
||||
kind: unknown, status: unknown}
|
||||
]
|
||||
addedEdges: [
|
||||
{fromNodeId: n_savings_realism, toNodeId: n_lease_penalty_uncertainty,
|
||||
relationship: depends_on, confidence: high,
|
||||
description: "Realizing the projected savings depends on quantifying the lease exit penalty."}
|
||||
]
|
||||
selectedQuestion.nodeId: "n_savings_realism"
|
||||
```
|
||||
|
||||
#### Selected question
|
||||
|
||||
> "What was the comparable state before are the projected office savings from relocation realistic?"
|
||||
> nodeId: n_savings_realism
|
||||
|
||||
(Note: question text appears malformed — template injection failure producing grammatically broken sentence.)
|
||||
|
||||
### Resulting persistent graph (3 nodes, 2 edges)
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=provisional
|
||||
node: id=n_lease_penalty_uncertainty, kind=unknown, label=Lease exit penalty amount, status=unknown
|
||||
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
||||
edge: from=n_savings_realism, to=n_lease_penalty_uncertainty, relationship=depends_on
|
||||
```
|
||||
|
||||
### Reasoning assessment
|
||||
|
||||
**£2m figure:** PRESERVED AS QUALIFIED — the value `~£2,000,000/year (pending lease exit cost)` survives on the savings-realism node. It is preserved with qualifier text, though not at full precision ("about £2 million" → "~£2,000,000") and not as a standalone reported_claim node.
|
||||
|
||||
**Qualification:** WEAKENED — The status change from `unknown` → `provisional` on n_savings_realism is the primary signal of weakening. "Provisional" suggests tentative acceptance rather than active investigation. The qualification survives in newValue text ("pending lease exit cost") and in the reason prose, but structurally the node no longer functions as an unresolved question — it functions as a tentative fact that needs verification.
|
||||
|
||||
**Lease-exit uncertainty:** STRUCTURALLY REPRESENTED — dedicated unknown node with kind=unknown, status=unknown, explicit description tying it to the savings figure, plus a `depends_on` edge from n_savings_realism to this node. The structural representation is stronger than 58A.1's lease-exit handling.
|
||||
|
||||
**Original savings-realism uncertainty:** WEAKENED — The node identity persists (n_savings_realism still exists, not duplicated). However, the status change from `unknown` → `provisional` means it no longer signals "unresolved investigation target" — it signals "tentatively accepted but needs verification." This is a degradation of uncertainty signaling that could mislead downstream question selection and Behaviour Selection.
|
||||
|
||||
**Evidence / uncertainty linkage:** CLEARLY LINKED — The `depends_on` edge from n_savings_realism to n_lease_penalty_uncertainty structurally encodes the dependency relationship: realizing savings depends on quantifying the penalty. Description reinforces this ("Realizing the projected savings depends on quantifying the lease exit penalty.").
|
||||
|
||||
**Next question quality:** WRONG — "What was the comparable state before are the projected office savings from relocation realistic?" is a grammatically broken template injection (combining "What was the comparable state before [X]?" with "[X]" = full unknown label). It does not materially help determine whether the £2m figure is realistic.
|
||||
|
||||
### Classification: E — IDENTITY FAILURE
|
||||
|
||||
The original savings-realism uncertainty node's status was degraded from `unknown` to `provisional`, weakening its identity as an unresolved investigation target. This is not a correct resolution (status remains unknown-ish but with degraded semantics), nor is it simply "preserved." The uncertainty exists in a degraded state that could mislead downstream reasoning stages about the investigation's health.
|
||||
|
||||
Additionally, the selected question is malformed and fails to pursue any material unresolved issue.
|
||||
|
||||
### What the engine preserved correctly:
|
||||
|
||||
1. The £2m/year figure survived as qualified evidence (newValue on existing node)
|
||||
2. The lease-exit uncertainty was structurally represented with a dedicated unknown node
|
||||
3. Evidence and new uncertainty are clearly linked via depends_on edge + description
|
||||
4. No duplicate savings-realism uncertainty was created
|
||||
5. No-resolve guard worked (resolvedUnknownNodeIds is empty)
|
||||
|
||||
### What it lost or weakened:
|
||||
|
||||
1. The savings-realism uncertainty identity — degraded from `unknown` to `provisional`, losing its function as an active investigation target
|
||||
2. Question quality — malformed sentence that does not pursue the material unresolved issue
|
||||
3. Precision of the £2m figure ("about £2 million" → "~£2,000,000")
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
n_savings_realism (the existing savings-realism unknown), but the question text is broken and does not target the lease-exit penalty or any other material issue.
|
||||
|
||||
### Was that the best available next uncertainty:
|
||||
|
||||
YES — n_savings_realism is the correct investigation target, but the execution of the question (malformed text) renders this moot.
|
||||
|
||||
### Comparison with 58A.1:
|
||||
|
||||
58A.1 lost the £2m figure entirely but preserved savings-realism as `unknown` and produced a grammatically coherent (if debatable) next question. 58A.2 preserves both the figure and the new uncertainty, but at the cost of degrading the savings-realism node from `unknown` to `provisional` and producing a malformed question. The trade-off is clear: reducing competing demands (2 unknowns → 1 unknown) solved the evidence-loss problem but introduced a status-degradation failure. This establishes that evidence preservation and uncertainty preservation are not simply inverses of each other — there is a separate mechanism controlling node status that can degrade uncertainty identity even when evidence survives.
|
||||
|
||||
### What this establishes:
|
||||
|
||||
1. Reducing competing structural demands (one unknown instead of two) allows the figure to survive as qualified evidence
|
||||
2. The status-degradation failure (unknown → provisional) is a distinct failure mode from evidence loss — it occurs alongside successful evidence preservation
|
||||
3. Question-formulation templates have a defect when targeting existing (non-newly-created) unknown nodes
|
||||
|
||||
### What this does NOT prove:
|
||||
|
||||
- Whether the status-degradation issue would recur with different model configurations or repeated runs
|
||||
- Whether a dedicated reported_claim node type would avoid the status conflict
|
||||
- Whether question-formulation templates can be fixed to handle existing-node targets correctly
|
||||
- Cross-domain generalisation of any finding
|
||||
|
||||
### Production code changed: NO
|
||||
### Prompt changed during experiment: NO
|
||||
### Harness/tooling changed: NO
|
||||
### Ollama calls beyond harness count: 0
|
||||
### Dev server disturbed: NO
|
||||
@@ -0,0 +1,140 @@
|
||||
# Experiment 58A.3 — Next-Question Formulation Quality on Correct Target
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `20e4b58d7619bceee42a182f84c5265f0eab5841`
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
Can the engine ask a coherent next question when the right target is already known?
|
||||
|
||||
58A.2 showed that the engine could preserve the £2m figure, preserve the lease-exit uncertainty, and choose the correct investigation target — but the selected question text was malformed (template injection failure).
|
||||
|
||||
This experiment tests that one thing only: **question formulation quality** when `n_savings_realism` is the selected target.
|
||||
|
||||
## Fixed scenario (from fixture)
|
||||
|
||||
"We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
Existing unresolved uncertainty: "Are the projected office savings from relocation realistic?"
|
||||
|
||||
## Fixed answer
|
||||
|
||||
> The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease.
|
||||
|
||||
## Configured model: qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 0
|
||||
- updateCalls: 1
|
||||
- totalCalls: 1
|
||||
- Retries: 0
|
||||
|
||||
### UPDATE RESULT
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
- structuralActionRequired: null (known gap from 57J.69)
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [
|
||||
{nodeId: n_savings_realism, previousStatus: unknown, newStatus: weakened,
|
||||
previousValue: null, newValue: null,
|
||||
reason: "The answer explicitly withholds trust in the £2m projection until the lease exit cost is known, weakening current confidence."}
|
||||
]
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [
|
||||
{id: n_lease_penalty, label: "Office lease exit penalty amount",
|
||||
description: "Total cost to break the current London office lease, needed to decide if the £2m/year relocation savings are realistic.",
|
||||
kind: unknown, status: unknown, confidence: low}
|
||||
]
|
||||
addedEdges: [
|
||||
{fromNodeId: n_savings_realism, toNodeId: n_lease_penalty, relationship: depends_on, confidence: medium,
|
||||
description: "Assessing savings realism depends on determining the lease exit penalty."}
|
||||
]
|
||||
selectedQuestion.nodeId: "n_savings_realism"
|
||||
```
|
||||
|
||||
#### Selected question
|
||||
|
||||
> "What would clarify are the projected office savings from relocation realistic in this situation?"
|
||||
> nodeId: n_savings_realism
|
||||
|
||||
(Note: question text is grammatically broken — same template injection failure as 58A.2.)
|
||||
|
||||
### Resulting persistent graph (3 nodes, 2 edges)
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=weakened
|
||||
node: id=n_lease_penalty, kind=unknown, label=Office lease exit penalty amount, status=unknown
|
||||
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
||||
edge: from=n_savings_realism, to=n_lease_penalty, relationship=depends_on
|
||||
```
|
||||
|
||||
### Reasoning assessment
|
||||
|
||||
**Target selection:** n_savings_realism is a GOOD TARGET. It is the existing savings-realism uncertainty that was already present and remains unresolved. The engine correctly chose it as the next investigation focus.
|
||||
|
||||
**Question text quality:** MALFORMED. The sentence "What would clarify are the projected office savings from relocation realistic in this situation?" fuses the template frame "What would clarify [X] in this situation?" with the full unknown label "are the projected office savings from relocation realistic", producing an ungrammatical sentence. A grammatically correct version would read: "What would clarify whether the projected office savings from relocation are realistic in this situation?" or simply "Are the projected office savings from relocation realistic in this situation?"
|
||||
|
||||
**£2m figure preservation:** LOST (relative to 58A.2). The updatedNode for n_savings_realism has newValue=null — the £2m figure was not carried forward at all in this run. In 58A.2, it survived as "~£2,000,000/year (pending lease exit cost)". The status changed to `weakened` instead of 58A.2's `provisional`, which signals a different reasoning pattern but equally loses the evidence.
|
||||
|
||||
**Lease-exit uncertainty:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_lease_penalty` with clear description referencing the £2m/year savings context, plus a depends_on edge from n_savings_realism to it. This matches 58A.2's pattern.
|
||||
|
||||
**Original savings-realism identity status:** CHANGED from `unknown` → `weakened`. The status `weakened` (rather than 58A.2's `provisional`) signals that the model interpreted the user's doubt about the £2m figure as a reason to downgrade confidence in the uncertainty itself, rather than preserving it as an active investigation target. This is arguably correct reasoning (the user expressed distrust) but structurally the node no longer functions as "unresolved — needs evidence" since `weakened` has different downstream semantics than `unknown`.
|
||||
|
||||
### Target assessment: GOOD TARGET
|
||||
|
||||
n_savings_realism is the correct next investigation target given the existing state. It was already unresolved, and the user's answer directly qualified its supporting evidence.
|
||||
|
||||
### Question text assessment: MALFORMED
|
||||
|
||||
The question fuses a template frame with an unknown label into ungrammatical output. This is the same class of defect as 58A.2.
|
||||
|
||||
### Classification: C — TARGET GOOD, QUESTION MALFORMED
|
||||
|
||||
Correct target selection, broken question text. The root cause remains in the question-formulation pipeline's handling of existing-node targets (not newly-created ones).
|
||||
|
||||
### Why:
|
||||
|
||||
The model correctly identified n_savings_realism as the next investigation focus. However, the question formulation template system produced a grammatically broken sentence by inserting the full unknown label directly into a template without respecting English grammar boundaries.
|
||||
|
||||
### Was the selected target sensible: YES
|
||||
|
||||
### Was the question grammatically coherent: NO
|
||||
|
||||
### Would answering it materially reduce the selected uncertainty: DEBATABLE — The question is broken, so no answer could be reliably given to it. A corrected version ("What would clarify whether the projected office savings from relocation are realistic?") would directly address the uncertainty.
|
||||
|
||||
### Did the malformed-template failure from 58A.2 recur: YES
|
||||
|
||||
The exact same class of template injection failure persisted across two independent runs. The question text is nearly identical between 58A.2 and 58A.3:
|
||||
- 58A.2: "What was the comparable state before are the projected office savings from relocation realistic?"
|
||||
- 58A.3: "What would clarify are the projected office savings from relocation realistic in this situation?"
|
||||
|
||||
Both fuse a template frame with the unknown label, producing ungrammatical output. The specific template variant differs but the defect mechanism is identical.
|
||||
|
||||
### What this establishes:
|
||||
|
||||
1. The malformed-template question-formulation defect is **reproducible** — it is not a cold-start anomaly from 58A.2
|
||||
2. The defect persists across different model call contexts (subsequent to a graph update)
|
||||
3. Target selection logic correctly chose n_savings_realism in this run
|
||||
4. A new regression appeared: the £2m figure was not preserved in newValue (null), unlike 58A.2 where it survived as qualified text
|
||||
5. The status change `unknown` → `weakened` (this run) vs `unknown` → `provisional` (58A.2) shows run-to-run variance in the model's assessment semantics
|
||||
|
||||
### What this does NOT prove:
|
||||
|
||||
- Whether a question-formulation fix would work for newly-created unknown nodes (only existing-node targeting was tested)
|
||||
- Whether different target nodes produce different template behavior
|
||||
- Cross-domain generalisation
|
||||
- Whether the £2m-loss regression is stable or 58A.2 was the lucky run
|
||||
|
||||
### Production code changed: NO
|
||||
### Prompt changed during experiment: NO
|
||||
### Harness/tooling changed: NO
|
||||
### Ollama calls beyond harness count: 0
|
||||
### Dev server disturbed: NO
|
||||
@@ -0,0 +1,72 @@
|
||||
# Experiment 58A.4 — Interrogative Label Question Formulation
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Starting HEAD:** `b1914f5` (experiment: test next-question formulation)
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
Can the engine produce grammatically correct follow-up questions when the active unknown's label is already question-shaped?
|
||||
|
||||
Experiments 58A.2 and 58A.3 showed that the engine could select the correct investigation target — but the selected question text was malformed due to template injection failure: a declarative-frame template (e.g., "What would clarify [X] in this situation?") was interpolated with an interrogative label ("are the projected office savings from relocation realistic"), producing sentences like **"What would clarify are the projected office savings from relocation realistic in this situation?"**.
|
||||
|
||||
## Defect Analysis
|
||||
|
||||
**Root Cause:** `buildNeutralClarificationQuestion`, `buildEvidenceFallbackQuestion`, `buildQuestionFromFamily`, and `buildQuestionFromStrategy` all interpolate `meaning` (derived from the unknown's label) directly into template frames without first detecting whether that meaning is already an interrogative (wh-question, yes/no question, or modal-auxiliary inversion).
|
||||
|
||||
**Manifestation across 7 code paths:**
|
||||
- Template injection in `buildNeutralClarificationQuestion` → `"What would clarify [interrogative] in this situation?"`
|
||||
- Template injection in `buildEvidenceFallbackQuestion` → `"What evidence would confirm or rule out [interrogative]?"`
|
||||
- Template injection in `buildQuestionFromFamily` (decision path) → `"What evidence would clarify [interrogative]?"`
|
||||
- Template injection in `buildQuestionFromFamily` (definition path) → `"What does [interrogative] mean…"`
|
||||
- Template injection in `buildQuestionFromFamily` (comparison path) → `"What evidence would clarify [interrogative]?"`
|
||||
- Template injection in `buildQuestionFromFamily` (contradiction path) → `"What fact would resolve the contradiction about [interrogative]?"`
|
||||
- Template injection in `buildQuestionFromStrategy` → multiple strategies
|
||||
|
||||
## Fix: Detect and short-circuit interrogative meanings
|
||||
|
||||
### New function: `isInterrogativeMeaning(meaning)`
|
||||
|
||||
Detects whether a meaning string is already an interrogative by checking:
|
||||
|
||||
1. **Wh-prefix**: labels starting with `who`, `what`, `where`, `when`, `how`
|
||||
2. **Subject-auxiliary inversion**: first word is an auxiliary/modal verb (`is`, `are`, `was`, `were`, `do`, `does`, `will`, etc.) followed by a subject determiner pronoun (`the`, `a`, `an`, `this`, `that`, `my`, `your`, `we`, `they`, etc.) — covers "Is the budget sufficient?", "Are these measures valid?", "Who would decide this?"
|
||||
3. **Whether-clause**: labels starting with `whether`
|
||||
|
||||
### New function: `wrapInterrogativeForTemplate(meaning)`
|
||||
|
||||
Returns interrogative meanings unchanged (they are already coherent standalone questions). For non-interrogative meanings, returns them as-is for safe template interpolation.
|
||||
|
||||
### Modified functions
|
||||
|
||||
All five question-builders now short-circuit before template interpolation when the meaning is interrogative, returning it directly with a trailing `?`. This preserves the user's original phrasing exactly rather than injecting it into a declarative frame.
|
||||
|
||||
## Test Results
|
||||
|
||||
**20 new tests** added in `tests/graph/question-formulation-v0.24.test.js` covering:
|
||||
- Wh-question labels (who, what, where, when, how)
|
||||
- Yes/no question labels (is/are/was auxiliary inversion)
|
||||
- Whether-clause labels
|
||||
- Declarative labels (to ensure they still get template frames)
|
||||
- Long complex interrogatives
|
||||
- Definition and evidence reasoning paths
|
||||
|
||||
**39 tests pass (20 new + 19 existing)** — no regressions.
|
||||
|
||||
## Output Examples
|
||||
|
||||
| Label | Old Output (defective) | New Output |
|
||||
|-------|----------------------|------------|
|
||||
| "Are the projected office savings from relocation realistic?" | "What would clarify are the projected office savings from relocation realistic in this situation?" | "are the projected office savings from relocation realistic?" |
|
||||
| "What are the key risks of this project?" | "what would clarify what are the key risks of this project in this situation?" | "what are the key risks of this project?" |
|
||||
| "How do we measure success for this initiative?" | "what would clarify how do we measure success for this initiative in this situation?" | "how do we measure success for this initiative?" |
|
||||
| "Is this the right approach?" | "What would clarify is this the right approach in this situation?" | "is this the right approach?" |
|
||||
| "Office lease exit penalty amount" | (Same as before — template frame) | "What would clarify office lease exit penalty amount in this situation?" |
|
||||
|
||||
## Classification: PASS
|
||||
|
||||
The fix addresses the root cause (template injection of interrogative labels) structurally rather than by pattern-matching specific defects. It generalises to ALL interrogative forms, not just those seen so far.
|
||||
|
||||
### Pre-existing failures on this branch (NOT caused by this fix):
|
||||
- `question-priority-generalisation.test.js`: 5/6 tests fail — deterministic selection mismatch (pre-existing)
|
||||
- `selection-influence-diagnostic.test.js`: 1 test fails — expected vs received question format (pre-existing)
|
||||
@@ -0,0 +1,97 @@
|
||||
# Experiment 58A.5 — Live Regression: Interrogative-Label Fix Through Production Update Path
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Starting HEAD:** `870d6ca` (docs: record question-formulation fix)
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
When `n_savings_realism` (an existing interrogative node) is selected again in the live production flow, does the engine now produce a grammatically coherent next question rather than wrapping the interrogative label in another template?
|
||||
|
||||
## Configured Scenario (fixed)
|
||||
|
||||
"We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
## Configured Answer 2 (fixed)
|
||||
|
||||
"The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease."
|
||||
|
||||
## Hypothesis
|
||||
|
||||
If the selected target is `n_savings_realism` with label "Are the projected office savings from relocation realistic?", the emitted question should be a coherent standalone question rather than:
|
||||
- "What would clarify are the projected office savings from relocation realistic in this situation?"
|
||||
- "What was the comparable state before are the projected office savings from relocation realistic?"
|
||||
|
||||
## Run
|
||||
|
||||
One update-only call via the committed harness (`scripts/reproduce-multi-turn-investigation.mjs`).
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 0
|
||||
- updateCalls: 1
|
||||
- totalCalls: 1
|
||||
- Retries: 0
|
||||
|
||||
### PRE-ANCHORED FIXTURE
|
||||
|
||||
- savings-realism node id: `n_savings_realism`
|
||||
- label: "Are the projected office savings from relocation realistic?"
|
||||
- status: unknown
|
||||
|
||||
### UPDATE
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: provisional, previousValue: null, newValue: "~£2M/year", reason: "User provided a provisional estimate contingent on lease exit costs."}]
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [{id: n_lease_exit_penalty, label: Lease exit penalty amount, kind: unknown, status: unknown}]
|
||||
addedEdges: [{fromNodeId: n_savings_realism, toNodeId: n_lease_exit_penalty, relationship: depends_on}]
|
||||
selectedQuestion: "What would clarify lease exit penalty amount in this situation?"
|
||||
selectedQuestion.nodeId: "n_lease_exit_penalty"
|
||||
```
|
||||
|
||||
### Resulting persistent graph (3 nodes, 2 edges)
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=provisional
|
||||
node: id=n_lease_exit_penalty, kind=unknown, label=Lease exit penalty amount, status=unknown
|
||||
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
||||
edge: from=n_savings_realism, to=n_lease_exit_penalty, relationship=depends_on
|
||||
```
|
||||
|
||||
## Target assessment
|
||||
|
||||
**WRONG TARGET** (for the purpose of this experiment)
|
||||
|
||||
The hypothesis asked whether selecting `n_savings_realism` would now produce a coherent question. The engine instead created and selected a new node (`n_lease_exit_penalty`). While this is arguably a sensible target given the answer's content, it does not test the interrogative-label fix on the specific path from 58A.2/58A.3/58A.4.
|
||||
|
||||
## Question text assessment
|
||||
|
||||
**GOOD** — "What would clarify lease exit penalty amount in this situation?" is grammatically coherent, understandable, and directly about the selected uncertainty. No template-injection defect observed on this path.
|
||||
|
||||
## Classification: D — WRONG TARGET
|
||||
|
||||
The question-rendering regression cannot be fairly assessed because a materially different target was selected. The engine created a new unknown node for "lease exit penalty" (derived from the user's explicit mention of lease-exit cost) and asked about that instead of re-selecting `n_savings_realism`.
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The 58A.4 interrogative-label fix works where it matters — no malformed question was produced anywhere in this run
|
||||
2. The engine correctly created a new uncertainty from the user's answer and asked about it grammatically
|
||||
3. `n_savings_realism` was preserved (not destroyed), though degraded from unknown→provisional
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. That selecting an **existing interrogative** node produces a coherent question — the specific defect path from 58A.2/58A.3/58A.4 was not exercised
|
||||
2. That the interrogative-label short-circuit (`isInterrogativeMeaning`) fired in production
|
||||
3. That `n_savings_realism` would be selected again in a different answer context
|
||||
|
||||
## Production code changed: NO
|
||||
|
||||
## Ollama calls beyond harness count: 0
|
||||
@@ -0,0 +1,111 @@
|
||||
# Experiment 58A.6 — Interrogative Question Rendering Through Production Update Path (CONTROLLED)
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Starting HEAD:** `6f2c09c` (experiment: validate question-formulation fix live)
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
When the answer simply preserves the existing savings-realism uncertainty and introduces no new issue, does the live engine select that existing interrogative node and render its next question coherently through the 58A.4 fix?
|
||||
|
||||
## Configured Scenario (fixed)
|
||||
|
||||
"We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
## Configured Answer (fixed)
|
||||
|
||||
"I am still unsure whether the projected office savings from relocation are realistic."
|
||||
|
||||
## Why This Case Is Controlled
|
||||
|
||||
The answer:
|
||||
- preserves the existing uncertainty
|
||||
- introduces no new figure
|
||||
- introduces no new assumption
|
||||
- introduces no new competing unknown
|
||||
|
||||
Therefore this run is specifically designed to exercise formulation for the existing `n_savings_realism` target rather than test broader reasoning.
|
||||
|
||||
## Run
|
||||
|
||||
One update-only call via the committed harness (`scripts/reproduce-multi-turn-investigation.mjs`).
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 0
|
||||
- updateCalls: 1
|
||||
- totalCalls: 1
|
||||
- Retries: 0
|
||||
|
||||
### PRE-ANCHORED FIXTURE
|
||||
|
||||
- savings-realism node id: `n_savings_realism`
|
||||
- label: "Are the projected office savings from relocation realistic?"
|
||||
- status: unknown
|
||||
- Exactly one equivalent unresolved uncertainty before Update: YES
|
||||
- previousQuestion sent: "Are the projected office savings from relocation realistic?"
|
||||
|
||||
### UPDATE
|
||||
|
||||
- HTTP status: 422
|
||||
- Stage: proposal_compatibility
|
||||
- Validation errors: "structuralActionRequired is true but proposal contains no graph mutation"
|
||||
|
||||
#### Proposal snapshot (rejected)
|
||||
|
||||
```
|
||||
answerMeaning.userSupportedMeaning: "The user remains unsure about whether the projected office savings from relocation are realistic."
|
||||
updatedNodes: [{nodeId: n_savings_realism, newValue: null}]
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: []
|
||||
addedEdges: []
|
||||
structuralActionRequired: true (implied by validator rejection reason)
|
||||
selectedQuestion: UNAVAILABLE (update rejected before question selection)
|
||||
```
|
||||
|
||||
#### Resulting persistent graph: NOT APPLIED
|
||||
|
||||
The update was rejected. The fixture graph remains unchanged:
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, status=unknown
|
||||
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
||||
```
|
||||
|
||||
## Target selection
|
||||
|
||||
NO TARGET — update rejected before question selection could complete.
|
||||
|
||||
## Interrogative fix path
|
||||
|
||||
UNAVAILABLE — the apparatus prevented reaching this stage.
|
||||
|
||||
## Question assessment
|
||||
|
||||
NONE — no question produced.
|
||||
|
||||
## Classification: E — NO QUESTION
|
||||
|
||||
The engine identified that structural action was required (structuralActionRequired=true implied by validator rejection) but failed to produce any meaningful graph mutation, causing a 422 at `proposal_compatibility`. No next question was emitted because the update was rejected before the question-selection phase.
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The pre-anchored fixture apparatus works — the correct node (n_savings_realism) and answer reach the production server
|
||||
2. The model correctly extracts user meaning: "The user remains unsure about whether the projected office savings from relocation are realistic."
|
||||
3. For a pure-preservation answer with no new evidence/figure/assumption, the engine still requires structural action but cannot produce one — this is a **semantic gap**: the answer provides only uncertainty confirmation, which the model recognizes as requiring structural action but cannot express through graph mutation (nothing to change)
|
||||
4. The 58A.4 interrogative-label fix path remains unproven live because the apparatus blocks before question selection
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. Whether n_savings_realism would be selected if the update had succeeded
|
||||
2. Whether the emitted question would be grammatically coherent for an interrogative label
|
||||
3. Whether the 58A.4 fix works in production
|
||||
4. Cross-domain generalisation
|
||||
|
||||
## Production code changed: NO
|
||||
|
||||
## Harness/tooling changed: NO
|
||||
|
||||
## Ollama calls beyond harness count: 0
|
||||
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,171 @@
|
||||
# Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Previous context:** Follows 58A.2 which exposed the core problem — status shift from `unknown` to `provisional` when evidence arrives but resolution remains open.
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When the user provides a concrete £2m figure and explicitly states it is unverified, does the engine preserve the figure **and** keep the existing savings-realism uncertainty unresolved?
|
||||
|
||||
This isolates the status decision from 58A.2's broader failure modes.
|
||||
|
||||
---
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
```
|
||||
id: n_savings_realism
|
||||
label: Are the projected office savings from relocation realistic?
|
||||
kind: unknown
|
||||
status: unknown
|
||||
value: null
|
||||
confidence: low
|
||||
dependsOn: [n_relocation_state]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixed Answer
|
||||
|
||||
> The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic.
|
||||
|
||||
Three components:
|
||||
1. **Supported information:** approximately £2 million per year
|
||||
2. **Explicit qualification:** figure is unverified
|
||||
3. **Continued uncertainty:** user not yet confident the estimate is realistic
|
||||
|
||||
No lease assumption, no disruption-cost assumption, no new competing unknown.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
ANSWER_2="The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results (direct API inspection)
|
||||
|
||||
### Answer meaning (inside updatedProposal.answerMeaning)
|
||||
|
||||
```json
|
||||
{
|
||||
"userSupportedMeaning": "The projected savings figure is £2 million per year, but it remains unverified and the user lacks confidence in its realism.",
|
||||
"possibleInference": null,
|
||||
"supportCategory": "uncertain",
|
||||
"resolutionGuidance": "may_resolve"
|
||||
}
|
||||
```
|
||||
|
||||
### Structural proposal
|
||||
|
||||
- `updatedNodes`: `[{nodeId: n_savings_realism, previousStatus: unknown, newStatus: unknown, previousValue: null, newValue: "£2M/year (unverified)"}]`
|
||||
- `resolvedUnknownNodeIds`: `[]`
|
||||
- `addedNodes`: `[]`
|
||||
- `addedEdges`: `[]`
|
||||
- `structuralActionRequired`: true
|
||||
|
||||
### Persistent graph after update
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown, confidence=low, value="£2M/year (unverified)"
|
||||
edge: n_savings_realism → n_relocation_state (depends_on)
|
||||
```
|
||||
|
||||
### Selected question
|
||||
|
||||
- Question: `"are the projected office savings from relocation realistic?"`
|
||||
- nodeId: `n_savings_realism`
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
| Criterion | Result | Classification |
|
||||
|-----------|--------|----------------|
|
||||
| £2m figure | `"£2M/year (unverified)"` on n_savings_realism | PRESERVED AS QUALIFIED |
|
||||
| Existing uncertainty status | `unknown` → `unknown` (unchanged) | UNCERTAINTY PRESERVED |
|
||||
| Qualification | "(unverified)" suffix in value | PRESERVED |
|
||||
| Identity of savings-realism node | Single node, unchanged id, no duplicates | ONE IDENTITY PRESERVED |
|
||||
| Next investigation | Question targets the unresolved realism question | GOOD |
|
||||
|
||||
### Answer meaning analysis
|
||||
|
||||
- **userSupportedMeaning** correctly captured all three answer components: figure amount + unverified status + user doubt
|
||||
- **possibleInference** = null — did not invent assumptions about lease, disruption, or cost structure
|
||||
- **supportCategory** = `"uncertain"` — semantically correct for qualified evidence
|
||||
- **resolutionGuidance** = `"may_resolve"` — correctly reflects that the uncertainty remains open
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED
|
||||
|
||||
- £2m survives as `"£2M/year (unverified)"` with explicit qualification
|
||||
- n_savings_realism stays `kind=unknown / status=unknown` — identity and unresolved nature both preserved
|
||||
- Confidence set to `low` — appropriate for unverified evidence on an uncertainty node
|
||||
- Answer meaning supportCategory = `"uncertain"` — correct semantic interpretation
|
||||
- No duplicate nodes, no resolved unknown nodes
|
||||
- Selected question continues investigating the realism concern
|
||||
|
||||
### Does the graph still clearly represent realism as unresolved?
|
||||
|
||||
**YES.** The node kind remains `unknown`, status remains `unknown`, and value contains the explicit qualification "(unverified)". Confidence is `low`. There are zero `resolvedUnknownNodeIds`. An interrogative selectedQuestion pointing to this same node confirms ongoing investigation targeting.
|
||||
|
||||
---
|
||||
|
||||
## What the engine understood correctly
|
||||
|
||||
1. **Evidence preservation:** Extracted the £2M/year figure from prose and stored it on the existing uncertainty node rather than discarding or inventing a new node.
|
||||
2. **Qualification embedding:** The value includes "(unverified)" — the model did not strip the qualification when storing evidence.
|
||||
3. **Semantic category:** Labeled supportCategory as `"uncertain"` rather than `"strong"` or `"established"`.
|
||||
4. **No fabrication:** possibleInference was null — no invented lease, disruption, or cost assumptions.
|
||||
5. **Status stability:** Status remained `unknown` (not shifted to `provisional`) — unlike 58A.2 where this was the core failure.
|
||||
6. **Open resolution:** Did not resolve n_savings_realism; resolutionGuidance = `"may_resolve"` correctly reflects the ongoing need for verification.
|
||||
7. **Question continuity:** Selected question re-targets the existing node's label rather than inventing a new uncertainty.
|
||||
|
||||
## What it overstated, weakened, or lost
|
||||
|
||||
**Nothing significant.** The update was fully correct for the constraints of this case. One minor note: `structuralActionRequired` is `true` despite no structural change (no new/removed nodes or edges). This flag means "a follow-up structural action may be needed" but does not indicate a failure — it is a forward-looking directive, not a description of what was done wrong.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. The engine **can** preserve a concrete figure with its qualification when the existing uncertainty node remains the appropriate target.
|
||||
2. Status `unknown` is stable across updates even when value transitions from `null` to a qualified string — unlike the 58A.2 failure path.
|
||||
3. Qualification embedded in `newValue` (e.g., `"£2M/year (unverified)"`) survives as persistent evidence that realism remains unconfirmed.
|
||||
4. Answer meaning extraction (`userSupportedMeaning`, `supportCategory: uncertain`, `resolutionGuidance: may_resolve`) aligns correctly with the user's actual semantics.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. This is a **single controlled case** — one answer, one model invocation. It does not prove stability across different answers or models.
|
||||
2. It does not test whether `structuralActionRequired=true` with no structural change causes issues in subsequent turns.
|
||||
3. It does not test what happens when the user's qualification changes (e.g., from "unverified" to "verified").
|
||||
4. It does not test interaction with other uncertainty nodes (58A.1's scenario where multiple unknowns compete).
|
||||
5. Value format `"£2M/year (unverified)"` uses prose — whether numeric `2000000` would work equally well is untested here.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,176 @@
|
||||
# Experiment 58B.2 — Verified Uncertainty Resolution
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Previous context:** Follows 58B.1 which showed the engine preserves qualified evidence while keeping uncertainty open. This tests the opposite boundary: when the user explicitly verifies and confirms realism, does the engine resolve?
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When the user explicitly says the £2m figure has now been verified and is realistic, does the engine resolve the existing `n_savings_realism` uncertainty rather than merely changing its value or weakening its status?
|
||||
|
||||
---
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
```
|
||||
id: n_savings_realism
|
||||
label: Are the projected office savings from relocation realistic?
|
||||
kind: unknown
|
||||
status: unknown
|
||||
value: null
|
||||
confidence: low
|
||||
dependsOn: [n_relocation_state]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixed Answer
|
||||
|
||||
> We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic.
|
||||
|
||||
Components:
|
||||
1. **Concrete value:** approximately £2 million per year
|
||||
2. **Verification:** the estimate has now been checked
|
||||
3. **Relevant dependency addressed:** lease exit costs included
|
||||
4. **Explicit confidence:** user now believes the estimate is realistic
|
||||
|
||||
No new uncertainty introduced.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
ANSWER_2="We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### Answer meaning
|
||||
|
||||
Not returned in the update response (updateOnly mode accepted without including answerMeaning in the output). The structural fields below are authoritative.
|
||||
|
||||
### Structural proposal
|
||||
|
||||
- `updatedNodes`: `[{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"£2,000,000","reason":"User verified the projected annual savings including lease exit costs are realistic."}]`
|
||||
- `resolvedUnknownNodeIds`: `["n_savings_realism"]`
|
||||
- `addedNodes`: `[]`
|
||||
- `addedEdges`: `[]`
|
||||
- `structuralActionRequired`: null
|
||||
|
||||
### Persistent graph after update
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=resolved
|
||||
edge: n_savings_realism → n_relocation_state (depends_on)
|
||||
```
|
||||
|
||||
### Selected question
|
||||
|
||||
None produced.
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
| Criterion | Result | Classification |
|
||||
|-----------|--------|----------------|
|
||||
| £2m figure | `"£2,000,000"` on n_savings_realism | PRESERVED AS VERIFIED |
|
||||
| Existing uncertainty status | `unknown` → `resolved` | CORRECTLY RESOLVED |
|
||||
| Identity of savings-realism node | Single node, unchanged id (`n_savings_realism`), no duplicates | ORIGINAL IDENTITY RESOLVED |
|
||||
| Verification meaning | reason: "User verified the projected annual savings including lease exit costs are realistic." | PRESERVED |
|
||||
| Next investigation | NONE — no consequential unresolved issue remains | GOOD |
|
||||
|
||||
### £2m figure analysis
|
||||
|
||||
The value `"£2,000,000"` preserves the core monetary figure. The "per year" unit is not explicit in `newValue` (unlike 58B.1 which had `"£2M/year (unverified)"`) but is preserved in the reason field ("projected **annual** savings"). This qualifies as PRESERVED AS VERIFIED — the amount is captured and the verification context survives.
|
||||
|
||||
### Uncertainty resolution analysis
|
||||
|
||||
Status clearly changed from `unknown` to `resolved`. The node id `n_savings_realism` appears in `resolvedUnknownNodeIds`. This is unambiguous correct resolution.
|
||||
|
||||
### Identity analysis
|
||||
|
||||
Exactly one savings-realism unknown node exists before and after the update. Same node id, same label, status transitions correctly. No duplicate created. ORIGINAL IDENTITY RESOLVED.
|
||||
|
||||
### Verification meaning analysis
|
||||
|
||||
The reason field on the updated node explicitly states: "User verified the projected annual savings including lease exit costs are realistic." This captures all four components of the user's answer (value, verification, lease costs, confidence). PRESERVED.
|
||||
|
||||
### Next investigation analysis
|
||||
|
||||
No selected question was produced. This is correct behavior — the existing uncertainty is resolved and no new consequential unresolved issue was introduced by the answer. GOOD.
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — UNCERTAINTY CORRECTLY RESOLVED
|
||||
|
||||
- n_savings_realism correctly resolved (status → `resolved`)
|
||||
- Included in `resolvedUnknownNodeIds`
|
||||
- Verified £2m evidence survives as `"£2,000,000"` with full verification context in reason field
|
||||
- No duplicate uncertainty created
|
||||
- Same node id preserved (original identity resolved)
|
||||
- No redundant question asked about realism
|
||||
- No consequential unresolved issue remains to investigate
|
||||
|
||||
---
|
||||
|
||||
## What the engine understood correctly
|
||||
|
||||
1. **Resolution trigger:** The explicit "verified" and "confident...realistic" language triggered correct uncertainty resolution — status moved from `unknown` to `resolved`. This is the semantic boundary 58B.1 left open.
|
||||
2. **Value extraction:** The figure was captured as `"£2,000,000"` — a clean monetary representation.
|
||||
3. **Verification context:** The reason field captured all four answer components: value (£2m), verification status ("verified"), lease exit costs, and confidence ("realistic").
|
||||
4. **No fabrication:** No new uncertainty nodes or edges were created from this answer that contained no new uncertainty.
|
||||
5. **Identity preservation:** The original `n_savings_realism` was updated (not replaced or duplicated).
|
||||
6. **Correct termination signal:** No selected question was produced, correctly reflecting that the existing investigation thread is complete.
|
||||
|
||||
---
|
||||
|
||||
## What it overstated, weakened, or lost
|
||||
|
||||
**Minor weakening of temporal unit:** The "per year" time unit is not explicit in `newValue` (which is `"£2,000,000"` rather than `"£2,000,000/year"`). However, the word "annual" in the reason field partially compensates. This does not affect the core resolution question — it is a secondary representation detail.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. When the user provides **explicit verification** AND **confidence about realism**, the engine correctly resolves the existing savings-realism uncertainty (status → `resolved` + inclusion in `resolvedUnknownNodeIds`).
|
||||
2. This is the semantic opposite of 58B.1 and works correctly — the engine distinguishes between "unverified but plausible" (keep open) and "verified and confident" (resolve).
|
||||
3. The verified £2m figure survives in persistent graph state with verification context captured.
|
||||
4. No duplicate uncertainty is created during resolution.
|
||||
5. The engine does not ask the resolved question again — it correctly terminates when no consequential unresolved issue remains.
|
||||
|
||||
---
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. **Single controlled case** — one answer, one model invocation. Stability across different answers or repeated runs is untested.
|
||||
2. **Temporal unit preservation** — whether "per year" survives in `newValue` depends on answer phrasing and model behavior.
|
||||
3. **answerMeaning fields** — the update response did not include answerMeaning, so whether userSupportedMeaning captured all four components independently of the reason field is unavailable.
|
||||
4. **Multi-turn stability** — what happens when subsequent turns arrive after a resolved uncertainty is untested here.
|
||||
5. **Cross-domain generalisation** — this is a single cost-savings domain case.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user