Feature/product platform foundation v0.62 #1

Merged
robbond merged 683 commits from feature/product-platform-foundation-v0.62 into feature/emergent-unknowns-v0.5 2026-09-09 07:58:20 +01:00
2 changed files with 148 additions and 0 deletions
Showing only changes of commit b1914f5da7 - Show all commits
+8
View File
@@ -529,6 +529,14 @@ Scenario, answers, and maxUpdates restored to canonical defaults before commit.
---
### Experiment 58A.3 — Next-Question Formulation Quality on Correct Target
**Objective:** When the existing savings-realism uncertainty is the correct next target, does the engine formulate a coherent, useful question for it?
**Classification: C — TARGET GOOD, QUESTION MALFORMED.** One update-only call via the committed harness. The engine selected `n_savings_realism` as the next-question target — this is a good target choice given the existing state. However, the question text "What would clarify are the projected office savings from relocation realistic in this situation?" is grammatically broken: it fuses "What would clarify [X] in this situation?" with the full unknown label "are the projected office savings from relocation realistic", producing an ungrammatical sentence that cannot be answered. **The exact same malformed-template failure from 58A.2 persists.** Additionally, a new regression appeared relative to 58A.2: the model changed n_savings_realism status from `unknown``weakened` (instead of 58A.2's `provisional`), setting newValue=null — meaning the £2m figure was not even carried forward in this run (58A.2 preserved it as "~£2,000,000/year (pending lease exit cost)"). So the question-formulation defect persists and an evidence-preservation regression appeared simultaneously. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
---
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.
+140
View File
@@ -0,0 +1,140 @@
# Experiment 58A.3 — Next-Question Formulation Quality on Correct Target
**Branch:** `feature/semantic-action-contract-v0.23`
**Starting HEAD:** `20e4b58d7619bceee42a182f84c5265f0eab5841`
**Experiment commit:** pending
## Objective
Can the engine ask a coherent next question when the right target is already known?
58A.2 showed that the engine could preserve the £2m figure, preserve the lease-exit uncertainty, and choose the correct investigation target — but the selected question text was malformed (template injection failure).
This experiment tests that one thing only: **question formulation quality** when `n_savings_realism` is the selected target.
## Fixed scenario (from fixture)
"We are considering relocating the engineering team to reduce operating costs."
Existing unresolved uncertainty: "Are the projected office savings from relocation realistic?"
## Fixed answer
> The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease.
## Configured model: qwen-claude:latest at http://192.168.1.111:11434
### CALL ACCOUNTING
- startCalls: 0
- updateCalls: 1
- totalCalls: 1
- Retries: 0
### UPDATE RESULT
- HTTP status: 200
- Stage: update_applied
- Validation errors: none
- structuralActionRequired: null (known gap from 57J.69)
#### Proposal mutations
```
updatedNodes: [
{nodeId: n_savings_realism, previousStatus: unknown, newStatus: weakened,
previousValue: null, newValue: null,
reason: "The answer explicitly withholds trust in the £2m projection until the lease exit cost is known, weakening current confidence."}
]
resolvedUnknownNodeIds: []
addedNodes: [
{id: n_lease_penalty, label: "Office lease exit penalty amount",
description: "Total cost to break the current London office lease, needed to decide if the £2m/year relocation savings are realistic.",
kind: unknown, status: unknown, confidence: low}
]
addedEdges: [
{fromNodeId: n_savings_realism, toNodeId: n_lease_penalty, relationship: depends_on, confidence: medium,
description: "Assessing savings realism depends on determining the lease exit penalty."}
]
selectedQuestion.nodeId: "n_savings_realism"
```
#### Selected question
> "What would clarify are the projected office savings from relocation realistic in this situation?"
> nodeId: n_savings_realism
(Note: question text is grammatically broken — same template injection failure as 58A.2.)
### Resulting persistent graph (3 nodes, 2 edges)
```
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=weakened
node: id=n_lease_penalty, kind=unknown, label=Office lease exit penalty amount, status=unknown
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
edge: from=n_savings_realism, to=n_lease_penalty, relationship=depends_on
```
### Reasoning assessment
**Target selection:** n_savings_realism is a GOOD TARGET. It is the existing savings-realism uncertainty that was already present and remains unresolved. The engine correctly chose it as the next investigation focus.
**Question text quality:** MALFORMED. The sentence "What would clarify are the projected office savings from relocation realistic in this situation?" fuses the template frame "What would clarify [X] in this situation?" with the full unknown label "are the projected office savings from relocation realistic", producing an ungrammatical sentence. A grammatically correct version would read: "What would clarify whether the projected office savings from relocation are realistic in this situation?" or simply "Are the projected office savings from relocation realistic in this situation?"
**£2m figure preservation:** LOST (relative to 58A.2). The updatedNode for n_savings_realism has newValue=null — the £2m figure was not carried forward at all in this run. In 58A.2, it survived as "~£2,000,000/year (pending lease exit cost)". The status changed to `weakened` instead of 58A.2's `provisional`, which signals a different reasoning pattern but equally loses the evidence.
**Lease-exit uncertainty:** STRUCTURALLY REPRESENTED — dedicated unknown node `n_lease_penalty` with clear description referencing the £2m/year savings context, plus a depends_on edge from n_savings_realism to it. This matches 58A.2's pattern.
**Original savings-realism identity status:** CHANGED from `unknown``weakened`. The status `weakened` (rather than 58A.2's `provisional`) signals that the model interpreted the user's doubt about the £2m figure as a reason to downgrade confidence in the uncertainty itself, rather than preserving it as an active investigation target. This is arguably correct reasoning (the user expressed distrust) but structurally the node no longer functions as "unresolved — needs evidence" since `weakened` has different downstream semantics than `unknown`.
### Target assessment: GOOD TARGET
n_savings_realism is the correct next investigation target given the existing state. It was already unresolved, and the user's answer directly qualified its supporting evidence.
### Question text assessment: MALFORMED
The question fuses a template frame with an unknown label into ungrammatical output. This is the same class of defect as 58A.2.
### Classification: C — TARGET GOOD, QUESTION MALFORMED
Correct target selection, broken question text. The root cause remains in the question-formulation pipeline's handling of existing-node targets (not newly-created ones).
### Why:
The model correctly identified n_savings_realism as the next investigation focus. However, the question formulation template system produced a grammatically broken sentence by inserting the full unknown label directly into a template without respecting English grammar boundaries.
### Was the selected target sensible: YES
### Was the question grammatically coherent: NO
### Would answering it materially reduce the selected uncertainty: DEBATABLE — The question is broken, so no answer could be reliably given to it. A corrected version ("What would clarify whether the projected office savings from relocation are realistic?") would directly address the uncertainty.
### Did the malformed-template failure from 58A.2 recur: YES
The exact same class of template injection failure persisted across two independent runs. The question text is nearly identical between 58A.2 and 58A.3:
- 58A.2: "What was the comparable state before are the projected office savings from relocation realistic?"
- 58A.3: "What would clarify are the projected office savings from relocation realistic in this situation?"
Both fuse a template frame with the unknown label, producing ungrammatical output. The specific template variant differs but the defect mechanism is identical.
### What this establishes:
1. The malformed-template question-formulation defect is **reproducible** — it is not a cold-start anomaly from 58A.2
2. The defect persists across different model call contexts (subsequent to a graph update)
3. Target selection logic correctly chose n_savings_realism in this run
4. A new regression appeared: the £2m figure was not preserved in newValue (null), unlike 58A.2 where it survived as qualified text
5. The status change `unknown``weakened` (this run) vs `unknown``provisional` (58A.2) shows run-to-run variance in the model's assessment semantics
### What this does NOT prove:
- Whether a question-formulation fix would work for newly-created unknown nodes (only existing-node targeting was tested)
- Whether different target nodes produce different template behavior
- Cross-domain generalisation
- Whether the £2m-loss regression is stable or 58A.2 was the lucky run
### Production code changed: NO
### Prompt changed during experiment: NO
### Harness/tooling changed: NO
### Ollama calls beyond harness count: 0
### Dev server disturbed: NO