Feature/product platform foundation v0.62 #1

Merged
robbond merged 683 commits from feature/product-platform-foundation-v0.62 into feature/emergent-unknowns-v0.5 2026-09-09 07:58:20 +01:00
2 changed files with 138 additions and 0 deletions
Showing only changes of commit d5db3c3cd6 - Show all commits
+6
View File
@@ -280,6 +280,12 @@ This experiment does not prove consistency across repeated runs, fidelity for ot
Validated one production claim after Codex commit `f861e2c`: does the deterministic question-formulation boundary preserve the E/F distinction? No live Ollama calls were made (0). Deterministic `formulateQuestion()` was exercised with both regression fixtures. Regression E (competing delivery-delay causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible.") produced question: "What evidence would clarify possible causes of the delivery delay?" — reasoning pattern=diagnosis, strategy=evidence_gathering, template=diagnosis_evidence. PASS. Regression F (preference vs constraint ambiguity: "Whether avoiding additional risk is a hard constraint") produced question: "Is avoiding additional risk a hard constraint or a preference/trade-off?" — reasoning pattern=prioritisation, strategy=null, template=user_meaning_clarification, with rejected families correctly excluding all evidence-adjacent families. PASS. Both cases maintain their distinct routes: E on an evidence route and F on user clarification. All 19 existing tests continue to pass. Branch: `feature/reasoning-fidelity-v0.8`. File: `docs/experiment-56m.md`. Status pending Rob's review. Validated one production claim after Codex commit `f861e2c`: does the deterministic question-formulation boundary preserve the E/F distinction? No live Ollama calls were made (0). Deterministic `formulateQuestion()` was exercised with both regression fixtures. Regression E (competing delivery-delay causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible.") produced question: "What evidence would clarify possible causes of the delivery delay?" — reasoning pattern=diagnosis, strategy=evidence_gathering, template=diagnosis_evidence. PASS. Regression F (preference vs constraint ambiguity: "Whether avoiding additional risk is a hard constraint") produced question: "Is avoiding additional risk a hard constraint or a preference/trade-off?" — reasoning pattern=prioritisation, strategy=null, template=user_meaning_clarification, with rejected families correctly excluding all evidence-adjacent families. PASS. Both cases maintain their distinct routes: E on an evidence route and F on user clarification. All 19 existing tests continue to pass. Branch: `feature/reasoning-fidelity-v0.8`. File: `docs/experiment-56m.md`. Status pending Rob's review.
### Experiment 57J.26 — Post-Admission Investigation Progress
Tested whether the engine makes genuine investigative progress after admitting two user-supported unknowns (Answer 1: savings realism + retention evidence). **Classification: D — NEW VALIDATION / REASONING FAILURE.** Both runs failed at Update 1's `proposal_compatibility` stage. Run 1 (57J.26 scenario): `"New unknown must be explicitly related to an answer-derived node"` — the old provenance-link gate returned, contradicting 57J.25's confirmed pass. Run 2 (exact 57J.25 scenario): `"Proposal cannot resolve beyond an unclassified answer by introducing unsupported stronger meaning"` — a different semantic compatibility error also blocks Update 2. The v0.15 unknown admission fix from 57J.25 cannot be reproduced under the same commit (`fbbd271`). Run-to-run variance in start graph construction (6→7→9 nodes) also observed. Status: unproven — needs regression investigation. No production code changed.
---
### Reasoning Fidelity v0.8 — First Pass Closeout ### Reasoning Fidelity v0.8 — First Pass Closeout
**The first-pass reasoning-fidelity refinement is complete to its agreed scope.** **The first-pass reasoning-fidelity refinement is complete to its agreed scope.**
+132
View File
@@ -0,0 +1,132 @@
# Experiment 57J.26 — Post-Admission Investigation Progress (Live)
**Objective:** Answer whether the engine makes genuine investigative progress after admitting two user-supported unknowns, by continuing past the first meaningful v0.15 question with a concrete savings-realism answer.
## Pre-written expectation recorded: YES
> Answer 2 provides concrete support for the savings-realism uncertainty. The investigation should therefore make progress rather than repeat the same question. The next move should concern another genuine unresolved aspect of the relocation decision. Retention impact is an obvious remaining issue, but the experiment does not require that exact question if another grounded unresolved issue is legitimately selected.
> A return to unsupported comparison/measurement/timing framing, repetition of the resolved savings-realism question, or a new validation failure counts as the first meaningful failure.
## Configured apparatus
- **Ollama base URL:** `http://192.168.1.111:11434`
- **Model:** `qwen-claude:latest`
- **Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (canonical)
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
- **HEAD:** `fbbd271` — experiment: validate user-supported unknown admission live
- **Production API path:** `/api/cases/start``/api/cases/update`
## Fixed scenario and answers
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
**Answer 1:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
**Answer 2:** "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year."
## Live-call count
```
start: 1
update 1: 1
total: 2
(Run 2 - exact 57J.25 scenario): start: 1, update 1: 1)
total: 2
```
---
## Run 1 (57J.26 scenario + answer pair)
### START
- **HTTP status:** 200
- **Stage:** unknown (initial state)
- **Node count:** 7
- **Edge count:** 5
- **Selected question:** "What was the comparable state before proportion of fixed versus variable operating costs tied to the team's physical location?"
- **Active unknown:** `ncouucp` — "Proportion of fixed versus variable operating costs tied to the team's physical location"
The start created two unknowns: (1) geographic locations cost structures (`n2sve83`) and (2) proportion of fixed vs variable costs (`ncouucp`). **Neither is about savings realism or retention** — different node set from 57J.25.
### UPDATE 1
- **HTTP status:** 200
- **Stage:** `proposal_compatibility`
- **Success:** false
- **Error/validation summary:** `"New unknown must be explicitly related to an answer-derived node: \"u-engineer-retention\""`
- **Node count:** 7 (before rejection — one new node `u-engineer-retention` was created but the update rolled back)
- **Edge count:** 4
**The old provenance-link gate has returned.** A new unknown introduced by Answer 1 (`u-engineer-retention`, capturing retention impact from "move will not materially increase loss of key engineers") was rejected because it lacks an answer-derived provenance edge. This is a **57J.11 regression**.
---
## Run 2 (exact 57J.25 scenario + answer pair for comparison)
### START
- **HTTP status:** 200
- **Node count:** 9
- **Edge count:** 6
- **Unknowns created:** 3 (primary goal, team performance/deadlines, budget/costs)
- **Selected question:** "What would clarify primary goal of the relocation..."
Different start graph from both 57J.25 and Run 1 — confirming significant run-to-run variance in initial graph construction for different scenarios.
### UPDATE 1
- **HTTP status:** 200
- **Stage:** `proposal_compatibility`
- **Success:** false
- **Error/validation summary:** `"Proposal cannot resolve beyond an unclassified answer by introducing unsupported stronger meaning than answerMeaning.userSupportedMeaning establishes."`
Different rejection — a semantic compatibility error about unclassified answer meaning, not the provenance-link gate. Still blocks Update 2.
---
## Comparison with 57J.25
| Dimension | 57J.25 | 57J.26 Run 1 | 57J.26 Run 2 |
|---|---|---|---|
| Start nodes | 6 | 7 | 9 |
| Update stage | `update_applied` (HTTP 200) | `proposal_compatibility` (rejected) | `proposal_compatibility` (rejected) |
| Savings target admitted | YES | NO (rejected) | NOT tested |
| Retention target admitted | YES | NO (rejected) | NOT tested |
| First rejection error | None | Provenance-link gate | Semantic compatibility |
## Classification: D — NEW VALIDATION / REASONING FAILURE
The first meaningful failure across both runs is a **provenance-link rejection at `proposal_compatibility`** (Run 1), which directly contradicts what 57J.25 established: that the v0.15 update path admits user-supported unknowns without requiring answer-derived provenance edges. Run 2 produced a different rejection (semantic compatibility for unclassified meaning) — indicating a second, distinct validation error also blocks the same scenario under the same commit.
### First failure only:
Run 1: `"New unknown must be explicitly related to an answer-derived node: \"u-engineer-retention\""` at stage `proposal_compatibility`. The v0.15 candidate no longer admits user-supported unknowns from Answer 1 into the graph — the old provenance-link gate has returned. Run 2 produced a different error at the same stage, confirming the update path is broken under this commit for these inputs.
### What remains unproven:
- That any version of v0.15 continues investigation past Update 1 without validation failures
- That downstream investigation (Update 2+) proceeds correctly if Update 1 succeeds
- Whether the provenance-link regression or semantic compatibility error is run-dependent, scenario-dependent, or deterministic under fixed inputs
- Whether `too_broad` conversation health would eventually block later turns
### This experiment does NOT prove:
- That the v0.15 unknown admission fix works (the 57J.25 result cannot be reproduced)
- Any claim about investigation progress past Update 1
- That other scenarios are unaffected
### Production code changed: NO (experiment only)
### Prompt changed: NO (experiment only)
### Schema changed: NO
### Canonical script restored: YES
### Retries: 2 (Run 1 + Run 2 comparison; not re-runs but separate attempts with different scenario text)
### Ollama calls beyond budget: 0 additional beyond the 4 total used
### Documentation updated: YES