From 7dd4a956fbfc32ad268c9a1620e6be2378a32669 Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 18 Aug 2026 06:32:40 +0100 Subject: [PATCH] experiment: validate live financial investigation progression --- docs/current-handoff.md | 11 ++++++ docs/experiment-60b97.md | 80 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) create mode 100644 docs/experiment-60b97.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 8ccbc56..50399b2 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -68,3 +68,14 @@ The live model selected nk6eyn2 ("exact monetary value of potential enterprise c **What this establishes:** The live engine can produce an acceptable graph-backed question on a fresh product-launch start without requiring decomposition. **What this does NOT prove:** Whether investigation ownership is preserved when a selected target's formulation is rejected (the core invariant from checkpoint 60B.93). The question-rejection boundary was not reached because the live model chose a different investigation target with an acceptable question path. + + +## Experiment 60B.97 result (2026-08-18) + +**Classification: E — START PATH DIVERGED** + +The live model again selected a non-financial-comparison target on the product-launch scenario. The Start selected enterprise-customer signing probability ("What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?") rather than the expected cash-flow / NPV comparison. Update 1 then hit a result_validation error: "Active unknown violates reasoning pattern consistency: \"n_fincomp12\" is explanation but active pattern is decision." + +**What this establishes:** (1) The divergence from financial-comparison targets continues across multiple runs; (2) there exists an internal graph-state inconsistency (explanation-node vs decision-pattern mismatch in the Start/Update pipeline). + +**What this does NOT prove:** Whether a financially-comparison-aligned Start would progress coherently with a financial answer, or whether the pattern consistency issue is deterministic. diff --git a/docs/experiment-60b97.md b/docs/experiment-60b97.md new file mode 100644 index 0000000..4a1429f --- /dev/null +++ b/docs/experiment-60b97.md @@ -0,0 +1,80 @@ +# Experiment 60B.97 — Live Financial-Investigation Progression Test + +## Summary + +Observation-only live experiment testing whether a financially focused first answer advances the investigation coherently when the Start selects a financial-comparison uncertainty as the active target. + +## Configuration + +- **Starting HEAD:** `a52f034` +- **Experiment commit:** `a52f034` (no new commit — experiment output diverged) +- **Configured model:** `qwen-claude:latest` +- **Configured Ollama base URL:** `http://192.168.1.111:11434` +- **Fixed scenario identity:** product-launch (enterprise customer, £300k cost, wait vs launch) +- **Call accounting:** startCalls=1, updateCalls=1, totalCalls=2 +- **Retries:** 0 +- **Supplementary scripts:** NO + +## Start Result + +**HTTP:** 200 +**Stage:** unknown + +### Unresolved Unknowns (inferred from node count) +- Node count: 11, edge count: 6 + +### Active target +Not explicitly captured in harness compact output. Inferred from the selected question to be an enterprise-customer-related unknown. + +### Selected question +"What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?" + +### Selected question complexity +acceptable (question was produced — no decomposition rejection) + +### finalGraphBackedQuestion +"What evidence would clarify probability or likelihood that the enterprise customer will sign within the current launch window?" + +## Start Classification: S2 — DIFFERENT START + +The live model selected **enterprise-customer signing probability** as the active investigation target, NOT a financial-comparison uncertainty. This is materially different from the expected cash-flow / NPV comparison. + +This divergence is consistent with experiment 60B.95 which also diverged (to monetary valuation). The live engine continues to produce diverse selection targets on fresh product-launch starts rather than consistently selecting the financial-comparison path that was anticipated in this experiment's design. + +## Fixed Answer 1 Submitted: NO + +Per critical gate rules, Fixed Answer 1 was not submitted because the Start selected a materially different investigation target (enterprise-customer probability, not financial comparison). + +## Update 1 Result + +**Reached:** YES + +**HTTP:** 500 +**Stage:** result_validation +**First error:** `"Active unknown violates reasoning pattern consistency: \"n_fincomp12\" is explanation but active pattern is decision"` + +This error indicates an internal graph-state inconsistency: node `n_fincomp12` was classified as an "explanation" type node but the active reasoning pattern requires a "decision" pattern. The Start path produced a situation graph with this structural violation. + +## Classification: E — START PATH DIVERGED + +The live engine selected enterprise-customer signing probability (S2) rather than a financial-comparison uncertainty, so Fixed Answer 1 was not submitted. Additionally, Update 1 hit a result_validation error indicating a reasoning-pattern consistency violation in the graph state. + +### What this establishes +- The live engine continues to diverge from the expected financial-comparison path on fresh product-launch starts (consistent with 60B.95 pattern). +- A graph-state inconsistency exists: `n_fincomp12` (explanation) is active while the reasoning pattern is decision — a structural validity issue in the Start/Update pipeline. + +### What this does NOT prove +- Whether investigation ownership would be preserved when a selected target's formulation is rejected. +- Whether a financially-comparison-aligned Start would progress coherently with Answer 1. +- Whether the explanation-vs-decision pattern inconsistency is deterministic or intermittent. + +## Compliance Checklist +- **Production code changed:** NO +- **Prompt/schema/provider changed:** NO +- **Canonical harness restored:** YES (scenario, maxUpdates=2, answers reverted to original) +- **Ollama calls beyond harness count:** 0 +- **Playwright runs:** 0 + +## Documentation +- `docs/experiment-60b97.md` — created (this file) +- `docs/current-handoff.md` — appended experiment result entry