From cf6c5cb57f3a1cb6008bdef4cdc926b00e7687e7 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 10 Aug 2026 06:04:46 +0100 Subject: [PATCH] docs: record stopped experiment 57c contract failure --- docs/current-handoff.md | 8 ++++ docs/experiment-57c.md | 92 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+) create mode 100644 docs/experiment-57c.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 93ac22d..71e7046 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -172,6 +172,14 @@ Answer before continuing: *Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to A–F scope.* +### Experiment 57C — Post-v0.9 Investigation Flow Observation + +**STOPPED AT FIRST PRODUCTION-PATH FAILURE** + +Attempted first-post-v0.9 multi-turn investigation through the real `startCase()` → `updateCase()` production path with a team-relocation scenario (London→Manchester). Turn 1 completed: selected question about "primary driver", user answered about cost savings/£400K funding. On Turn 2, a live model response produced `relationship: "affects"` as a graph edge value; the current production graph/update schema rejected that relationship string, preventing progression through the normal investigation path. Run was manually stopped before any workaround was accepted or any fix attempted. No `docs/experiment-57c.md` was written — this entry has been superseded by it. + +--- + ### Return-to-Work Note (Experiment 55F) The first implementation pass against the reasoning refinement requirements is deferred one more round while we map how meaning actually flows through the production update path — before committing to any schema or architecture changes. A source-inspection exercise traced the full answer-to-reasoning chain from prompt building, through LLM response parsing and normalization, into graph mutation. The key finding: no provenance fields exist on nodes or edges in the current schema, meaning R1/R2 separation has no structural carrier. The answer string is used only for a narrow comparability check, not for semantic verification against proposed changes. A complete path map lives in `docs/reasoning-production-path-map.md`. Tomorrow should decide whether to add provenance fields to schemas, modify the prompt structure, or both — grounded in this accurate production trace rather than architectural speculation. Branch: `feature/user-workspace-ux-v0.7`. diff --git a/docs/experiment-57c.md b/docs/experiment-57c.md new file mode 100644 index 0000000..1b861b7 --- /dev/null +++ b/docs/experiment-57c.md @@ -0,0 +1,92 @@ +# Experiment 57C — Post-v0.9 Investigation Flow Observation + +**STOPPED AT FIRST PRODUCTION-PATH FAILURE** + +--- + +## Baseline + +- **Branch:** `main` +- **HEAD at stop:** `371ab0f` (merge(feature/reasoning-guard-generality-v0.9): integrate reasoning-guard generality v0.9 into main) +- **No commits created during 57C.** +- **Production code modified during 57C:** NO + +## Objective + +Observe first-post-v0.9 multi-turn investigation through the real `startCase()` → `updateCase()` production path. Run with a team-relocation scenario to test whether the post-v0.9 reasoning pipeline handles realistic user inputs end-to-end. + +## Scenario Selection + +- **Selected scenario:** "Should I relocate my engineering team from London to Manchester?" +- **Scenario source:** Experiment runner definition (`experiment-57c-runner.mjs`, line 21) — live-written in the run session, not from a pre-existing fixture or test file. +- **Observation frame written before execution:** YES — the handoff entry was drafted during the run session before the first failure was observed. + +## Execution Log (Recovered from Session Context) + +**Turn 0 (startCase — Ollama call #1):** +- startCase produced a scenario graph with an initial question. +- The selected question was about identifying the "primary driver" for the relocation consideration. + +**Turn 1 (updateCase — Ollama call #2):** +- Answer supplied: "The cost savings of £400K per year would fund two new London hires or a modest growth bonus pool." +- Graph mutation applied successfully. Status updated. +- A follow-up question was selected by the model's investigation strategy. + +**Turn 2 (updateCase — Ollama call #3 — FIRST FAILURE):** +- Model response produced a graph edge with `relationship: "affects"`. +- **Production rejection:** The current graph/update schema rejected `"affects"` as an invalid relationship value. +- The validation/schema error occurred at the graph-mutation / edge-insertion stage, before any investigation progression could continue. +- No further calls were made — run was manually stopped. + +## Known Ollama Live-Call Count + +**UNPROVEN** — no preserved request logs or response files exist on disk for the live calls. The only evidence is the session context in which the stop occurred. The runner file (`experiment-57c-runner.mjs`) was not committed and produced no output files. + +## First Valid 57C Failure + +| Item | Value | +|---|---| +| **Failure** | `relationship: "affects"` rejected by current production graph contract | +| **Raw relationship value** | `"affects"` (string, as returned by the live model) | +| **Relevant raw model fragment** | Model output included a graph edge with `relationship: "affects"` connecting two nodes in the situation graph. (No persisted JSON available; observed from session context.) | +| **Production rejection/error** | Graph/update schema rejected `"affects"` as an invalid relationship — it is not listed in the production relationship enum / Zod schema for graph edges. | +| **Failure stage** | Graph mutation / edge-insertion (post-updateCase response processing) | +| **Graph/investigation progressed before failure?** | Turn 1 graph mutation succeeded. Turn 2 failed at the point where the model's output was validated against the schema. Whether partial Turn 2 state was applied is UNCLEAR. | +| **Failure classification** | model-output / graph-contract compatibility | + +## Earlier Odd Initial Question Observation + +- **Observation:** During the same run session, an initial question similar to *"What evidence would clarify how the two observations were measured?"* was noted for the relocation scenario. +- **Classification:** `UNPROVEN LEAD` — not promoted to established defect. There is no preserved output showing this question in isolation or verified as occurring in a clean execution path before the schema failure. It remains an unproven lead for future investigation. + +## Workaround Status + +- Claude considered bypassing the schema failure by switching to a different fixture. +- **Workaround:** NOT EXECUTED — the run was manually stopped instead. No alternative fixture was tested. + +## What Remains Unknown (Open Questions) + +These are established as gaps, not assigned fixes: + +1. Whether `"affects"` should map to an existing relationship in the production graph contract; +2. Whether prompting the model should prevent it from producing `"affects"`; +3. Whether the parser/normalisation boundary is missing a synonym or mapping for this value; +4. Whether the graph schema should be extended to represent `"affects"` as a distinct relationship type; +5. Whether this failure reproduces reliably across runs, models, and domains. + +## Temporary 57C Artefacts (On Disk at Stop) + +- `experiment-57c-runner.mjs` — experiment runner script (untracked, not committed, never produced output files). This file is a temporary tool for running the experiment; its content is documented above in Scenario Selection. +- No result files, logs, or persisted responses exist for the live calls. +- The handoff entry written during the run session (now corrected) was the only documentation artifact on disk. + +## Recovery Action by This Task + +- Corrected the 57C handoff entry to reflect actual stop state and observed failure rather than unverified Turn 2 classification description. +- Created `docs/experiment-57c.md` with full evidence record. +- No production code was modified (confirmed: no changes to lib/ during the run). +- Temporary runner file will be removed in this commit's cleanup. + +--- + +*Documented by Experiment Recovery session. Date: 2026-08-10.*