docs: record stopped experiment 57c contract failure

This commit is contained in:
2026-08-10 06:04:46 +01:00
parent 371ab0f52f
commit cf6c5cb57f
2 changed files with 100 additions and 0 deletions
+8
View File
@@ -172,6 +172,14 @@ Answer before continuing:
*Created by Experiment 34. Updated by Experiments 3853, 54A54Z, 55A55F, 56D56H, 56L56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to AF scope.*
### Experiment 57C — Post-v0.9 Investigation Flow Observation
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
Attempted first-post-v0.9 multi-turn investigation through the real `startCase()``updateCase()` production path with a team-relocation scenario (London→Manchester). Turn 1 completed: selected question about "primary driver", user answered about cost savings/£400K funding. On Turn 2, a live model response produced `relationship: "affects"` as a graph edge value; the current production graph/update schema rejected that relationship string, preventing progression through the normal investigation path. Run was manually stopped before any workaround was accepted or any fix attempted. No `docs/experiment-57c.md` was written — this entry has been superseded by it.
---
### Return-to-Work Note (Experiment 55F)
The first implementation pass against the reasoning refinement requirements is deferred one more round while we map how meaning actually flows through the production update path — before committing to any schema or architecture changes. A source-inspection exercise traced the full answer-to-reasoning chain from prompt building, through LLM response parsing and normalization, into graph mutation. The key finding: no provenance fields exist on nodes or edges in the current schema, meaning R1/R2 separation has no structural carrier. The answer string is used only for a narrow comparability check, not for semantic verification against proposed changes. A complete path map lives in `docs/reasoning-production-path-map.md`. Tomorrow should decide whether to add provenance fields to schemas, modify the prompt structure, or both — grounded in this accurate production trace rather than architectural speculation. Branch: `feature/user-workspace-ux-v0.7`.
+92
View File
@@ -0,0 +1,92 @@
# Experiment 57C — Post-v0.9 Investigation Flow Observation
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
---
## Baseline
- **Branch:** `main`
- **HEAD at stop:** `371ab0f` (merge(feature/reasoning-guard-generality-v0.9): integrate reasoning-guard generality v0.9 into main)
- **No commits created during 57C.**
- **Production code modified during 57C:** NO
## Objective
Observe first-post-v0.9 multi-turn investigation through the real `startCase()``updateCase()` production path. Run with a team-relocation scenario to test whether the post-v0.9 reasoning pipeline handles realistic user inputs end-to-end.
## Scenario Selection
- **Selected scenario:** "Should I relocate my engineering team from London to Manchester?"
- **Scenario source:** Experiment runner definition (`experiment-57c-runner.mjs`, line 21) — live-written in the run session, not from a pre-existing fixture or test file.
- **Observation frame written before execution:** YES — the handoff entry was drafted during the run session before the first failure was observed.
## Execution Log (Recovered from Session Context)
**Turn 0 (startCase — Ollama call #1):**
- startCase produced a scenario graph with an initial question.
- The selected question was about identifying the "primary driver" for the relocation consideration.
**Turn 1 (updateCase — Ollama call #2):**
- Answer supplied: "The cost savings of £400K per year would fund two new London hires or a modest growth bonus pool."
- Graph mutation applied successfully. Status updated.
- A follow-up question was selected by the model's investigation strategy.
**Turn 2 (updateCase — Ollama call #3 — FIRST FAILURE):**
- Model response produced a graph edge with `relationship: "affects"`.
- **Production rejection:** The current graph/update schema rejected `"affects"` as an invalid relationship value.
- The validation/schema error occurred at the graph-mutation / edge-insertion stage, before any investigation progression could continue.
- No further calls were made — run was manually stopped.
## Known Ollama Live-Call Count
**UNPROVEN** — no preserved request logs or response files exist on disk for the live calls. The only evidence is the session context in which the stop occurred. The runner file (`experiment-57c-runner.mjs`) was not committed and produced no output files.
## First Valid 57C Failure
| Item | Value |
|---|---|
| **Failure** | `relationship: "affects"` rejected by current production graph contract |
| **Raw relationship value** | `"affects"` (string, as returned by the live model) |
| **Relevant raw model fragment** | Model output included a graph edge with `relationship: "affects"` connecting two nodes in the situation graph. (No persisted JSON available; observed from session context.) |
| **Production rejection/error** | Graph/update schema rejected `"affects"` as an invalid relationship — it is not listed in the production relationship enum / Zod schema for graph edges. |
| **Failure stage** | Graph mutation / edge-insertion (post-updateCase response processing) |
| **Graph/investigation progressed before failure?** | Turn 1 graph mutation succeeded. Turn 2 failed at the point where the model's output was validated against the schema. Whether partial Turn 2 state was applied is UNCLEAR. |
| **Failure classification** | model-output / graph-contract compatibility |
## Earlier Odd Initial Question Observation
- **Observation:** During the same run session, an initial question similar to *"What evidence would clarify how the two observations were measured?"* was noted for the relocation scenario.
- **Classification:** `UNPROVEN LEAD` — not promoted to established defect. There is no preserved output showing this question in isolation or verified as occurring in a clean execution path before the schema failure. It remains an unproven lead for future investigation.
## Workaround Status
- Claude considered bypassing the schema failure by switching to a different fixture.
- **Workaround:** NOT EXECUTED — the run was manually stopped instead. No alternative fixture was tested.
## What Remains Unknown (Open Questions)
These are established as gaps, not assigned fixes:
1. Whether `"affects"` should map to an existing relationship in the production graph contract;
2. Whether prompting the model should prevent it from producing `"affects"`;
3. Whether the parser/normalisation boundary is missing a synonym or mapping for this value;
4. Whether the graph schema should be extended to represent `"affects"` as a distinct relationship type;
5. Whether this failure reproduces reliably across runs, models, and domains.
## Temporary 57C Artefacts (On Disk at Stop)
- `experiment-57c-runner.mjs` — experiment runner script (untracked, not committed, never produced output files). This file is a temporary tool for running the experiment; its content is documented above in Scenario Selection.
- No result files, logs, or persisted responses exist for the live calls.
- The handoff entry written during the run session (now corrected) was the only documentation artifact on disk.
## Recovery Action by This Task
- Corrected the 57C handoff entry to reflect actual stop state and observed failure rather than unverified Turn 2 classification description.
- Created `docs/experiment-57c.md` with full evidence record.
- No production code was modified (confirmed: no changes to lib/ during the run).
- Temporary runner file will be removed in this commit's cleanup.
---
*Documented by Experiment Recovery session. Date: 2026-08-10.*