Experiment 57C — Post-v0.9 Investigation Flow Observation

STOPPED AT FIRST PRODUCTION-PATH FAILURE

Baseline

Branch: main
HEAD at stop: 371ab0f (merge(feature/reasoning-guard-generality-v0.9): integrate reasoning-guard generality v0.9 into main)
No commits created during 57C.
Production code modified during 57C: NO

Objective

Observe first-post-v0.9 multi-turn investigation through the real startCase() → updateCase() production path. Run with a team-relocation scenario to test whether the post-v0.9 reasoning pipeline handles realistic user inputs end-to-end.

Scenario Selection

Selected scenario: "Should I relocate my engineering team from London to Manchester?"
Scenario source: Experiment runner definition (experiment-57c-runner.mjs, line 21) — live-written in the run session, not from a pre-existing fixture or test file.
Observation frame written before execution: YES — the handoff entry was drafted during the run session before the first failure was observed.

Execution Log (Recovered from Session Context)

Turn 0 (startCase — Ollama call #1):

startCase produced a scenario graph with an initial question.
The selected question was about identifying the "primary driver" for the relocation consideration.

Turn 1 (updateCase — Ollama call #2):

Answer supplied: "The cost savings of £400K per year would fund two new London hires or a modest growth bonus pool."
Graph mutation applied successfully. Status updated.
A follow-up question was selected by the model's investigation strategy.

Turn 2 (updateCase — Ollama call #3 — FIRST FAILURE):

Model response produced a graph edge with relationship: "affects".
Production rejection: The current graph/update schema rejected "affects" as an invalid relationship value.
The validation/schema error occurred at the graph-mutation / edge-insertion stage, before any investigation progression could continue.
No further calls were made — run was manually stopped.

Known Ollama Live-Call Count

UNPROVEN — no preserved request logs or response files exist on disk for the live calls. The only evidence is the session context in which the stop occurred. The runner file (experiment-57c-runner.mjs) was not committed and produced no output files.

First Valid 57C Failure

Item	Value
Failure	`relationship: "affects"` rejected by current production graph contract
Raw relationship value	`"affects"` (string, as returned by the live model)
Relevant raw model fragment	Model output included a graph edge with `relationship: "affects"` connecting two nodes in the situation graph. (No persisted JSON available; observed from session context.)
Production rejection/error	Graph/update schema rejected `"affects"` as an invalid relationship — it is not listed in the production relationship enum / Zod schema for graph edges.
Failure stage	Graph mutation / edge-insertion (post-updateCase response processing)
Graph/investigation progressed before failure?	Turn 1 graph mutation succeeded. Turn 2 failed at the point where the model's output was validated against the schema. Whether partial Turn 2 state was applied is UNCLEAR.
Failure classification	model-output / graph-contract compatibility

Earlier Odd Initial Question Observation

Observation: During the same run session, an initial question similar to "What evidence would clarify how the two observations were measured?" was noted for the relocation scenario.
Classification: UNPROVEN LEAD — not promoted to established defect. There is no preserved output showing this question in isolation or verified as occurring in a clean execution path before the schema failure. It remains an unproven lead for future investigation.

Workaround Status

Claude considered bypassing the schema failure by switching to a different fixture.
Workaround: NOT EXECUTED — the run was manually stopped instead. No alternative fixture was tested.

What Remains Unknown (Open Questions)

These are established as gaps, not assigned fixes:

Whether "affects" should map to an existing relationship in the production graph contract;
Whether prompting the model should prevent it from producing "affects";
Whether the parser/normalisation boundary is missing a synonym or mapping for this value;
Whether the graph schema should be extended to represent "affects" as a distinct relationship type;
Whether this failure reproduces reliably across runs, models, and domains.

Temporary 57C Artefacts (On Disk at Stop)

experiment-57c-runner.mjs — experiment runner script (untracked, not committed, never produced output files). This file is a temporary tool for running the experiment; its content is documented above in Scenario Selection.
No result files, logs, or persisted responses exist for the live calls.
The handoff entry written during the run session (now corrected) was the only documentation artifact on disk.

Recovery Action by This Task

Corrected the 57C handoff entry to reflect actual stop state and observed failure rather than unverified Turn 2 classification description.
Created docs/experiment-57c.md with full evidence record.
No production code was modified (confirmed: no changes to lib/ during the run).
Temporary runner file will be removed in this commit's cleanup.

Documented by Experiment Recovery session. Date: 2026-08-10.

5.4 KiB Raw Blame History