93 lines
5.4 KiB
Markdown
93 lines
5.4 KiB
Markdown
# Experiment 57C — Post-v0.9 Investigation Flow Observation
|
|
|
|
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
|
|
|
|
---
|
|
|
|
## Baseline
|
|
|
|
- **Branch:** `main`
|
|
- **HEAD at stop:** `371ab0f` (merge(feature/reasoning-guard-generality-v0.9): integrate reasoning-guard generality v0.9 into main)
|
|
- **No commits created during 57C.**
|
|
- **Production code modified during 57C:** NO
|
|
|
|
## Objective
|
|
|
|
Observe first-post-v0.9 multi-turn investigation through the real `startCase()` → `updateCase()` production path. Run with a team-relocation scenario to test whether the post-v0.9 reasoning pipeline handles realistic user inputs end-to-end.
|
|
|
|
## Scenario Selection
|
|
|
|
- **Selected scenario:** "Should I relocate my engineering team from London to Manchester?"
|
|
- **Scenario source:** Experiment runner definition (`experiment-57c-runner.mjs`, line 21) — live-written in the run session, not from a pre-existing fixture or test file.
|
|
- **Observation frame written before execution:** YES — the handoff entry was drafted during the run session before the first failure was observed.
|
|
|
|
## Execution Log (Recovered from Session Context)
|
|
|
|
**Turn 0 (startCase — Ollama call #1):**
|
|
- startCase produced a scenario graph with an initial question.
|
|
- The selected question was about identifying the "primary driver" for the relocation consideration.
|
|
|
|
**Turn 1 (updateCase — Ollama call #2):**
|
|
- Answer supplied: "The cost savings of £400K per year would fund two new London hires or a modest growth bonus pool."
|
|
- Graph mutation applied successfully. Status updated.
|
|
- A follow-up question was selected by the model's investigation strategy.
|
|
|
|
**Turn 2 (updateCase — Ollama call #3 — FIRST FAILURE):**
|
|
- Model response produced a graph edge with `relationship: "affects"`.
|
|
- **Production rejection:** The current graph/update schema rejected `"affects"` as an invalid relationship value.
|
|
- The validation/schema error occurred at the graph-mutation / edge-insertion stage, before any investigation progression could continue.
|
|
- No further calls were made — run was manually stopped.
|
|
|
|
## Known Ollama Live-Call Count
|
|
|
|
**UNPROVEN** — no preserved request logs or response files exist on disk for the live calls. The only evidence is the session context in which the stop occurred. The runner file (`experiment-57c-runner.mjs`) was not committed and produced no output files.
|
|
|
|
## First Valid 57C Failure
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| **Failure** | `relationship: "affects"` rejected by current production graph contract |
|
|
| **Raw relationship value** | `"affects"` (string, as returned by the live model) |
|
|
| **Relevant raw model fragment** | Model output included a graph edge with `relationship: "affects"` connecting two nodes in the situation graph. (No persisted JSON available; observed from session context.) |
|
|
| **Production rejection/error** | Graph/update schema rejected `"affects"` as an invalid relationship — it is not listed in the production relationship enum / Zod schema for graph edges. |
|
|
| **Failure stage** | Graph mutation / edge-insertion (post-updateCase response processing) |
|
|
| **Graph/investigation progressed before failure?** | Turn 1 graph mutation succeeded. Turn 2 failed at the point where the model's output was validated against the schema. Whether partial Turn 2 state was applied is UNCLEAR. |
|
|
| **Failure classification** | model-output / graph-contract compatibility |
|
|
|
|
## Earlier Odd Initial Question Observation
|
|
|
|
- **Observation:** During the same run session, an initial question similar to *"What evidence would clarify how the two observations were measured?"* was noted for the relocation scenario.
|
|
- **Classification:** `UNPROVEN LEAD` — not promoted to established defect. There is no preserved output showing this question in isolation or verified as occurring in a clean execution path before the schema failure. It remains an unproven lead for future investigation.
|
|
|
|
## Workaround Status
|
|
|
|
- Claude considered bypassing the schema failure by switching to a different fixture.
|
|
- **Workaround:** NOT EXECUTED — the run was manually stopped instead. No alternative fixture was tested.
|
|
|
|
## What Remains Unknown (Open Questions)
|
|
|
|
These are established as gaps, not assigned fixes:
|
|
|
|
1. Whether `"affects"` should map to an existing relationship in the production graph contract;
|
|
2. Whether prompting the model should prevent it from producing `"affects"`;
|
|
3. Whether the parser/normalisation boundary is missing a synonym or mapping for this value;
|
|
4. Whether the graph schema should be extended to represent `"affects"` as a distinct relationship type;
|
|
5. Whether this failure reproduces reliably across runs, models, and domains.
|
|
|
|
## Temporary 57C Artefacts (On Disk at Stop)
|
|
|
|
- `experiment-57c-runner.mjs` — experiment runner script (untracked, not committed, never produced output files). This file is a temporary tool for running the experiment; its content is documented above in Scenario Selection.
|
|
- No result files, logs, or persisted responses exist for the live calls.
|
|
- The handoff entry written during the run session (now corrected) was the only documentation artifact on disk.
|
|
|
|
## Recovery Action by This Task
|
|
|
|
- Corrected the 57C handoff entry to reflect actual stop state and observed failure rather than unverified Turn 2 classification description.
|
|
- Created `docs/experiment-57c.md` with full evidence record.
|
|
- No production code was modified (confirmed: no changes to lib/ during the run).
|
|
- Temporary runner file will be removed in this commit's cleanup.
|
|
|
|
---
|
|
|
|
*Documented by Experiment Recovery session. Date: 2026-08-10.*
|