Feature/product platform foundation v0.62 #1
@@ -172,6 +172,14 @@ Answer before continuing:
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to A–F scope.*
|
||||
|
||||
### Experiment 57C — Post-v0.9 Investigation Flow Observation
|
||||
|
||||
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
|
||||
|
||||
Attempted first-post-v0.9 multi-turn investigation through the real `startCase()` → `updateCase()` production path with a team-relocation scenario (London→Manchester). Turn 1 completed: selected question about "primary driver", user answered about cost savings/£400K funding. On Turn 2, a live model response produced `relationship: "affects"` as a graph edge value; the current production graph/update schema rejected that relationship string, preventing progression through the normal investigation path. Run was manually stopped before any workaround was accepted or any fix attempted. No `docs/experiment-57c.md` was written — this entry has been superseded by it.
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 55F)
|
||||
|
||||
The first implementation pass against the reasoning refinement requirements is deferred one more round while we map how meaning actually flows through the production update path — before committing to any schema or architecture changes. A source-inspection exercise traced the full answer-to-reasoning chain from prompt building, through LLM response parsing and normalization, into graph mutation. The key finding: no provenance fields exist on nodes or edges in the current schema, meaning R1/R2 separation has no structural carrier. The answer string is used only for a narrow comparability check, not for semantic verification against proposed changes. A complete path map lives in `docs/reasoning-production-path-map.md`. Tomorrow should decide whether to add provenance fields to schemas, modify the prompt structure, or both — grounded in this accurate production trace rather than architectural speculation. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# Experiment 57C — Post-v0.9 Investigation Flow Observation
|
||||
|
||||
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
|
||||
|
||||
---
|
||||
|
||||
## Baseline
|
||||
|
||||
- **Branch:** `main`
|
||||
- **HEAD at stop:** `371ab0f` (merge(feature/reasoning-guard-generality-v0.9): integrate reasoning-guard generality v0.9 into main)
|
||||
- **No commits created during 57C.**
|
||||
- **Production code modified during 57C:** NO
|
||||
|
||||
## Objective
|
||||
|
||||
Observe first-post-v0.9 multi-turn investigation through the real `startCase()` → `updateCase()` production path. Run with a team-relocation scenario to test whether the post-v0.9 reasoning pipeline handles realistic user inputs end-to-end.
|
||||
|
||||
## Scenario Selection
|
||||
|
||||
- **Selected scenario:** "Should I relocate my engineering team from London to Manchester?"
|
||||
- **Scenario source:** Experiment runner definition (`experiment-57c-runner.mjs`, line 21) — live-written in the run session, not from a pre-existing fixture or test file.
|
||||
- **Observation frame written before execution:** YES — the handoff entry was drafted during the run session before the first failure was observed.
|
||||
|
||||
## Execution Log (Recovered from Session Context)
|
||||
|
||||
**Turn 0 (startCase — Ollama call #1):**
|
||||
- startCase produced a scenario graph with an initial question.
|
||||
- The selected question was about identifying the "primary driver" for the relocation consideration.
|
||||
|
||||
**Turn 1 (updateCase — Ollama call #2):**
|
||||
- Answer supplied: "The cost savings of £400K per year would fund two new London hires or a modest growth bonus pool."
|
||||
- Graph mutation applied successfully. Status updated.
|
||||
- A follow-up question was selected by the model's investigation strategy.
|
||||
|
||||
**Turn 2 (updateCase — Ollama call #3 — FIRST FAILURE):**
|
||||
- Model response produced a graph edge with `relationship: "affects"`.
|
||||
- **Production rejection:** The current graph/update schema rejected `"affects"` as an invalid relationship value.
|
||||
- The validation/schema error occurred at the graph-mutation / edge-insertion stage, before any investigation progression could continue.
|
||||
- No further calls were made — run was manually stopped.
|
||||
|
||||
## Known Ollama Live-Call Count
|
||||
|
||||
**UNPROVEN** — no preserved request logs or response files exist on disk for the live calls. The only evidence is the session context in which the stop occurred. The runner file (`experiment-57c-runner.mjs`) was not committed and produced no output files.
|
||||
|
||||
## First Valid 57C Failure
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| **Failure** | `relationship: "affects"` rejected by current production graph contract |
|
||||
| **Raw relationship value** | `"affects"` (string, as returned by the live model) |
|
||||
| **Relevant raw model fragment** | Model output included a graph edge with `relationship: "affects"` connecting two nodes in the situation graph. (No persisted JSON available; observed from session context.) |
|
||||
| **Production rejection/error** | Graph/update schema rejected `"affects"` as an invalid relationship — it is not listed in the production relationship enum / Zod schema for graph edges. |
|
||||
| **Failure stage** | Graph mutation / edge-insertion (post-updateCase response processing) |
|
||||
| **Graph/investigation progressed before failure?** | Turn 1 graph mutation succeeded. Turn 2 failed at the point where the model's output was validated against the schema. Whether partial Turn 2 state was applied is UNCLEAR. |
|
||||
| **Failure classification** | model-output / graph-contract compatibility |
|
||||
|
||||
## Earlier Odd Initial Question Observation
|
||||
|
||||
- **Observation:** During the same run session, an initial question similar to *"What evidence would clarify how the two observations were measured?"* was noted for the relocation scenario.
|
||||
- **Classification:** `UNPROVEN LEAD` — not promoted to established defect. There is no preserved output showing this question in isolation or verified as occurring in a clean execution path before the schema failure. It remains an unproven lead for future investigation.
|
||||
|
||||
## Workaround Status
|
||||
|
||||
- Claude considered bypassing the schema failure by switching to a different fixture.
|
||||
- **Workaround:** NOT EXECUTED — the run was manually stopped instead. No alternative fixture was tested.
|
||||
|
||||
## What Remains Unknown (Open Questions)
|
||||
|
||||
These are established as gaps, not assigned fixes:
|
||||
|
||||
1. Whether `"affects"` should map to an existing relationship in the production graph contract;
|
||||
2. Whether prompting the model should prevent it from producing `"affects"`;
|
||||
3. Whether the parser/normalisation boundary is missing a synonym or mapping for this value;
|
||||
4. Whether the graph schema should be extended to represent `"affects"` as a distinct relationship type;
|
||||
5. Whether this failure reproduces reliably across runs, models, and domains.
|
||||
|
||||
## Temporary 57C Artefacts (On Disk at Stop)
|
||||
|
||||
- `experiment-57c-runner.mjs` — experiment runner script (untracked, not committed, never produced output files). This file is a temporary tool for running the experiment; its content is documented above in Scenario Selection.
|
||||
- No result files, logs, or persisted responses exist for the live calls.
|
||||
- The handoff entry written during the run session (now corrected) was the only documentation artifact on disk.
|
||||
|
||||
## Recovery Action by This Task
|
||||
|
||||
- Corrected the 57C handoff entry to reflect actual stop state and observed failure rather than unverified Turn 2 classification description.
|
||||
- Created `docs/experiment-57c.md` with full evidence record.
|
||||
- No production code was modified (confirmed: no changes to lib/ during the run).
|
||||
- Temporary runner file will be removed in this commit's cleanup.
|
||||
|
||||
---
|
||||
|
||||
*Documented by Experiment Recovery session. Date: 2026-08-10.*
|
||||
Reference in New Issue
Block a user