diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 2f5c002..ccefd3d 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -467,3 +467,31 @@ Configured Ollama: none used. Production code changed: NO. Tests permanently cha **What this does NOT prove:** That all cold-starts produce 5 nodes; that zero-node proposals are always produced on accepted updates; whether the merged-unknown behavior is intentional or a defect. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 4 live calls total (2 starts + 2 updates in final pipeline; 1 start + 1 update rejected from harness). No production code changed. + +### Experiment 57J.36 — Multi-Turn Investigation Progress After Accepted Update 1 (Clean Run) + +**Objective:** One clean guarded multi-turn case using the hardened canonical harness (bounded execution, no-retry) to determine whether a semantically faithful first answer produces genuine downstream progress on the second turn. + +**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs." +**Answer 1:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." +**Answer 2 (not reached):** "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year." + +**Live-call count:** start: 1, update: 1, total: 2 + +**START:** HTTP 200 | stage: unknown | nodes: 10 | edges: 5 | question: "What would clarify total projected costs at the new location, including one-time relocation expenses and long-term savings in this situation?" + +**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "Update contains no meaningful change" + +Rejected proposal snapshot: `answerMeaning.userSupportedMeaning` preserved both evidence dimensions faithfully; all structural fields empty (zero addedNodes, zero addedEdges, zero resolvedUnknownNodeIds, zero updatedNodes). + +**Classification: B — DIFFERENT REJECTION.** Rejected for "Update contains no meaningful change" at `proposal_compatibility`, not for semantic strengthening. The LLM produced a structurally null proposal despite semantically faithful meaning extraction. U1-B (not U1-A because no strengthening present; not U1-C/U1-D because nothing was applied). + +**Savings realism:** UNCLEAR +**Retention impact:** UNCLEAR + +**What this establishes:** (1) When the LLM produces a structurally empty proposal with semantically faithful meaning, `proposal_compatibility` correctly rejects it as a no-op; (2) The gap between semantic extraction and structural creation exists independently of the semantic fidelity gate; (3) Cold-start produced 10 nodes (continuing the variance from 5→8 in 57J.34). + +**What this does NOT prove:** That the LLM can produce a structurally non-empty faithful proposal; that downstream progress would occur with an accepted proposal; run-to-run stability of node counts or proposal structure. + +Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. + diff --git a/docs/experiment-57j36.md b/docs/experiment-57j36.md new file mode 100644 index 0000000..1de7e1a --- /dev/null +++ b/docs/experiment-57j36.md @@ -0,0 +1,115 @@ +# Experiment 57J.36 — Multi-Turn Investigation Progress After Accepted Update 1 (Clean Run) + +## Objective + +Run one clean case to answer: **If the first relocation answer produces an acceptable proposal, does answering the resulting savings-realism question make genuine investigation progress on the next turn?** + +This is a hardened replacement for 57J.34/35, using only the canonical harness with bounded execution and no-retry semantics. + +## Pre-written expectation recorded: YES + +> If Update 1 produces a faithful proposal, the user's two evidence needs should remain represented as genuine unresolved issues and the engine should select a grounded next question. If Answer 2 then supplies the requested savings evidence, the investigation should progress rather than substantially repeat savings realism or move into unsupported comparison/timing reasoning. + +> If Update 1 is correctly rejected for semantic strengthening, that is a valid protected outcome and the experiment stops there. Do not retry to obtain an accepted case. + +## Starting HEAD + +`4998de5` — tooling: enforce no-retry live experiment harness + +## Fixed Inputs + +**Scenario:** "We are considering relocating the engineering team to reduce operating costs." + +**Answer 1:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +**Answer 2:** "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year." + +## Harness Configuration + +- `maxUpdates = 2` +- `config.answers[0]` → Answer 1 +- `config.answers[1]` → Answer 2 +- No loops, no attempts, single execution path + +## Results + +### START + +- **HTTP:** 200 +- **Stage:** unknown +- **Nodes:** 10 +- **Edges:** 5 +- **Selected question:** "What would clarify total projected costs at the new location, including one-time relocation expenses and long-term savings in this situation?" + +### UPDATE 1 + +- **HTTP:** 422 +- **Stage:** proposal_compatibility +- **First error:** "Update contains no meaningful change" +- **Nodes:** 10 (unchanged) +- **Edges:** 5 (unchanged) +- **Selected question:** null +- **Savings realism:** UNCLEAR +- **Retention impact:** UNCLEAR + +**Rejected proposal snapshot:** +```json +{ + "answerMeaning": { + "userSupportedMeaning": "Before deciding on relocation, the user requires two specific pieces of evidence: verification that projected office savings are realistic, and assurance that the move will not materially increase the loss of key engineers.", + "possibleInference": null + }, + "updatedNodes": [], + "resolvedUnknownNodeIds": [], + "addedNodes": [], + "addedEdges": [] +} +``` + +**Update 1 classification: U1-B — DIFFERENT REJECTION** + +The rejection is for "Update contains no meaningful change" at `proposal_compatibility`, not for semantic strengthening. The LLM produced a null structural proposal (zero addedNodes, zero addedEdges) even though the answer clearly introduced two new evidence dimensions. This is neither a correct fidelity rejection nor an applied proposal — it is a structurally empty proposal rejected by a different gate. + +### UPDATE 2 + +- **Reached:** NO + +## Call Accounting + +- **startCalls:** 1 +- **updateCalls:** 1 +- **totalCalls:** 2 +- **Valid maximum:** 3 ✓ + +## Supplementary scripts used: NO +## Retries: 0 + +## Classification + +**U1-B — DIFFERENT REJECTION.** Rejected for "Update contains no meaningful change" at the `proposal_compatibility` stage. This differs from: +- U1-A (correct fidelity rejection): no semantic strengthening was present in `userSupportedMeaning` +- U1-C (applied with both dimensions): no nodes or edges were added at all +- U1-D (applied but degraded): nothing was applied + +The LLM's answer meaning extraction was semantically faithful (preserved both evidence dimensions), but produced zero structural change — no addedNodes, no addedEdges, no resolvedUnknownNodeIds, no updatedNodes. The proposal compatibility gate correctly blocked a structurally empty update. + +## What this clean run establishes + +1. When the LLM produces a **structurally empty** proposal (zero additions) even with semantically faithful answer meaning, the `proposal_compatibility` gate rejects it with "Update contains no meaningful change" — a valid protection against no-op updates. +2. The LLM did not strengthen meaning beyond the raw answer in this run (U1-A would have been appropriate if strengthening were present). +3. Cold-start produced 10 nodes (different from prior runs: Ex 57J.34 got 6–8; Ex 57J.32 got 5–8), confirming cold-start node variance persists. + +## What it does NOT prove + +1. Whether the LLM can produce a **structurally non-empty** faithful proposal that passes `proposal_compatibility` (the structural creation step may be separately impaired). +2. That downstream progress on Update 2 would occur even with an accepted proposal. +3. Run-to-run stability of node counts or proposal structure for this scenario. +4. Whether the "no meaningful change" rejection is desirable behaviour when the user clearly introduces new information but the model fails to act on it structurally. + +## Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434 +## Production code changed: NO +## Prompt changed: NO +## Canonical harness restored: YES +## Hardened no-retry behaviour preserved: YES +## Dev server disturbed: NO +## Ollama calls beyond harness count: 0