From 47509d307b4efa73383e3b24872dfcfbdc8c046e Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 19:43:11 +0100 Subject: [PATCH] docs: record accepted-update capture hardening --- docs/current-handoff.md | 18 +++++++ docs/experiment-57j62.md | 113 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+) create mode 100644 docs/experiment-57j62.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 5d3a198..b1c9df1 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -203,6 +203,24 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea --- +### Experiment 57J.62 — Accepted-Update Capture Hardening + +**Classification: A — FIX VALIDATED.** Diagnosed that 57J.61 failed because the harness accepted-update path (script lines ~97–108) printed only `HTTP status`, `stage`, `selected question`, `node count`, and `edge count` — zero answer-meaning fields, zero structural mutation fields. After Update 1 applied successfully with HTTP 200 at `update_applied`, the harness could not identify which nodes were added or what the resulting persistent graph looked like. + +**Fix:** Extended the accepted-update console block in `scripts/reproduce-multi-turn-investigation.mjs` to print: +- `answerMeaning.userSupportedMeaning`, `.possibleInference`, `.supportCategory`, `.resolutionGuidance` +- `updatedProposal.updatedNodes[]`, `.resolvedUnknownNodeIds[]`, `.addedNodes[]`, `.addedEdges[]` +- `selectedQuestion.nodeId` (node reference) +- Compact structural snapshot of `resulting graph` (id, kind, label/description, status per node; from/to/relationship per edge) + +Fixed a co-occurring bug where the accepted-update block referenced `startResult.status` instead of `updateResult.status`. + +**Tests:** 10 new harness tests (8 in test suite + 2 for existing guarantees), all pass. Mocked API responses only. Zero Ollama calls. No production code changed. No extra HTTP calls introduced. No-retry contract preserved intact. + +**What this establishes:** Future experiments will produce deterministic, inspectable evidence of every accepted update's graph mutations without requiring a second API call or manual inspection. + +--- + ### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test **Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`. diff --git a/docs/experiment-57j62.md b/docs/experiment-57j62.md new file mode 100644 index 0000000..d15fc60 --- /dev/null +++ b/docs/experiment-57j62.md @@ -0,0 +1,113 @@ +# Experiment 57J.62 — Accepted-Update Capture Hardening + +**Branch:** `feature/selected-question-contract-v0.22` +**Starting HEAD:** `929486c` (experiment: validate equivalent uncertainty identity live) + +## Objective + +Answer exactly: + +> Why did the canonical harness fail to retain enough accepted Update 1 detail in 57J.61 to identify the persistent node that was added, and what is the smallest tooling change that makes accepted-update evidence reliable for the next live experiment? + +## Classification: D — harness prints summary counts but not accepted proposal detail + +## Exact capture failure cause + +The harness's accepted-update output block (lines ~97–108 of `scripts/reproduce-multi-turn-investigation.mjs`) printed only: + +```text +HTTP status +stage +proposal/apply success +selected question +node count +edge count +``` + +It did NOT print any of the response body fields that describe graph mutations: + +- `answerMeaning.userSupportedMeaning` — absent +- `answerMeaning.possibleInference` — absent +- `answerMeaning.supportCategory` — absent +- `answerMeaning.resolutionGuidance` — absent +- `updatedProposal.updatedNodes[]` — absent +- `updatedProposal.resolvedUnknownNodeIds[]` — absent +- `updatedProposal.addedNodes[]` — absent +- `updatedProposal.addedEdges[]` — absent +- `selectedQuestion.nodeId` (node reference) — absent +- Resulting graph node/edge details — absent + +After 57J.61's Update 1 returned HTTP 200 at `update_applied` with node count 6→7 and edge count 5→6, the harness produced no tooling-level evidence of **which** node was added or **what** it contained. The identity invariant ("equivalent unresolved meaning must not multiply graph state") cannot be tested when the evidence is missing. + +A co-occurring bug: line ~102 referenced `startResult.status` instead of `updateResult.status`, printing the Start HTTP status in the Update block (cosmetic, not evidentiary). + +## Changes made + +### `scripts/reproduce-multi-turn-investigation.mjs` + +Extended accepted-update output block to print: + +```javascript +// answerMeaning fields +answerMeaning.userSupportedMeaning +answerMeaning.possibleInference +answerMeaning.supportCategory +answerMeaning.resolutionGuidance + +// structural mutation fields +updatedProposal.updatedNodes[] +updatedProposal.resolvedUnknownNodeIds[] +updatedProposal.addedNodes[] +updatedProposal.addedEdges[] + +// selectedQuestion node reference +selectedQuestion.nodeId + +// Compact structural snapshot of resulting persistent graph +resulting graph: {id, kind, label/description, status} per node + : {from/to/relationship} per edge +``` + +Fixed `startResult.status` → `updateResult.status`. + +### `tests/reproduce-multi-turn-investigation.harness.test.js` + +Added 10 new deterministic harness tests via a companion simulation function (`runSimulationWithResponseShape`) that records capture outputs: + +1. accepted Update exposes addedNodes details +2. accepted Update exposes updatedNodes details +3. accepted Update exposes resolvedUnknownNodeIds +4. accepted Update exposes selectedQuestion (question + nodeId) +5. accepted Update exposes answerMeaning structured fields +6. accepted Update exposes resulting persistent graph nodes/edges +7. rejected Update still exposes rejectedProposalSnapshot (existing behavior verified) +8. Update 1 accepted → Update 2 receives exactly that resulting graph state +9. no extra HTTP call is introduced for diagnostics +10. existing no-retry and call-accounting guarantees remain intact + +All tests use mocked API responses only. Zero Ollama calls. Zero dev-server calls. + +## Invariants preserved + +- One Start invocation = one API call +- One Update invocation = one API call +- No semantic retries +- No transport retries +- Update failure stops the chain +- Call accounting remains exact +- RejectedProposalSnapshot path unchanged for rejected updates + +## What this does NOT change + +- Production API behavior +- Production reasoning code +- Prompt instructions +- Schema definitions +- Validator logic +- Provider/model integration + +## Configured Ollama: none used. Dev server disturbed: NO. + +## Tests + +18 tests pass (8 existing + 10 new). 0 failed. \ No newline at end of file