docs: record accepted-update capture hardening
This commit is contained in:
@@ -203,6 +203,24 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.62 — Accepted-Update Capture Hardening
|
||||
|
||||
**Classification: A — FIX VALIDATED.** Diagnosed that 57J.61 failed because the harness accepted-update path (script lines ~97–108) printed only `HTTP status`, `stage`, `selected question`, `node count`, and `edge count` — zero answer-meaning fields, zero structural mutation fields. After Update 1 applied successfully with HTTP 200 at `update_applied`, the harness could not identify which nodes were added or what the resulting persistent graph looked like.
|
||||
|
||||
**Fix:** Extended the accepted-update console block in `scripts/reproduce-multi-turn-investigation.mjs` to print:
|
||||
- `answerMeaning.userSupportedMeaning`, `.possibleInference`, `.supportCategory`, `.resolutionGuidance`
|
||||
- `updatedProposal.updatedNodes[]`, `.resolvedUnknownNodeIds[]`, `.addedNodes[]`, `.addedEdges[]`
|
||||
- `selectedQuestion.nodeId` (node reference)
|
||||
- Compact structural snapshot of `resulting graph` (id, kind, label/description, status per node; from/to/relationship per edge)
|
||||
|
||||
Fixed a co-occurring bug where the accepted-update block referenced `startResult.status` instead of `updateResult.status`.
|
||||
|
||||
**Tests:** 10 new harness tests (8 in test suite + 2 for existing guarantees), all pass. Mocked API responses only. Zero Ollama calls. No production code changed. No extra HTTP calls introduced. No-retry contract preserved intact.
|
||||
|
||||
**What this establishes:** Future experiments will produce deterministic, inspectable evidence of every accepted update's graph mutations without requiring a second API call or manual inspection.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test
|
||||
|
||||
**Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Experiment 57J.62 — Accepted-Update Capture Hardening
|
||||
|
||||
**Branch:** `feature/selected-question-contract-v0.22`
|
||||
**Starting HEAD:** `929486c` (experiment: validate equivalent uncertainty identity live)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> Why did the canonical harness fail to retain enough accepted Update 1 detail in 57J.61 to identify the persistent node that was added, and what is the smallest tooling change that makes accepted-update evidence reliable for the next live experiment?
|
||||
|
||||
## Classification: D — harness prints summary counts but not accepted proposal detail
|
||||
|
||||
## Exact capture failure cause
|
||||
|
||||
The harness's accepted-update output block (lines ~97–108 of `scripts/reproduce-multi-turn-investigation.mjs`) printed only:
|
||||
|
||||
```text
|
||||
HTTP status
|
||||
stage
|
||||
proposal/apply success
|
||||
selected question
|
||||
node count
|
||||
edge count
|
||||
```
|
||||
|
||||
It did NOT print any of the response body fields that describe graph mutations:
|
||||
|
||||
- `answerMeaning.userSupportedMeaning` — absent
|
||||
- `answerMeaning.possibleInference` — absent
|
||||
- `answerMeaning.supportCategory` — absent
|
||||
- `answerMeaning.resolutionGuidance` — absent
|
||||
- `updatedProposal.updatedNodes[]` — absent
|
||||
- `updatedProposal.resolvedUnknownNodeIds[]` — absent
|
||||
- `updatedProposal.addedNodes[]` — absent
|
||||
- `updatedProposal.addedEdges[]` — absent
|
||||
- `selectedQuestion.nodeId` (node reference) — absent
|
||||
- Resulting graph node/edge details — absent
|
||||
|
||||
After 57J.61's Update 1 returned HTTP 200 at `update_applied` with node count 6→7 and edge count 5→6, the harness produced no tooling-level evidence of **which** node was added or **what** it contained. The identity invariant ("equivalent unresolved meaning must not multiply graph state") cannot be tested when the evidence is missing.
|
||||
|
||||
A co-occurring bug: line ~102 referenced `startResult.status` instead of `updateResult.status`, printing the Start HTTP status in the Update block (cosmetic, not evidentiary).
|
||||
|
||||
## Changes made
|
||||
|
||||
### `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Extended accepted-update output block to print:
|
||||
|
||||
```javascript
|
||||
// answerMeaning fields
|
||||
answerMeaning.userSupportedMeaning
|
||||
answerMeaning.possibleInference
|
||||
answerMeaning.supportCategory
|
||||
answerMeaning.resolutionGuidance
|
||||
|
||||
// structural mutation fields
|
||||
updatedProposal.updatedNodes[]
|
||||
updatedProposal.resolvedUnknownNodeIds[]
|
||||
updatedProposal.addedNodes[]
|
||||
updatedProposal.addedEdges[]
|
||||
|
||||
// selectedQuestion node reference
|
||||
selectedQuestion.nodeId
|
||||
|
||||
// Compact structural snapshot of resulting persistent graph
|
||||
resulting graph: {id, kind, label/description, status} per node
|
||||
: {from/to/relationship} per edge
|
||||
```
|
||||
|
||||
Fixed `startResult.status` → `updateResult.status`.
|
||||
|
||||
### `tests/reproduce-multi-turn-investigation.harness.test.js`
|
||||
|
||||
Added 10 new deterministic harness tests via a companion simulation function (`runSimulationWithResponseShape`) that records capture outputs:
|
||||
|
||||
1. accepted Update exposes addedNodes details
|
||||
2. accepted Update exposes updatedNodes details
|
||||
3. accepted Update exposes resolvedUnknownNodeIds
|
||||
4. accepted Update exposes selectedQuestion (question + nodeId)
|
||||
5. accepted Update exposes answerMeaning structured fields
|
||||
6. accepted Update exposes resulting persistent graph nodes/edges
|
||||
7. rejected Update still exposes rejectedProposalSnapshot (existing behavior verified)
|
||||
8. Update 1 accepted → Update 2 receives exactly that resulting graph state
|
||||
9. no extra HTTP call is introduced for diagnostics
|
||||
10. existing no-retry and call-accounting guarantees remain intact
|
||||
|
||||
All tests use mocked API responses only. Zero Ollama calls. Zero dev-server calls.
|
||||
|
||||
## Invariants preserved
|
||||
|
||||
- One Start invocation = one API call
|
||||
- One Update invocation = one API call
|
||||
- No semantic retries
|
||||
- No transport retries
|
||||
- Update failure stops the chain
|
||||
- Call accounting remains exact
|
||||
- RejectedProposalSnapshot path unchanged for rejected updates
|
||||
|
||||
## What this does NOT change
|
||||
|
||||
- Production API behavior
|
||||
- Production reasoning code
|
||||
- Prompt instructions
|
||||
- Schema definitions
|
||||
- Validator logic
|
||||
- Provider/model integration
|
||||
|
||||
## Configured Ollama: none used. Dev server disturbed: NO.
|
||||
|
||||
## Tests
|
||||
|
||||
18 tests pass (8 existing + 10 new). 0 failed.
|
||||
Reference in New Issue
Block a user