tooling: enforce no-retry live experiment harness

2026-08-11 11:32:19 +01:00
parent 06f67da1b4
commit 4998de54d2
4 changed files with 379 additions and 78 deletions
@@ -418,6 +418,39 @@ Configured Ollama: none used. Production code changed: NO. Tests permanently cha

 ---

+### Experiment 57J.35 — No-Retry Live Experiment Harness Enforcement
+
+**Objective:** Make the canonical live harness physically incapable of hidden retries. Enforce one-shot execution semantics: every requested Start maps to exactly one `/api/cases/start` call, every requested Update maps to exactly one `/api/cases/update` call, and rejections are returned immediately without implicit retry.
+
+**Protocol breach prevention:** This change directly addresses the protocol breach from Experiment 57J.32 where an implicit retry loop consumed multiple Update calls per trial, contaminating evidence. Future prompts may rely on the canonical harness to enforce one-call/no-retry semantics; Claude must not create supplementary retry scripts during bounded experiments.
+
+**Approach:** Bounded execution configuration (`maxUpdates`) + explicit call accounting (startCalls/updateCalls/totalCalls counters reflecting actual API invocations) + rejection-immediate-stop semantics + rejectedProposalSnapshot preservation for v0.16 diagnostic visibility.
+
+**Changes to canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`):**
+- Hardcoded `Start → Update 1 → Update 2` sequence replaced with configurable bounded loop (`config.maxUpdates`)
+- Call accounting added: `calls.startCalls`, `calls.updateCalls`, reported as `totalCalls`
+- Rejection returns immediately; no retry path exists for any semantic outcome (proposal_compatibility, validation failure, etc.)
+- `rejectedProposalSnapshot` preserved and logged when present in Update rejection diagnostics
+- Every update call is explicit in the loop; `config.answers[i]` maps positionally to `Update i+1`
+
+**Tests added (`tests/reproduce-multi-turn-investigation.harness.test.js`):** 8 deterministic cases via synchronous simulation mirror of harness logic — all pass (0 Ollama calls, no dev-server needed). Test cases:
+1. Start success → exactly 1 Start call.
+2. Start failure → exactly 1 Start call, no retry.
+3. Update success → exactly 1 Update call.
+4. `proposal_compatibility` rejection → exactly 1 Update call, rejection returned unchanged.
+5. Update 1 rejection → Update 2 never called.
+6. Update 1 success → Update 2 called exactly once when explicitly requested.
+7. Call counters equal actual mocked API invocations.
+8. No semantic retry after HTTP 422/valid rejection response.
+
+**What this tooling change guarantees:** Future live experiment runs via the canonical harness are physically incapable of consuming more API calls than explicitly configured. Each Start request = exactly one call; each Update request = exactly one call; rejections stop the chain immediately without retry. Call accounting always reflects actual HTTP invocations, not inferred successes.
+
+**What this does NOT guarantee:** That production reasoning is correct (no production code changed). That cold-start variance in node counts is resolved (start graph stability remains an open issue). That semantic validation outcomes change (only the harness wrapper changed). That transport-level failures are handled (not addressed by this tooling change).
+
+**Configured Ollama:** none used. **Production code changed:** NO. **Tests run:** 8 passed, 0 failed.
+
+---
+
 ### Experiment 57J.34 — Multi-Turn Investigation Progress After Accepted Update 1

 **Objective:** On one fresh live run, if the first relocation answer passes the current reasoning safeguards, does answering the savings-realism question produce genuine investigation progress rather than repetition or irrelevant reasoning?