tooling: enforce no-retry live experiment harness
This commit is contained in:
@@ -0,0 +1,121 @@
|
||||
# Experiment 57J.35 — No-Retry Live Experiment Harness Enforcement
|
||||
|
||||
## Objective
|
||||
|
||||
Make the canonical live harness (`scripts/reproduce-multi-turn-investigation.mjs`) physically incapable of hidden retries. Enforce one-shot execution semantics:
|
||||
|
||||
- One requested Start = exactly one `/api/cases/start` call
|
||||
- One requested Update = exactly one `/api/cases/update` call
|
||||
- A rejection is returned immediately and is never retried implicitly
|
||||
|
||||
This directly addresses the protocol breach from Experiment 57J.32 where an implicit retry loop consumed multiple Update calls per trial, contaminating evidence.
|
||||
|
||||
## Pre-written expectation recorded: YES
|
||||
|
||||
> The canonical harness must enforce one-call/no-retry semantics for all bounded experiments. Future prompts may rely on this; Claude must not create supplementary retry scripts during bounded experiments.
|
||||
|
||||
## Protocol breach referenced: Experiment 57J.32
|
||||
|
||||
Experiment 57J.32 documented a protocol breach where the original harness used an implicit retry loop for accepted results — meaning each "trial" potentially consumed multiple Update calls. This experiment enforces that the canonical apparatus cannot repeat that error.
|
||||
|
||||
## Starting HEAD
|
||||
|
||||
`06f67da` — experiment: observe guarded multi-turn progress
|
||||
|
||||
## Original Harness (commit 7533e47)
|
||||
|
||||
The original harness was a hardcoded sequential script:
|
||||
|
||||
```
|
||||
Start → Update 1 → Update 2
|
||||
```
|
||||
|
||||
Issues with original:
|
||||
- No configuration system (scenario and answers hardcoded)
|
||||
- No call accounting
|
||||
- No rejection diagnostics (`rejectedProposalSnapshot` not handled)
|
||||
- Not flexible for bounded experiments (always exactly 2 updates)
|
||||
- However: no explicit retry loops existed in the original — but the lack of bounded config allowed ad-hoc supplementary scripts with retries (as happened in 57J.32)
|
||||
|
||||
## Changes to Canonical Harness
|
||||
|
||||
### Before (original, commit 7533e47)
|
||||
- Hardcoded sequential flow: `Start → Update 1 → Update 2`
|
||||
- No configuration object
|
||||
- No call accounting
|
||||
- No rejection diagnostics
|
||||
- No explicit "no retry" documentation
|
||||
|
||||
### After (current working tree)
|
||||
- **Bounded execution configuration:** `config.maxUpdates` + `config.answers[]` positional mapping
|
||||
- **Call accounting:** `calls.startCalls`, `calls.updateCalls` incremented at actual API call sites, reported as `totalCalls`
|
||||
- **One-shot semantics:** Start makes exactly 1 call; each Update iteration makes exactly 1 call; rejection returns immediately with no retry path
|
||||
- **Rejection diagnostics:** `rejectedProposalSnapshot` preserved and logged when present in Update rejection
|
||||
- **Explicit documentation:** Comments clarify "exactly one", "no retry", "bounded" semantics
|
||||
|
||||
## No-Retry Invariant Verification
|
||||
|
||||
### Semantic retries present: NO
|
||||
No loop, no attempt counter, no run-until-success. Rejection at any stage causes immediate chain stop via `return`.
|
||||
|
||||
### Transport retries present: NO
|
||||
The harness makes raw `fetch()` calls with no retry wrapper. Any transport-level retry would need to be added explicitly (and is not part of this task).
|
||||
|
||||
### Implicit second start/update: NO
|
||||
Start is called exactly once at the top level. Updates are loop-bound by `config.maxUpdates`. Each loop iteration makes exactly one call.
|
||||
|
||||
### Sequential flow enforcement
|
||||
- Update 1 rejection → chain stops, Update 2 never called
|
||||
- Update 1 success → Update 2 may be called exactly once (if `maxUpdates >= 2` and `answers.length >= 2`)
|
||||
|
||||
## Test Results
|
||||
|
||||
All 8 deterministic harness tests pass via synchronous simulation mirror:
|
||||
|
||||
| Case | Description | Result |
|
||||
|------|-------------|--------|
|
||||
| 1 | Start success → exactly 1 Start call | PASS |
|
||||
| 2 | Start failure → exactly 1 Start call, no retry | PASS |
|
||||
| 3 | Update success → exactly 1 Update call | PASS |
|
||||
| 4 | `proposal_compatibility` rejection → exactly 1 Update call, unchanged rejection | PASS |
|
||||
| 5 | Update 1 rejection → Update 2 never called | PASS |
|
||||
| 6 | Update 1 success → Update 2 called exactly once when explicitly requested | PASS |
|
||||
| 7 | Call counters equal actual mocked API invocations | PASS |
|
||||
| 8 | No semantic retry after HTTP 422/valid rejection | PASS |
|
||||
|
||||
**Test totals:** 8 passed, 0 failed.
|
||||
**Ollama calls made:** 0.
|
||||
|
||||
## What This Tooling Change Guarantees
|
||||
|
||||
1. Future live experiment runs via the canonical harness are physically incapable of consuming more API calls than explicitly configured.
|
||||
2. Each Start request = exactly one HTTP call (countered by `startCalls`).
|
||||
3. Each Update request = exactly one HTTP call (countered by `updateCalls`).
|
||||
4. Rejections stop the chain immediately without retry for any semantic outcome (proposal_compatibility, validation failure, etc.).
|
||||
5. Call accounting always reflects actual API invocations at the point of calling, not inferred from success/failure results.
|
||||
6. `rejectedProposalSnapshot` diagnostics are preserved and reported when present in Update rejection responses.
|
||||
|
||||
## What This Does NOT Guarantee
|
||||
|
||||
1. That production reasoning correctness is improved (no production code changed).
|
||||
2. That cold-start variance in node counts is resolved (start graph stability remains an open issue from Experiments 57J.30, 57J.29).
|
||||
3. That semantic validation outcomes change (only the harness wrapper changed, not any reasoning logic or validator).
|
||||
4. That transport-level HTTP failures are handled (no transport retry was added by this task).
|
||||
5. That zero-node proposals (from Experiment 57J.34) are prevented — a structurally empty proposal can still pass semantic validation.
|
||||
|
||||
## Files Changed
|
||||
|
||||
- `scripts/reproduce-multi-turn-investigation.mjs` — harness hardening: bounded execution, call accounting, no-retry semantics
|
||||
- `tests/reproduce-multi-turn-investigation.harness.test.js` — 8 deterministic harness behavior tests
|
||||
- `docs/experiment-57j35.md` — this document
|
||||
- `docs/current-handoff.md` — handoff entry
|
||||
|
||||
## Production Impact Assessment
|
||||
|
||||
Production reasoning code: **UNCHANGED**
|
||||
Production API behaviour: **UNCHANGED**
|
||||
Prompts: **UNCHANGED**
|
||||
Schemas: **UNCHANGED**
|
||||
Provider/model integration: **UNCHANGED**
|
||||
|
||||
This is a pure harness/tooling change. No production paths are affected.
|
||||
Reference in New Issue
Block a user