confidence-engine/docs/experiment-57j35.md

Experiment 57J.35 — No-Retry Live Experiment Harness Enforcement

Objective

Make the canonical live harness (scripts/reproduce-multi-turn-investigation.mjs) physically incapable of hidden retries. Enforce one-shot execution semantics:

• One requested Start = exactly one /api/cases/start call
• One requested Update = exactly one /api/cases/update call
• A rejection is returned immediately and is never retried implicitly

This directly addresses the protocol breach from Experiment 57J.32 where an implicit retry loop consumed multiple Update calls per trial, contaminating evidence.

Pre-written expectation recorded: YES

The canonical harness must enforce one-call/no-retry semantics for all bounded experiments. Future prompts may rely on this; Claude must not create supplementary retry scripts during bounded experiments.

Protocol breach referenced: Experiment 57J.32

Experiment 57J.32 documented a protocol breach where the original harness used an implicit retry loop for accepted results — meaning each "trial" potentially consumed multiple Update calls. This experiment enforces that the canonical apparatus cannot repeat that error.

Starting HEAD

06f67da — experiment: observe guarded multi-turn progress

Original Harness (commit 7533e47)

The original harness was a hardcoded sequential script:

Start → Update 1 → Update 2

Issues with original:

• No configuration system (scenario and answers hardcoded)
• No call accounting
• No rejection diagnostics (rejectedProposalSnapshot not handled)
• Not flexible for bounded experiments (always exactly 2 updates)
• However: no explicit retry loops existed in the original — but the lack of bounded config allowed ad-hoc supplementary scripts with retries (as happened in 57J.32)

Changes to Canonical Harness

Before (original, commit 7533e47)

• Hardcoded sequential flow: Start → Update 1 → Update 2
• No configuration object
• No call accounting
• No rejection diagnostics
• No explicit "no retry" documentation

After (current working tree)

• Bounded execution configuration: config.maxUpdates + config.answers[] positional mapping
• Call accounting: calls.startCalls, calls.updateCalls incremented at actual API call sites, reported as totalCalls
• One-shot semantics: Start makes exactly 1 call; each Update iteration makes exactly 1 call; rejection returns immediately with no retry path
• Rejection diagnostics: rejectedProposalSnapshot preserved and logged when present in Update rejection
• Explicit documentation: Comments clarify "exactly one", "no retry", "bounded" semantics

No-Retry Invariant Verification

Semantic retries present: NO

No loop, no attempt counter, no run-until-success. Rejection at any stage causes immediate chain stop via return.

Transport retries present: NO

The harness makes raw fetch() calls with no retry wrapper. Any transport-level retry would need to be added explicitly (and is not part of this task).

Implicit second start/update: NO

Start is called exactly once at the top level. Updates are loop-bound by config.maxUpdates. Each loop iteration makes exactly one call.

Sequential flow enforcement

• Update 1 rejection → chain stops, Update 2 never called
• Update 1 success → Update 2 may be called exactly once (if maxUpdates >= 2 and answers.length >= 2)

Test Results

All 8 deterministic harness tests pass via synchronous simulation mirror:

Case	Description	Result
1	Start success → exactly 1 Start call	PASS
2	Start failure → exactly 1 Start call, no retry	PASS
3	Update success → exactly 1 Update call	PASS
4	proposal_compatibility rejection → exactly 1 Update call, unchanged rejection	PASS
5	Update 1 rejection → Update 2 never called	PASS
6	Update 1 success → Update 2 called exactly once when explicitly requested	PASS
7	Call counters equal actual mocked API invocations	PASS
8	No semantic retry after HTTP 422/valid rejection	PASS

Test totals: 8 passed, 0 failed. Ollama calls made: 0.

What This Tooling Change Guarantees

1. Future live experiment runs via the canonical harness are physically incapable of consuming more API calls than explicitly configured.
2. Each Start request = exactly one HTTP call (countered by startCalls).
3. Each Update request = exactly one HTTP call (countered by updateCalls).
4. Rejections stop the chain immediately without retry for any semantic outcome (proposal_compatibility, validation failure, etc.).
5. Call accounting always reflects actual API invocations at the point of calling, not inferred from success/failure results.
6. rejectedProposalSnapshot diagnostics are preserved and reported when present in Update rejection responses.

What This Does NOT Guarantee

1. That production reasoning correctness is improved (no production code changed).
2. That cold-start variance in node counts is resolved (start graph stability remains an open issue from Experiments 57J.30, 57J.29).
3. That semantic validation outcomes change (only the harness wrapper changed, not any reasoning logic or validator).
4. That transport-level HTTP failures are handled (no transport retry was added by this task).
5. That zero-node proposals (from Experiment 57J.34) are prevented — a structurally empty proposal can still pass semantic validation.

Files Changed

• scripts/reproduce-multi-turn-investigation.mjs — harness hardening: bounded execution, call accounting, no-retry semantics
• tests/reproduce-multi-turn-investigation.harness.test.js — 8 deterministic harness behavior tests
• docs/experiment-57j35.md — this document
• docs/current-handoff.md — handoff entry

Production Impact Assessment

Production reasoning code: UNCHANGED Production API behaviour: UNCHANGED Prompts: UNCHANGED Schemas: UNCHANGED Provider/model integration: UNCHANGED

This is a pure harness/tooling change. No production paths are affected.