confidence-engine/docs/experiment-57j36.md

Experiment 57J.36 — Multi-Turn Investigation Progress After Accepted Update 1 (Clean Run)

Objective

Run one clean case to answer: If the first relocation answer produces an acceptable proposal, does answering the resulting savings-realism question make genuine investigation progress on the next turn?

This is a hardened replacement for 57J.34/35, using only the canonical harness with bounded execution and no-retry semantics.

Pre-written expectation recorded: YES

If Update 1 produces a faithful proposal, the user's two evidence needs should remain represented as genuine unresolved issues and the engine should select a grounded next question. If Answer 2 then supplies the requested savings evidence, the investigation should progress rather than substantially repeat savings realism or move into unsupported comparison/timing reasoning.

If Update 1 is correctly rejected for semantic strengthening, that is a valid protected outcome and the experiment stops there. Do not retry to obtain an accepted case.

Starting HEAD

4998de5 — tooling: enforce no-retry live experiment harness

Fixed Inputs

Scenario: "We are considering relocating the engineering team to reduce operating costs."

Answer 1: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."

Answer 2: "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year."

Harness Configuration

• maxUpdates = 2
• config.answers[0] → Answer 1
• config.answers[1] → Answer 2
• No loops, no attempts, single execution path

Results

START

• HTTP: 200
• Stage: unknown
• Nodes: 10
• Edges: 5
• Selected question: "What would clarify total projected costs at the new location, including one-time relocation expenses and long-term savings in this situation?"

UPDATE 1

• HTTP: 422
• Stage: proposal_compatibility
• First error: "Update contains no meaningful change"
• Nodes: 10 (unchanged)
• Edges: 5 (unchanged)
• Selected question: null
• Savings realism: UNCLEAR
• Retention impact: UNCLEAR

Rejected proposal snapshot:

{
  "answerMeaning": {
    "userSupportedMeaning": "Before deciding on relocation, the user requires two specific pieces of evidence: verification that projected office savings are realistic, and assurance that the move will not materially increase the loss of key engineers.",
    "possibleInference": null
  },
  "updatedNodes": [],
  "resolvedUnknownNodeIds": [],
  "addedNodes": [],
  "addedEdges": []
}

Update 1 classification: U1-B — DIFFERENT REJECTION

The rejection is for "Update contains no meaningful change" at proposal_compatibility, not for semantic strengthening. The LLM produced a null structural proposal (zero addedNodes, zero addedEdges) even though the answer clearly introduced two new evidence dimensions. This is neither a correct fidelity rejection nor an applied proposal — it is a structurally empty proposal rejected by a different gate.

UPDATE 2

• Reached: NO

Call Accounting

• startCalls: 1
• updateCalls: 1
• totalCalls: 2
• Valid maximum: 3 ✓

Supplementary scripts used: NO

Retries: 0

Classification

U1-B — DIFFERENT REJECTION. Rejected for "Update contains no meaningful change" at the proposal_compatibility stage. This differs from:

• U1-A (correct fidelity rejection): no semantic strengthening was present in userSupportedMeaning
• U1-C (applied with both dimensions): no nodes or edges were added at all
• U1-D (applied but degraded): nothing was applied

The LLM's answer meaning extraction was semantically faithful (preserved both evidence dimensions), but produced zero structural change — no addedNodes, no addedEdges, no resolvedUnknownNodeIds, no updatedNodes. The proposal compatibility gate correctly blocked a structurally empty update.

What this clean run establishes

1. When the LLM produces a structurally empty proposal (zero additions) even with semantically faithful answer meaning, the proposal_compatibility gate rejects it with "Update contains no meaningful change" — a valid protection against no-op updates.
2. The LLM did not strengthen meaning beyond the raw answer in this run (U1-A would have been appropriate if strengthening were present).
3. Cold-start produced 10 nodes (different from prior runs: Ex 57J.34 got 6–8; Ex 57J.32 got 5–8), confirming cold-start node variance persists.

What it does NOT prove

1. Whether the LLM can produce a structurally non-empty faithful proposal that passes proposal_compatibility (the structural creation step may be separately impaired).
2. That downstream progress on Update 2 would occur even with an accepted proposal.
3. Run-to-run stability of node counts or proposal structure for this scenario.
4. Whether the "no meaningful change" rejection is desirable behaviour when the user clearly introduces new information but the model fails to act on it structurally.

Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434

Production code changed: NO

Prompt changed: NO

Canonical harness restored: YES

Hardened no-retry behaviour preserved: YES

Dev server disturbed: NO

Ollama calls beyond harness count: 0