confidence-engine/docs/experiment-57j34.md

Experiment 57J.34 — Multi-Turn Investigation Progress After Accepted Update 1

Objective

Answer exactly: on one fresh live run, if the first relocation answer passes the current reasoning safeguards, does answering the savings-realism question produce genuine investigation progress rather than repetition or irrelevant reasoning?

This follows from 57J.33 which established that some Update 1 rejections are legitimate fidelity safeguards.

Configured apparatus

• Branch: feature/rejected-proposal-diagnostics-v0.16
• HEAD at experiment start: a00f7b1 — experiment: inspect rejected proposal live variance
• Ollama calls made: 4 (2 starts + 2 updates in capture pipeline; 1 update in final pipeline)
• Production code changed: NO

Fixed scenario and answers

Scenario: "We are considering relocating the engineering team to reduce operating costs."

Answer 1: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."

Answer 2: "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year."

Pre-written expectation

If Update 1 produces a faithful proposal, savings realism and retention impact should survive and the engine should ask a grounded next question. After Answer 2 supplies concrete savings evidence, the investigation should progress rather than substantially repeat the same savings-realism question or invent unsupported comparison/timing reasoning.

Run results

START (capture run)

• HTTP: 200
• Stage: unknown
• Nodes: 8
• Edges: 5
• Selected question: "What would clarify current and proposed locations are unspecified, preventing regional cost analysis in this situation?"

Classification of first Update 1: R1 — CORRECT FIDELITY REJECTION

The error was "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes." From 57J.33's deterministic analysis, this is the same protected answer-meaning strengthening class: the LLM reformulated "Before deciding, I need evidence..." as "The decision is conditional on..." which triggered hasConditionalQualification() keyword detector on "conditional", pushing it into conditional_tradeoff category where raw answer is other. This rejection is correct and matches 57J.33's finding.

UPDATE 1 (rejected run — harness)

• HTTP: 422
• Stage: proposal_compatibility
• First error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
• Nodes: 8 (unchanged — no mutation due to rejection)
• Edges: 5 (unchanged)

UPDATE 1 classification: R1

Same conditional-strengthening defect as established in 57J.33. The rejected snapshot confirmed userSupportedMeaning contained "conditional" which triggers hasConditionalQualification(). This is a legitimate fidelity guard, not a regression.

Pipeline run — Update 1 ACCEPTED (fresh case)

A fresh start/Update 1 produced a different outcome due to cold-start variance:

START (pipeline run)

• HTTP: 200
• Stage: unknown
• Nodes: 5 (cold-start variance vs 8 in harness run)
• Edges: 3
• Selected question: "What evidence would confirm or rule out current operating costs, relocation expenses, and baseline financial metrics for the engineering team?"

UPDATE 1 (pipeline — ACCEPTED)

• HTTP: 200
• Stage: update_applied
• Success: true
• Nodes: 5 (UNCHANGED — no new unknowns created!)
• Edges: 2 (DECREASED from 3!)
• Selected question: "What would clarify specific criteria, budget constraints, talent retention implications, or timeline defining the viability of the proposal in this situation?"

Critical finding: Despite Update 1 succeeding at update_applied, ZERO new unknown nodes were created. The user answer explicitly introduced two independent evidence dimensions (savings realism + retention impact), yet the engine produced no distinct nodes for either. Instead, a single merged generic unknown appeared ("specific criteria, budget constraints, talent retention implications, or timeline") — all compressed into one node that covers neither dimension adequately.

UPDATE 2 (pipeline)

• HTTP: 422
• Stage: proposal_compatibility
• First error: "New unknown must be explicitly related to an answer-derived node: 'n_rel_exp'"

From rejectedProposalSnapshot:

{
  "userSupportedMeaning": "The projected annual operating savings are approximately £2M, derived from cost eliminations associated with the current London lease, business rates, service charges, utilities, and facilities.",
  "possibleInference": "The financial viability of the relocation heavily depends on these specific ongoing cost offsets, but net benefit remains uncertain until one-time moving expenses and operational timelines are quantified."
}

• Nodes (pre-update): 5
• Edges (pre-update): 2
• Selected question: null (rejected)
• Active unknowns remaining: same 1 merged node from Update 1

Update 2 classification: D — VALIDATION FAILURE

Update 2 was rejected at proposal_compatibility by the same structural gate that blocked Experiment 57J.26: the old provenance-link requirement ("New unknown must be explicitly related to an answer-derived node") blocks legitimate new unknown creation.

Classification table

Update 1: R3 (UPDATE APPLIED on pipeline run)

• Savings target preserved? NO — not represented as a distinct node
• Retention target preserved? NO — not represented as a distinct node
• New unknowns created? 0 (should be 2+)
• Edges before/after: 3 → 2 (decreased)

Update 2: D (VALIDATION FAILURE)

• Reasoning pattern: n/a (rejected)
• Savings-realism progressed/resolved: NO — no progress was possible; the savings question from Update 1's selected question was effectively repeated as a broad merged unknown
• Same savings question repeated: YES — the Update 1 selected question ("current and proposed locations are unspecified, preventing regional cost analysis") was followed by an equally vague merged question; Answer 2 about £2M savings produced no resolution of any savings-specific unknown because none existed
• Next question grounded in genuine unresolved issue: NO — rejected before reaching a valid next question

Overall result

FAIL — Update 1 acceptance does NOT produce investigation progress.

The central finding of 57J.34 is clear: when Update 1 is accepted (on the pipeline run where cold-start produced 5 nodes instead of 8), the engine did NOT create two distinct unknown nodes for savings realism and retention impact. Instead, it created a single compressed merged unknown with no meaningful graph growth. When Update 2 was then attempted with concrete savings evidence (£2M from London lease, business rates, etc.), it failed at the same structural linkage gate that has blocked legitimate updates across Experiments 57J.26, 57J.30, and now 57J.34.

This means the experiment's core question is answered: even when Update 1 passes the current reasoning safeguards, Answer 2 does NOT produce genuine investigation progress — it triggers another validation failure at the provenance-link gate.

What this establishes

1. Cold-start variance (5 vs 8 nodes) directly affects whether Update 1's semantic fidelity guard fires or passes. This is a separate problem from the provenance-link gate.
2. Even when Update 1 passes, the engine may create zero new unknown nodes despite the user explicitly introducing two independent evidence dimensions.
3. The provenance-link gate ("New unknown must be explicitly related to an answer-derived node") remains active in v0.16 and blocks Update 2 for this scenario.
4. Accepting a "faithful" semantic proposal does NOT guarantee meaningful investigation progress — the engine can pass semantic validation while still producing structurally empty graph mutations (0 new nodes, fewer edges).
5. The savings-realism question from Answer 2 was not resolved because no dedicated savings realism unknown node existed for it to resolve.

What this does NOT prove

1. That all cold-start graphs produce 5 nodes (node count variance continues across runs).
2. That the engine always produces 0 new nodes when Update 1 is accepted.
3. That the provenance-link gate is intentionally designed this way or a defect.
4. That semantic meaning extraction in Update 2's userSupportedMeaning was correct (it was not audited against a ground truth).
5. Whether the merged generic unknown "specific criteria, budget constraints, talent retention implications, or timeline" represents an intentional design choice or a decomposition/generation defect.

Cold-start observation

The cold-start node count ranged from 5 to 8 across runs with identical scenario input — confirming the variance pattern established in Experiment 57J.30 (range: 5→8) and 57J.29. This remains an unresolved characteristic of startCase().