confidence-engine/docs/experiment-57j31.md

Experiment 57J.31 — Rejected Proposal Diagnostics Integration

Objective

Provide diagnostic visibility into the parsed proposal that fails at proposal_compatibility (Experiment 57J.30's blocking gap). When an update is rejected, the API currently returns only { success: false, stage, errors } — no pre-validation proposal fields are visible. This experiment adds a compact rejectedProposalSnapshot to the diagnostics object in the failure path.

Pre-written expectation recorded: YES

Adding a snapshot of key proposal fields (answerMeaning, addedNodes, addedEdges, updatedNodes, resolvedUnknownNodeIds) to the rejection diagnostics will allow developers to determine whether the rejection was caused by stronger answerMeaning category language or a different structural element — without needing to modify production code that controls which proposals are rejected. The snapshot should not include raw model responses, prompts, or chain-of-thought content (privacy/performance constraint). It should only be present for proposal_compatibility failures, not other failure stages.

Configured apparatus

• Branch: feature/rejected-proposal-diagnostics-v0.16
• HEAD at experiment start: 7937767 — experiment: capture proposal-boundary live variance
• Host/model: qwen-claude:latest at http://192.168.1.111:11434
• Ollama calls: 0 (diagnostic instrumentation does not invoke the model)

Implementation

orchestrator.js change (1 location, lines ~690–725)

In the !applicationResult.success return path of updateCaseWithDependencies:

// Compact rejected-proposal snapshot for proposal_compatibility diagnostics.
const rejectedProposalSnapshot =
  applicationResult.stage === "proposal_compatibility" && parsedProposal.proposal
    ? {
        answerMeaning: parsedProposal.proposal.answerMeaning
          ? {
              userSupportedMeaning: ...,
              possibleInference: ...,
            }
          : null,
        updatedNodes: (parsedProposal.proposal.updatedNodes ?? []).map(n => ({ nodeId, newValue })),
        resolvedUnknownNodeIds: ...,
        addedNodes: (parsedProposal.proposal.addedNodes ?? []).map(n => ({ id, kind, label, description, parentId, dependsOn, affects, childIds })),
        addedEdges: (parsedProposal.proposal.addedEdges ?? []).map(e => ({ fromNodeId, toNodeId, relationship })),
      }
    : null;

// Then in the return diagnostics object:
...(rejectedProposalSnapshot && { rejectedProposalSnapshot }),

Key design constraints

1. Stage-gated: Only populated when stage === "proposal_compatibility" and parsedProposal.proposal is truthy. Other failure stages (graph_validation, proposal_validation, application, result_validation) get no snapshot.
2. Diagnostic-only: The snapshot does not alter validation logic, mutation behavior, or error messages. It is purely evidence for developers.
3. Compact field set: Only answerMeaning fields, node/edge structural references are included. No raw model response, no prompt, no chain_of_thought.
4. No production code changed outside orchestrator.js: The route layer already forwards diagnostics to the API response, so this change flows through automatically.

Validation approach

Automated tests (8 new + 2 existing-verification tests)

1. tests/graph/rejected-proposal-snapshot.test.js (7 tests, all pass):

   • "includes rejectedProposalSnapshot when stage is proposal_compatibility" — confirms snapshot presence for the target failure stage.
   • "exposes answerMeaning.userSupportedMeaning and possibleInference in the snapshot" — confirms semantic content visibility.
   • "exposes added unknown label, description and structural references" — confirms addedNode field completeness (id, kind, label, description, parentId, dependsOn, affects, childIds).
   • "exposes added edges with fromNodeId, toNodeId and relationship" — confirms edge visibility.
   • "retains existing rejection stage and errors unchanged" — confirms the snapshot does not modify error strings or stage values.
   • "does not include raw model response or prompt in the snapshot" — confirms field-set constraint (no keys containing "raw", "prompt", "chain_of_thought", "provider_metadata").
   • "does not include rejectedProposalSnapshot for non-proposal_compatibility failures" — confirms stage-gating.

2. tests/graph/apply-proposal.test.js (2 new verification tests):

   • "identical rejected fixture still rejects" — confirms the rejection path in applyValidatedProposal is unchanged (same errors, no mutations).
   • "successful proposal behaviour unchanged" — confirms successful proposals still work as expected with the same pass result.

3. tests/app/api/cases-update-route.test.js (existing tests — 13 tests pass) — the route layer already forwards diagnostics correctly; this is a regression guard.

Test results

✓ tests/graph/rejected-proposal-snapshot.test.js (7 tests) 9ms
✓ tests/graph/apply-proposal.test.js (64 tests) 208ms  [includes 2 new]
✓ tests/app/api/cases-update-route.test.js (13 tests) 113ms
Total: 84 passed, 0 failed

Pre-existing failure confirmed independent of this change:
✗ tests/graph/orchestrator.test.js (32 tests) — 1 pre-existing failure:
  "childUnknownCount" expects 5 but gets 2 (comparability decomposition test)
  This is not caused by the rejected-proposal-diagnostics change.

Blocked observations

No live model calls were made in this experiment. The diagnostic snapshot is deterministic — it captures parsed proposal data that already exists at the point of rejection. No Ollama inference is needed.

What remains unproven:

• Live rejection analysis: Whether the actual rejected trial from Experiment 57J.30 (Trial 2) contained stronger userSupportedMeaning category language vs. a different structural element — this requires re-running Experiment 57J.30 with the new diagnostics field now available in the API response.
• Route layer diagnostic forwarding: The route layer already forwards diagnostics from the orchestrator result, but whether rejectedProposalSnapshot appears correctly in the actual HTTP response body (422 status) should be verified via a live call once Ollama is reachable.

What this establishes

1. The blocking visibility gap identified in Experiment 57J.30 is now closed at the code level. Developers can inspect diagnostics.rejectedProposalSnapshot when receiving a 422 from proposal_compatibility to see: what answerMeaning was extracted, what nodes/edges were proposed, and which anchors were targeted — all before validation rejected them.
2. No validation or mutation behavior changed. The rejection itself (errors, stage, HTTP status code) is identical. Only the diagnostic surface is expanded.
3. Stage gating ensures no snapshot leakage for other failure types. Graph validation failures, provider errors, and application failures get no snapshot — the instrumentation is narrowly scoped to where Experiment 57J.30 identified the gap: proposal_compatibility.

Production code changed

• lib/graph/orchestrator.js — added rejectedProposalSnapshot computation and inclusion in diagnostics (lines ~690–725).
• No changes to schema, route layer, validation logic, or mutation paths.

Prompt changed: NO

Schema changed: NO

Temporary instrumentation removed: YES (no instrumentation added)

Ollama calls beyond budget: 0

Documentation updated: YES