experiment: add rejected proposal diagnostics to failure path
Adds rejectedProposalSnapshot to orchestrator diagnostics for proposal_compatibility rejections — exposing answerMeaning (userSupportedMeaning, possibleInference), addedNodes structural fields, addedEdges structural fields, updatedNodes summaries, and resolvedUnknownNodeIds. Diagnostic evidence only; does not alter validation, mutation, or error messages. Stage-gated to proposal_compatibility only.
This commit is contained in:
@@ -0,0 +1,110 @@
|
||||
# Experiment 57J.31 — Rejected Proposal Diagnostics Integration
|
||||
|
||||
## Objective
|
||||
|
||||
Provide diagnostic visibility into the parsed proposal that fails at `proposal_compatibility` (Experiment 57J.30's blocking gap). When an update is rejected, the API currently returns only `{ success: false, stage, errors }` — no pre-validation proposal fields are visible. This experiment adds a compact `rejectedProposalSnapshot` to the diagnostics object in the failure path.
|
||||
|
||||
## Pre-written expectation recorded: YES
|
||||
|
||||
> Adding a snapshot of key proposal fields (answerMeaning, addedNodes, addedEdges, updatedNodes, resolvedUnknownNodeIds) to the rejection diagnostics will allow developers to determine whether the rejection was caused by stronger answerMeaning category language or a different structural element — without needing to modify production code that controls which proposals are rejected. The snapshot should not include raw model responses, prompts, or chain-of-thought content (privacy/performance constraint). It should only be present for `proposal_compatibility` failures, not other failure stages.
|
||||
|
||||
## Configured apparatus
|
||||
|
||||
- **Branch:** `feature/rejected-proposal-diagnostics-v0.16`
|
||||
- **HEAD at experiment start:** `7937767` — experiment: capture proposal-boundary live variance
|
||||
- **Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
- **Ollama calls:** 0 (diagnostic instrumentation does not invoke the model)
|
||||
|
||||
## Implementation
|
||||
|
||||
### orchestrator.js change (1 location, lines ~690–725)
|
||||
|
||||
In the `!applicationResult.success` return path of `updateCaseWithDependencies`:
|
||||
|
||||
```javascript
|
||||
// Compact rejected-proposal snapshot for proposal_compatibility diagnostics.
|
||||
const rejectedProposalSnapshot =
|
||||
applicationResult.stage === "proposal_compatibility" && parsedProposal.proposal
|
||||
? {
|
||||
answerMeaning: parsedProposal.proposal.answerMeaning
|
||||
? {
|
||||
userSupportedMeaning: ...,
|
||||
possibleInference: ...,
|
||||
}
|
||||
: null,
|
||||
updatedNodes: (parsedProposal.proposal.updatedNodes ?? []).map(n => ({ nodeId, newValue })),
|
||||
resolvedUnknownNodeIds: ...,
|
||||
addedNodes: (parsedProposal.proposal.addedNodes ?? []).map(n => ({ id, kind, label, description, parentId, dependsOn, affects, childIds })),
|
||||
addedEdges: (parsedProposal.proposal.addedEdges ?? []).map(e => ({ fromNodeId, toNodeId, relationship })),
|
||||
}
|
||||
: null;
|
||||
|
||||
// Then in the return diagnostics object:
|
||||
...(rejectedProposalSnapshot && { rejectedProposalSnapshot }),
|
||||
```
|
||||
|
||||
### Key design constraints
|
||||
|
||||
1. **Stage-gated:** Only populated when `stage === "proposal_compatibility"` and `parsedProposal.proposal` is truthy. Other failure stages (graph_validation, proposal_validation, application, result_validation) get no snapshot.
|
||||
2. **Diagnostic-only:** The snapshot does not alter validation logic, mutation behavior, or error messages. It is purely evidence for developers.
|
||||
3. **Compact field set:** Only answerMeaning fields, node/edge structural references are included. No raw model response, no prompt, no chain_of_thought.
|
||||
4. **No production code changed outside orchestrator.js:** The route layer already forwards `diagnostics` to the API response, so this change flows through automatically.
|
||||
|
||||
## Validation approach
|
||||
|
||||
### Automated tests (8 new + 2 existing-verification tests)
|
||||
|
||||
1. **tests/graph/rejected-proposal-snapshot.test.js** (7 tests, all pass):
|
||||
- "includes rejectedProposalSnapshot when stage is proposal_compatibility" — confirms snapshot presence for the target failure stage.
|
||||
- "exposes answerMeaning.userSupportedMeaning and possibleInference in the snapshot" — confirms semantic content visibility.
|
||||
- "exposes added unknown label, description and structural references" — confirms addedNode field completeness (id, kind, label, description, parentId, dependsOn, affects, childIds).
|
||||
- "exposes added edges with fromNodeId, toNodeId and relationship" — confirms edge visibility.
|
||||
- "retains existing rejection stage and errors unchanged" — confirms the snapshot does not modify error strings or stage values.
|
||||
- "does not include raw model response or prompt in the snapshot" — confirms field-set constraint (no keys containing "raw", "prompt", "chain_of_thought", "provider_metadata").
|
||||
- "does not include rejectedProposalSnapshot for non-proposal_compatibility failures" — confirms stage-gating.
|
||||
|
||||
2. **tests/graph/apply-proposal.test.js** (2 new verification tests):
|
||||
- "identical rejected fixture still rejects" — confirms the rejection path in applyValidatedProposal is unchanged (same errors, no mutations).
|
||||
- "successful proposal behaviour unchanged" — confirms successful proposals still work as expected with the same pass result.
|
||||
|
||||
3. **tests/app/api/cases-update-route.test.js** (existing tests — 13 tests pass) — the route layer already forwards diagnostics correctly; this is a regression guard.
|
||||
|
||||
### Test results
|
||||
|
||||
```
|
||||
✓ tests/graph/rejected-proposal-snapshot.test.js (7 tests) 9ms
|
||||
✓ tests/graph/apply-proposal.test.js (64 tests) 208ms [includes 2 new]
|
||||
✓ tests/app/api/cases-update-route.test.js (13 tests) 113ms
|
||||
Total: 84 passed, 0 failed
|
||||
|
||||
Pre-existing failure confirmed independent of this change:
|
||||
✗ tests/graph/orchestrator.test.js (32 tests) — 1 pre-existing failure:
|
||||
"childUnknownCount" expects 5 but gets 2 (comparability decomposition test)
|
||||
This is not caused by the rejected-proposal-diagnostics change.
|
||||
```
|
||||
|
||||
## Blocked observations
|
||||
|
||||
**No live model calls were made in this experiment.** The diagnostic snapshot is deterministic — it captures parsed proposal data that already exists at the point of rejection. No Ollama inference is needed.
|
||||
|
||||
What remains unproven:
|
||||
- **Live rejection analysis:** Whether the actual rejected trial from Experiment 57J.30 (Trial 2) contained stronger `userSupportedMeaning` category language vs. a different structural element — this requires re-running Experiment 57J.30 with the new diagnostics field now available in the API response.
|
||||
- **Route layer diagnostic forwarding:** The route layer already forwards `diagnostics` from the orchestrator result, but whether `rejectedProposalSnapshot` appears correctly in the actual HTTP response body (422 status) should be verified via a live call once Ollama is reachable.
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. **The blocking visibility gap identified in Experiment 57J.30 is now closed at the code level.** Developers can inspect `diagnostics.rejectedProposalSnapshot` when receiving a 422 from proposal_compatibility to see: what answerMeaning was extracted, what nodes/edges were proposed, and which anchors were targeted — all before validation rejected them.
|
||||
2. **No validation or mutation behavior changed.** The rejection itself (errors, stage, HTTP status code) is identical. Only the diagnostic surface is expanded.
|
||||
3. **Stage gating ensures no snapshot leakage for other failure types.** Graph validation failures, provider errors, and application failures get no snapshot — the instrumentation is narrowly scoped to where Experiment 57J.30 identified the gap: proposal_compatibility.
|
||||
|
||||
## Production code changed
|
||||
|
||||
- `lib/graph/orchestrator.js` — added rejectedProposalSnapshot computation and inclusion in diagnostics (lines ~690–725).
|
||||
- No changes to schema, route layer, validation logic, or mutation paths.
|
||||
|
||||
## Prompt changed: NO
|
||||
## Schema changed: NO
|
||||
## Temporary instrumentation removed: YES (no instrumentation added)
|
||||
## Ollama calls beyond budget: 0
|
||||
|
||||
## Documentation updated: YES
|
||||
Reference in New Issue
Block a user