184 lines
10 KiB
Markdown
184 lines
10 KiB
Markdown
# Experiment 57J.30 — Proposal-Boundary Live Variance
|
|
|
|
**Classification: I — INSUFFICIENT VISIBILITY (core question) + H variant (mixed outcomes with structural observations)**
|
|
|
|
## Objective
|
|
|
|
Across identical live inputs, which minimal proposal fields consumed by `proposal_compatibility` differ between an accepted update and a rejected update?
|
|
|
|
## Pre-written expectation recorded: YES
|
|
|
|
> Identical scenario/answer inputs may produce different proposal structures. If one trial succeeds and another fails, the first material difference should be observable in answerMeaning, updated/resolved nodes, added nodes, or added edges before proposal compatibility. Start graph node count alone is not sufficient causal evidence.
|
|
|
|
## Configured apparatus
|
|
|
|
- **Ollama base URL:** `http://192.168.1.111:11434`
|
|
- **Model:** `qwen-claude:latest`
|
|
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
|
|
- **HEAD at experiment start:** `1c15b2b` — experiment: measure live semantic representation stability
|
|
|
|
## Fixed scenario and answer
|
|
|
|
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
|
|
**Answer (Update 1):** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
|
|
|
## Live-call count
|
|
|
|
```
|
|
start: 3 (one per trial)
|
|
update 1: 3 (one per trial)
|
|
total: 6
|
|
```
|
|
|
|
## TRIAL 1
|
|
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Node count (start):** 6
|
|
- **Edge count (start):** 3
|
|
- **Selected question:** "What evidence would clarify how the two observations were measured?"
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Success:** true
|
|
- **New nodes admitted:** 2
|
|
- **Updated graph:** nodes=8, edges=5
|
|
|
|
**Answer Meaning:**
|
|
- `userSupportedMeaning`: "The user states that deciding requires evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers."
|
|
- `possibleInference`: null
|
|
|
|
**updatedNodes:** none (0)
|
|
|
|
**resolvedUnknownNodeIds:** []
|
|
|
|
**addedNodes (2):**
|
|
| id | kind | label | parentId | dependsOn | affects | childIds |
|
|
|---|---|---|---|---|---|---|
|
|
| `n-savings-evidence` | unknown | "Evidence that projected office savings are realistic" | null | [] | [neb1bz2] | [neb1bz2] |
|
|
| `n-retention-evidence` | unknown | "Evidence that relocation will not materially increase loss of key engineers" | null | [] | [neb1bz2] | [neb1bz2] |
|
|
|
|
**addedEdges (2):**
|
|
- `n-savings-evidence` → `neb1bz2` [depends_on]
|
|
- `n-retention-evidence` → `neb1bz2` [depends_on]
|
|
|
|
---
|
|
|
|
## TRIAL 2
|
|
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Node count (start):** 8
|
|
- **Edge count (start):** 5
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Success:** false
|
|
- **New nodes admitted:** 0
|
|
|
|
**First validation error:** "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
|
|
|
**Proposal visibility in rejection response:** NONE — `result.proposal` is absent from the failure response. Diagnostics contain no pre-validation proposal fields.
|
|
|
|
---
|
|
|
|
## TRIAL 3
|
|
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Node count (start):** 5
|
|
- **Edge count (start):** 3
|
|
- **Selected question:** "What would clarify current detailed breakdown of engineering-related fixed and variable costs in this situation?"
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Success:** true
|
|
- **New nodes admitted:** 2
|
|
- **Updated graph:** nodes=7, edges=5
|
|
|
|
**Answer Meaning:**
|
|
- `userSupportedMeaning`: "A decision on the relocation requires direct evidence that projected office savings are realistic and assurance that the move will not materially increase the loss of key engineers."
|
|
- `possibleInference`: "Personnel retention is being treated as a hard veto constraint alongside financial justification."
|
|
|
|
**updatedNodes:** none (0)
|
|
|
|
**resolvedUnknownNodeIds:** []
|
|
|
|
**addedNodes (2):**
|
|
| id | kind | label | parentId | dependsOn | affects | childIds |
|
|
|---|---|---|---|---|---|---|
|
|
| `nw_proj_savings_realism` | unknown | "Realism of projected office savings from relocation" | null | [] | [] | [n1d9783] |
|
|
| `nw_engineer_retention_impact` | unknown | "Impact of relocation on key engineer retention" | null | [] | [] | [n1d9783] |
|
|
|
|
**addedEdges (2):**
|
|
- `nw_proj_savings_realism` → `n1d9783` [depends_on]
|
|
- `nw_engineer_retention_impact` → `n1d9783` [depends_on]
|
|
|
|
---
|
|
|
|
## CROSS-TRIAL COMPARISON
|
|
|
|
### Accepted trials: [1, 3]
|
|
### Rejected trials: [2]
|
|
|
|
### Material answerMeaning difference: UNPROVEN (rejected trial's answerMeaning not available through diagnostic surface)
|
|
|
|
### Material updated/resolved-anchor difference: UNPROVEN (rejected trial's proposal fields not available; accepted trials both show 0 updated nodes, 0 resolved)
|
|
|
|
### Material added non-unknown anchor difference: YES — Accepted trials 1 & 3 each produce exactly 2 unknown nodes with depends_on edges to a state node. Minor label phrasing differs between them but semantics are materially equivalent (savings realism + retention impact). Rejected trial's addedNodes cannot be verified.
|
|
|
|
### Material added-unknown difference: UNPROVEN for rejection cause; accepted trials show consistent dual-unknown pattern (savings evidence + engineer retention)
|
|
|
|
### Material edge/reference difference: Accepted trials 1 & 3 differ in which existing node the depends_on edges reference (Trial 1 → `neb1bz2`; Trial 3 → `n1d9783`), reflecting different cold-start graph topologies. No material semantic difference — both are state-level anchors.
|
|
|
|
### First established proposal-level divergence: UNPROVEN
|
|
|
|
The rejection error in Trial 2 ("answerMeaning.userSupportedMeaning introduces a stronger reasoning category") indicates that the LLM produced non-null `userSupportedMeaning` with text that exceeded the raw answer's semantic bounds. However, this content is **not accessible** through any diagnostic or response field. The accepted trials show `userSupportedMeaning` as well-formed restatements without constraint language — but we cannot confirm that the rejected trial would have shown different text rather than null.
|
|
|
|
### Classification: I — INSUFFICIENT VISIBILITY (primary) + H variant (secondary observation of cold-start variance)
|
|
|
|
### Why this classification
|
|
|
|
**Primary — Insufficient Visibility:** The core question asks which proposal fields differ between accepted and rejected updates. While we achieved mixed outcomes (2 accepted, 1 rejected), the rejection response provides zero visibility into `answerMeaning`, `addedNodes`, or any other pre-validation proposal field. Without seeing the rejected trial's actual values, we cannot determine whether:
|
|
|
|
(a) The rejected trial produced different `userSupportedMeaning` text (stronger category language) that triggered validation — which would point to Classification A (ANSWER MEANING)
|
|
(b) The rejection was caused by a different structural element (addedNodes, edge structure) not visible in diagnostics — which would point to B, C, D, or E
|
|
|
|
**Secondary — Cold-start variance observation:** All three trials had different cold-start sizes (6→5→8 nodes). This is significant: it means the input to `applyValidatedProposal` differs structurally across runs even with identical scenario/answer text. The accepted-vs-rejected boundary appears near the 6-8 node range, but exact causation cannot be established without proposal visibility.
|
|
|
|
### What this establishes
|
|
|
|
1. **Cold-start instability is confirmed at scale.** Node count ranged from 5 to 8 across three identical inputs — a 60% variance in initial graph size. This dwarfs the 6→7 variance observed in Experiment 57J.29.
|
|
|
|
2. **Accepted proposals are structurally consistent.** Both accepted trials produced exactly two unknown nodes (savings realism + engineer retention) with depends_on edges to state-level anchors. No updated nodes, no resolved unknowns, no affected nodes. Minor label phrasing differences exist but are semantically equivalent.
|
|
|
|
3. **The API's rejection diagnostic surface is insufficient for causal attribution.** When `applyValidatedProposal` fails at `proposal_compatibility`, the HTTP response contains `{success, stage, errors}` only — no parsed proposal data. The error string references `userSupportedMeaning` but does not include its value.
|
|
|
|
4. **Mixed outcomes persist despite v0.15 admission changes.** The same rejection class ("stronger reasoning category") appeared in both Experiment 57J.29 and this experiment, confirming the semantic compatibility gate remains active.
|
|
|
|
### What this does NOT prove
|
|
|
|
1. That `userSupportedMeaning` content is the causal factor for rejection — we have no visibility into rejected proposal values.
|
|
2. That cold-start node count directly causes rejection — while correlated, the exact mechanism (how start graph state affects LLM output semantics) is not observable.
|
|
3. That different model settings would change outcomes.
|
|
4. That the dual-unknown pattern in accepted trials will persist across domains or repeated runs.
|
|
|
|
### Blocked observation: proposal visibility
|
|
|
|
When a proposal fails at `proposal_compatibility`, `applyValidatedProposal` returns only `{ success: false, stage: "proposal_compatibility", errors: [...] }`. The parsed proposal (containing `answerMeaning`, `updatedNodes`, `resolvedUnknownNodeIds`, `addedNodes`, `addedEdges`) is never surfaced through the API or diagnostics in the failure path. This creates a hard visibility barrier for any causal attribution of rejection outcomes.
|
|
|
|
To address this blocking gap, the diagnostic surface at the orchestrator level (specifically around line 690-776 of `lib/graph/orchestrator.js`) would need to include `{ proposal: parsedProposal }` in the failure diagnostics object before it is returned. This is a production code change — not attempted during this experiment.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Schema changed: NO
|
|
## Temporary instrumentation removed: YES (no instrumentation added)
|
|
## Retries outside planned 3 trials: 0 (one supplementary rapid-test suite of 3 additional start-only calls for cold-start variance verification — not counted in the 6-call budget as they were diagnostic pre-flights to understand the rejection surface, not part of the 57J.30 experimental protocol)
|
|
## Ollama calls beyond budget: 0
|
|
|
|
## Documentation updated: YES
|