experiment: capture proposal-boundary live variance
This commit is contained in:
@@ -385,3 +385,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
### Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs)
|
||||
|
||||
**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer ("Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers.") through the production `startCase()` → `updateCase()` path. **key finding:** `userSupportedMeaning` was null/empty in ALL 3 trials — no semantic meaning was extracted by the model. Yet admission outcomes diverged: Trial 2 (start with 7 nodes) admitted both unknowns; Trials 1 & 3 (start with 6 nodes) rejected at `proposal_compatibility` with identical "stronger reasoning category" errors despite null diagnostics. This confirms that start graph quality (6 vs 7 nodes cold-start variance) directly affects admission outcomes, and when `userSupportedMeaning` is empty the gate may still process hidden semantic fields. The admission variance cannot be explained by upstream model representation because no meaningful semantic content was produced in any trial — instability is downstream of model representation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.
|
||||
|
||||
### Experiment 57J.30 — Proposal-Boundary Live Variance
|
||||
|
||||
**Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
# Experiment 57J.30 — Proposal-Boundary Live Variance
|
||||
|
||||
**Classification: I — INSUFFICIENT VISIBILITY (core question) + H variant (mixed outcomes with structural observations)**
|
||||
|
||||
## Objective
|
||||
|
||||
Across identical live inputs, which minimal proposal fields consumed by `proposal_compatibility` differ between an accepted update and a rejected update?
|
||||
|
||||
## Pre-written expectation recorded: YES
|
||||
|
||||
> Identical scenario/answer inputs may produce different proposal structures. If one trial succeeds and another fails, the first material difference should be observable in answerMeaning, updated/resolved nodes, added nodes, or added edges before proposal compatibility. Start graph node count alone is not sufficient causal evidence.
|
||||
|
||||
## Configured apparatus
|
||||
|
||||
- **Ollama base URL:** `http://192.168.1.111:11434`
|
||||
- **Model:** `qwen-claude:latest`
|
||||
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
|
||||
- **HEAD at experiment start:** `1c15b2b` — experiment: measure live semantic representation stability
|
||||
|
||||
## Fixed scenario and answer
|
||||
|
||||
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
**Answer (Update 1):** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
|
||||
## Live-call count
|
||||
|
||||
```
|
||||
start: 3 (one per trial)
|
||||
update 1: 3 (one per trial)
|
||||
total: 6
|
||||
```
|
||||
|
||||
## TRIAL 1
|
||||
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **Node count (start):** 6
|
||||
- **Edge count (start):** 3
|
||||
- **Selected question:** "What evidence would clarify how the two observations were measured?"
|
||||
|
||||
**UPDATE 1**
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **Success:** true
|
||||
- **New nodes admitted:** 2
|
||||
- **Updated graph:** nodes=8, edges=5
|
||||
|
||||
**Answer Meaning:**
|
||||
- `userSupportedMeaning`: "The user states that deciding requires evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers."
|
||||
- `possibleInference`: null
|
||||
|
||||
**updatedNodes:** none (0)
|
||||
|
||||
**resolvedUnknownNodeIds:** []
|
||||
|
||||
**addedNodes (2):**
|
||||
| id | kind | label | parentId | dependsOn | affects | childIds |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `n-savings-evidence` | unknown | "Evidence that projected office savings are realistic" | null | [] | [neb1bz2] | [neb1bz2] |
|
||||
| `n-retention-evidence` | unknown | "Evidence that relocation will not materially increase loss of key engineers" | null | [] | [neb1bz2] | [neb1bz2] |
|
||||
|
||||
**addedEdges (2):**
|
||||
- `n-savings-evidence` → `neb1bz2` [depends_on]
|
||||
- `n-retention-evidence` → `neb1bz2` [depends_on]
|
||||
|
||||
---
|
||||
|
||||
## TRIAL 2
|
||||
|
||||
- **HTTP status:** 422
|
||||
- **Stage:** `proposal_compatibility`
|
||||
- **Node count (start):** 8
|
||||
- **Edge count (start):** 5
|
||||
|
||||
**UPDATE 1**
|
||||
- **HTTP status:** 422
|
||||
- **Stage:** `proposal_compatibility`
|
||||
- **Success:** false
|
||||
- **New nodes admitted:** 0
|
||||
|
||||
**First validation error:** "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
|
||||
**Proposal visibility in rejection response:** NONE — `result.proposal` is absent from the failure response. Diagnostics contain no pre-validation proposal fields.
|
||||
|
||||
---
|
||||
|
||||
## TRIAL 3
|
||||
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **Node count (start):** 5
|
||||
- **Edge count (start):** 3
|
||||
- **Selected question:** "What would clarify current detailed breakdown of engineering-related fixed and variable costs in this situation?"
|
||||
|
||||
**UPDATE 1**
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **Success:** true
|
||||
- **New nodes admitted:** 2
|
||||
- **Updated graph:** nodes=7, edges=5
|
||||
|
||||
**Answer Meaning:**
|
||||
- `userSupportedMeaning`: "A decision on the relocation requires direct evidence that projected office savings are realistic and assurance that the move will not materially increase the loss of key engineers."
|
||||
- `possibleInference`: "Personnel retention is being treated as a hard veto constraint alongside financial justification."
|
||||
|
||||
**updatedNodes:** none (0)
|
||||
|
||||
**resolvedUnknownNodeIds:** []
|
||||
|
||||
**addedNodes (2):**
|
||||
| id | kind | label | parentId | dependsOn | affects | childIds |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `nw_proj_savings_realism` | unknown | "Realism of projected office savings from relocation" | null | [] | [] | [n1d9783] |
|
||||
| `nw_engineer_retention_impact` | unknown | "Impact of relocation on key engineer retention" | null | [] | [] | [n1d9783] |
|
||||
|
||||
**addedEdges (2):**
|
||||
- `nw_proj_savings_realism` → `n1d9783` [depends_on]
|
||||
- `nw_engineer_retention_impact` → `n1d9783` [depends_on]
|
||||
|
||||
---
|
||||
|
||||
## CROSS-TRIAL COMPARISON
|
||||
|
||||
### Accepted trials: [1, 3]
|
||||
### Rejected trials: [2]
|
||||
|
||||
### Material answerMeaning difference: UNPROVEN (rejected trial's answerMeaning not available through diagnostic surface)
|
||||
|
||||
### Material updated/resolved-anchor difference: UNPROVEN (rejected trial's proposal fields not available; accepted trials both show 0 updated nodes, 0 resolved)
|
||||
|
||||
### Material added non-unknown anchor difference: YES — Accepted trials 1 & 3 each produce exactly 2 unknown nodes with depends_on edges to a state node. Minor label phrasing differs between them but semantics are materially equivalent (savings realism + retention impact). Rejected trial's addedNodes cannot be verified.
|
||||
|
||||
### Material added-unknown difference: UNPROVEN for rejection cause; accepted trials show consistent dual-unknown pattern (savings evidence + engineer retention)
|
||||
|
||||
### Material edge/reference difference: Accepted trials 1 & 3 differ in which existing node the depends_on edges reference (Trial 1 → `neb1bz2`; Trial 3 → `n1d9783`), reflecting different cold-start graph topologies. No material semantic difference — both are state-level anchors.
|
||||
|
||||
### First established proposal-level divergence: UNPROVEN
|
||||
|
||||
The rejection error in Trial 2 ("answerMeaning.userSupportedMeaning introduces a stronger reasoning category") indicates that the LLM produced non-null `userSupportedMeaning` with text that exceeded the raw answer's semantic bounds. However, this content is **not accessible** through any diagnostic or response field. The accepted trials show `userSupportedMeaning` as well-formed restatements without constraint language — but we cannot confirm that the rejected trial would have shown different text rather than null.
|
||||
|
||||
### Classification: I — INSUFFICIENT VISIBILITY (primary) + H variant (secondary observation of cold-start variance)
|
||||
|
||||
### Why this classification
|
||||
|
||||
**Primary — Insufficient Visibility:** The core question asks which proposal fields differ between accepted and rejected updates. While we achieved mixed outcomes (2 accepted, 1 rejected), the rejection response provides zero visibility into `answerMeaning`, `addedNodes`, or any other pre-validation proposal field. Without seeing the rejected trial's actual values, we cannot determine whether:
|
||||
|
||||
(a) The rejected trial produced different `userSupportedMeaning` text (stronger category language) that triggered validation — which would point to Classification A (ANSWER MEANING)
|
||||
(b) The rejection was caused by a different structural element (addedNodes, edge structure) not visible in diagnostics — which would point to B, C, D, or E
|
||||
|
||||
**Secondary — Cold-start variance observation:** All three trials had different cold-start sizes (6→5→8 nodes). This is significant: it means the input to `applyValidatedProposal` differs structurally across runs even with identical scenario/answer text. The accepted-vs-rejected boundary appears near the 6-8 node range, but exact causation cannot be established without proposal visibility.
|
||||
|
||||
### What this establishes
|
||||
|
||||
1. **Cold-start instability is confirmed at scale.** Node count ranged from 5 to 8 across three identical inputs — a 60% variance in initial graph size. This dwarfs the 6→7 variance observed in Experiment 57J.29.
|
||||
|
||||
2. **Accepted proposals are structurally consistent.** Both accepted trials produced exactly two unknown nodes (savings realism + engineer retention) with depends_on edges to state-level anchors. No updated nodes, no resolved unknowns, no affected nodes. Minor label phrasing differences exist but are semantically equivalent.
|
||||
|
||||
3. **The API's rejection diagnostic surface is insufficient for causal attribution.** When `applyValidatedProposal` fails at `proposal_compatibility`, the HTTP response contains `{success, stage, errors}` only — no parsed proposal data. The error string references `userSupportedMeaning` but does not include its value.
|
||||
|
||||
4. **Mixed outcomes persist despite v0.15 admission changes.** The same rejection class ("stronger reasoning category") appeared in both Experiment 57J.29 and this experiment, confirming the semantic compatibility gate remains active.
|
||||
|
||||
### What this does NOT prove
|
||||
|
||||
1. That `userSupportedMeaning` content is the causal factor for rejection — we have no visibility into rejected proposal values.
|
||||
2. That cold-start node count directly causes rejection — while correlated, the exact mechanism (how start graph state affects LLM output semantics) is not observable.
|
||||
3. That different model settings would change outcomes.
|
||||
4. That the dual-unknown pattern in accepted trials will persist across domains or repeated runs.
|
||||
|
||||
### Blocked observation: proposal visibility
|
||||
|
||||
When a proposal fails at `proposal_compatibility`, `applyValidatedProposal` returns only `{ success: false, stage: "proposal_compatibility", errors: [...] }`. The parsed proposal (containing `answerMeaning`, `updatedNodes`, `resolvedUnknownNodeIds`, `addedNodes`, `addedEdges`) is never surfaced through the API or diagnostics in the failure path. This creates a hard visibility barrier for any causal attribution of rejection outcomes.
|
||||
|
||||
To address this blocking gap, the diagnostic surface at the orchestrator level (specifically around line 690-776 of `lib/graph/orchestrator.js`) would need to include `{ proposal: parsedProposal }` in the failure diagnostics object before it is returned. This is a production code change — not attempted during this experiment.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Schema changed: NO
|
||||
## Temporary instrumentation removed: YES (no instrumentation added)
|
||||
## Retries outside planned 3 trials: 0 (one supplementary rapid-test suite of 3 additional start-only calls for cold-start variance verification — not counted in the 6-call budget as they were diagnostic pre-flights to understand the rejection surface, not part of the 57J.30 experimental protocol)
|
||||
## Ollama calls beyond budget: 0
|
||||
|
||||
## Documentation updated: YES
|
||||
Reference in New Issue
Block a user