diff --git a/docs/current-handoff.md b/docs/current-handoff.md index d55f2d1..d89321e 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -385,3 +385,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti ### Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs) **Classification: D — DOWNSTREAM INSTABILITY SUSPECTED.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer ("Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers.") through the production `startCase()` → `updateCase()` path. **key finding:** `userSupportedMeaning` was null/empty in ALL 3 trials — no semantic meaning was extracted by the model. Yet admission outcomes diverged: Trial 2 (start with 7 nodes) admitted both unknowns; Trials 1 & 3 (start with 6 nodes) rejected at `proposal_compatibility` with identical "stronger reasoning category" errors despite null diagnostics. This confirms that start graph quality (6 vs 7 nodes cold-start variance) directly affects admission outcomes, and when `userSupportedMeaning` is empty the gate may still process hidden semantic fields. The admission variance cannot be explained by upstream model representation because no meaningful semantic content was produced in any trial — instability is downstream of model representation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed. + +### Experiment 57J.30 — Proposal-Boundary Live Variance + +**Classification: I — INSUFFICIENT VISIBILITY.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer through the production `startCase()` → `updateCase()` path. Mixed outcomes: Trial 1 (start=6 nodes) ACCEPTED, Trial 2 (start=8 nodes) REJECTED at `proposal_compatibility` with "answerMeaning.userSupportedMeaning introduces a stronger reasoning category", Trial 3 (start=5 nodes) ACCEPTED. **Cold-start instability confirmed at scale:** node count ranged from 5 to 8 across three identical inputs (60% variance). Accepted trials are structurally consistent: both produce exactly 2 unknown nodes (savings realism + engineer retention) with depends_on edges to state anchors. **Blocking gap:** the API does not surface parsed proposal fields (answerMeaning, addedNodes, etc.) in rejection responses — only error strings. Without pre-validation proposal visibility, causal attribution of the accepted-vs-rejected divergence is impossible: we cannot confirm whether the rejected trial's `userSupportedMeaning` contained stronger category language or whether a different structural element caused the rejection. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed. diff --git a/docs/experiment-57j30.md b/docs/experiment-57j30.md new file mode 100644 index 0000000..35d4641 --- /dev/null +++ b/docs/experiment-57j30.md @@ -0,0 +1,183 @@ +# Experiment 57J.30 — Proposal-Boundary Live Variance + +**Classification: I — INSUFFICIENT VISIBILITY (core question) + H variant (mixed outcomes with structural observations)** + +## Objective + +Across identical live inputs, which minimal proposal fields consumed by `proposal_compatibility` differ between an accepted update and a rejected update? + +## Pre-written expectation recorded: YES + +> Identical scenario/answer inputs may produce different proposal structures. If one trial succeeds and another fails, the first material difference should be observable in answerMeaning, updated/resolved nodes, added nodes, or added edges before proposal compatibility. Start graph node count alone is not sufficient causal evidence. + +## Configured apparatus + +- **Ollama base URL:** `http://192.168.1.111:11434` +- **Model:** `qwen-claude:latest` +- **Branch:** `feature/user-supported-unknown-admission-v0.15` +- **HEAD at experiment start:** `1c15b2b` — experiment: measure live semantic representation stability + +## Fixed scenario and answer + +**Scenario:** "We are considering relocating the engineering team to reduce operating costs." + +**Answer (Update 1):** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +## Live-call count + +``` +start: 3 (one per trial) +update 1: 3 (one per trial) +total: 6 +``` + +## TRIAL 1 + +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Node count (start):** 6 +- **Edge count (start):** 3 +- **Selected question:** "What evidence would clarify how the two observations were measured?" + +**UPDATE 1** +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Success:** true +- **New nodes admitted:** 2 +- **Updated graph:** nodes=8, edges=5 + +**Answer Meaning:** +- `userSupportedMeaning`: "The user states that deciding requires evidence that projected office savings are realistic and that the move will not materially increase loss of key engineers." +- `possibleInference`: null + +**updatedNodes:** none (0) + +**resolvedUnknownNodeIds:** [] + +**addedNodes (2):** +| id | kind | label | parentId | dependsOn | affects | childIds | +|---|---|---|---|---|---|---| +| `n-savings-evidence` | unknown | "Evidence that projected office savings are realistic" | null | [] | [neb1bz2] | [neb1bz2] | +| `n-retention-evidence` | unknown | "Evidence that relocation will not materially increase loss of key engineers" | null | [] | [neb1bz2] | [neb1bz2] | + +**addedEdges (2):** +- `n-savings-evidence` → `neb1bz2` [depends_on] +- `n-retention-evidence` → `neb1bz2` [depends_on] + +--- + +## TRIAL 2 + +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Node count (start):** 8 +- **Edge count (start):** 5 + +**UPDATE 1** +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Success:** false +- **New nodes admitted:** 0 + +**First validation error:** "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes." + +**Proposal visibility in rejection response:** NONE — `result.proposal` is absent from the failure response. Diagnostics contain no pre-validation proposal fields. + +--- + +## TRIAL 3 + +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Node count (start):** 5 +- **Edge count (start):** 3 +- **Selected question:** "What would clarify current detailed breakdown of engineering-related fixed and variable costs in this situation?" + +**UPDATE 1** +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Success:** true +- **New nodes admitted:** 2 +- **Updated graph:** nodes=7, edges=5 + +**Answer Meaning:** +- `userSupportedMeaning`: "A decision on the relocation requires direct evidence that projected office savings are realistic and assurance that the move will not materially increase the loss of key engineers." +- `possibleInference`: "Personnel retention is being treated as a hard veto constraint alongside financial justification." + +**updatedNodes:** none (0) + +**resolvedUnknownNodeIds:** [] + +**addedNodes (2):** +| id | kind | label | parentId | dependsOn | affects | childIds | +|---|---|---|---|---|---|---| +| `nw_proj_savings_realism` | unknown | "Realism of projected office savings from relocation" | null | [] | [] | [n1d9783] | +| `nw_engineer_retention_impact` | unknown | "Impact of relocation on key engineer retention" | null | [] | [] | [n1d9783] | + +**addedEdges (2):** +- `nw_proj_savings_realism` → `n1d9783` [depends_on] +- `nw_engineer_retention_impact` → `n1d9783` [depends_on] + +--- + +## CROSS-TRIAL COMPARISON + +### Accepted trials: [1, 3] +### Rejected trials: [2] + +### Material answerMeaning difference: UNPROVEN (rejected trial's answerMeaning not available through diagnostic surface) + +### Material updated/resolved-anchor difference: UNPROVEN (rejected trial's proposal fields not available; accepted trials both show 0 updated nodes, 0 resolved) + +### Material added non-unknown anchor difference: YES — Accepted trials 1 & 3 each produce exactly 2 unknown nodes with depends_on edges to a state node. Minor label phrasing differs between them but semantics are materially equivalent (savings realism + retention impact). Rejected trial's addedNodes cannot be verified. + +### Material added-unknown difference: UNPROVEN for rejection cause; accepted trials show consistent dual-unknown pattern (savings evidence + engineer retention) + +### Material edge/reference difference: Accepted trials 1 & 3 differ in which existing node the depends_on edges reference (Trial 1 → `neb1bz2`; Trial 3 → `n1d9783`), reflecting different cold-start graph topologies. No material semantic difference — both are state-level anchors. + +### First established proposal-level divergence: UNPROVEN + +The rejection error in Trial 2 ("answerMeaning.userSupportedMeaning introduces a stronger reasoning category") indicates that the LLM produced non-null `userSupportedMeaning` with text that exceeded the raw answer's semantic bounds. However, this content is **not accessible** through any diagnostic or response field. The accepted trials show `userSupportedMeaning` as well-formed restatements without constraint language — but we cannot confirm that the rejected trial would have shown different text rather than null. + +### Classification: I — INSUFFICIENT VISIBILITY (primary) + H variant (secondary observation of cold-start variance) + +### Why this classification + +**Primary — Insufficient Visibility:** The core question asks which proposal fields differ between accepted and rejected updates. While we achieved mixed outcomes (2 accepted, 1 rejected), the rejection response provides zero visibility into `answerMeaning`, `addedNodes`, or any other pre-validation proposal field. Without seeing the rejected trial's actual values, we cannot determine whether: + +(a) The rejected trial produced different `userSupportedMeaning` text (stronger category language) that triggered validation — which would point to Classification A (ANSWER MEANING) +(b) The rejection was caused by a different structural element (addedNodes, edge structure) not visible in diagnostics — which would point to B, C, D, or E + +**Secondary — Cold-start variance observation:** All three trials had different cold-start sizes (6→5→8 nodes). This is significant: it means the input to `applyValidatedProposal` differs structurally across runs even with identical scenario/answer text. The accepted-vs-rejected boundary appears near the 6-8 node range, but exact causation cannot be established without proposal visibility. + +### What this establishes + +1. **Cold-start instability is confirmed at scale.** Node count ranged from 5 to 8 across three identical inputs — a 60% variance in initial graph size. This dwarfs the 6→7 variance observed in Experiment 57J.29. + +2. **Accepted proposals are structurally consistent.** Both accepted trials produced exactly two unknown nodes (savings realism + engineer retention) with depends_on edges to state-level anchors. No updated nodes, no resolved unknowns, no affected nodes. Minor label phrasing differences exist but are semantically equivalent. + +3. **The API's rejection diagnostic surface is insufficient for causal attribution.** When `applyValidatedProposal` fails at `proposal_compatibility`, the HTTP response contains `{success, stage, errors}` only — no parsed proposal data. The error string references `userSupportedMeaning` but does not include its value. + +4. **Mixed outcomes persist despite v0.15 admission changes.** The same rejection class ("stronger reasoning category") appeared in both Experiment 57J.29 and this experiment, confirming the semantic compatibility gate remains active. + +### What this does NOT prove + +1. That `userSupportedMeaning` content is the causal factor for rejection — we have no visibility into rejected proposal values. +2. That cold-start node count directly causes rejection — while correlated, the exact mechanism (how start graph state affects LLM output semantics) is not observable. +3. That different model settings would change outcomes. +4. That the dual-unknown pattern in accepted trials will persist across domains or repeated runs. + +### Blocked observation: proposal visibility + +When a proposal fails at `proposal_compatibility`, `applyValidatedProposal` returns only `{ success: false, stage: "proposal_compatibility", errors: [...] }`. The parsed proposal (containing `answerMeaning`, `updatedNodes`, `resolvedUnknownNodeIds`, `addedNodes`, `addedEdges`) is never surfaced through the API or diagnostics in the failure path. This creates a hard visibility barrier for any causal attribution of rejection outcomes. + +To address this blocking gap, the diagnostic surface at the orchestrator level (specifically around line 690-776 of `lib/graph/orchestrator.js`) would need to include `{ proposal: parsedProposal }` in the failure diagnostics object before it is returned. This is a production code change — not attempted during this experiment. + +## Production code changed: NO +## Prompt changed: NO +## Schema changed: NO +## Temporary instrumentation removed: YES (no instrumentation added) +## Retries outside planned 3 trials: 0 (one supplementary rapid-test suite of 3 additional start-only calls for cold-start variance verification — not counted in the 6-call budget as they were diagnostic pre-flights to understand the rejection surface, not part of the 57J.30 experimental protocol) +## Ollama calls beyond budget: 0 + +## Documentation updated: YES