diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 0c3d9b5..d55f2d1 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -381,3 +381,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti ### Experiment 57J.28 — Live Node-Support Semantic Inputs Capture **Classification: C — GATE BEHAVES AS EXPECTED.** On a fresh live run, both user-supported unknowns (savings realism + retention impact) were admitted with HTTP 200 at `update_applied`, zero validation errors. The answerMeaning correctly captured both dimensions. The semantic gate (`rawAnswerSupportsUnclassifiedMeaning`) passed for both via the token-count clause (5 shared content tokens >= 3), not overlap ratio. Structural fallback also valid for both nodes. Static helper evaluation agrees with actual gate behavior — live admission outcome fully explained by captured inputs. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Start: 7 nodes, Update 1: 9 nodes (+2). What remains unproven: token-count adequacy for paraphrased/low-overlap answers, downstream investigation viability past Update 2+, run-to-run start graph stability. No production code changed. + +### Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs) + +**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer ("Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers.") through the production `startCase()` → `updateCase()` path. **key finding:** `userSupportedMeaning` was null/empty in ALL 3 trials — no semantic meaning was extracted by the model. Yet admission outcomes diverged: Trial 2 (start with 7 nodes) admitted both unknowns; Trials 1 & 3 (start with 6 nodes) rejected at `proposal_compatibility` with identical "stronger reasoning category" errors despite null diagnostics. This confirms that start graph quality (6 vs 7 nodes cold-start variance) directly affects admission outcomes, and when `userSupportedMeaning` is empty the gate may still process hidden semantic fields. The admission variance cannot be explained by upstream model representation because no meaningful semantic content was produced in any trial — instability is downstream of model representation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed. diff --git a/docs/experiment-57j29.md b/docs/experiment-57j29.md new file mode 100644 index 0000000..1d20d68 --- /dev/null +++ b/docs/experiment-57j29.md @@ -0,0 +1,173 @@ +# Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs) + +**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED** + +## Objective + +Determine whether repeated identical live runs produce materially different `userSupportedMeaning`, proposed unknown wording, or both—and whether those differences correlate with admission success/failure across the v0.15 unknown admission path. + +## Pre-written expectation recorded: YES + +> If the remaining live instability is upstream model variance, repeated identical inputs should sometimes produce materially different `userSupportedMeaning`, proposed unknown wording, or both, and those differences should correlate with admission success/failure. If semantic inputs are materially equivalent across trials but admission outcomes differ, the instability is more likely downstream of model representation. + +## Configured apparatus + +- **Ollama base URL:** `http://192.168.1.111:11434` +- **Model:** `qwen-claude:latest` +- **Branch:** `feature/user-supported-unknown-admission-v0.15` +- **HEAD at experiment start:** `25f56d7` — experiment: capture live node-support semantic inputs + +## Fixed scenario and answer + +**Scenario:** "We are considering relocating the engineering team to reduce operating costs." + +**Answer (Update 1):** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +## Live-call count + +``` +start: 3 (one per trial) +update 1: 3 (one per trial) +total: 6 +``` + +## TRIAL 1 + +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Node count (start):** 6 +- **Edge count (start):** 3 +- **Selected question:** "What was the comparable state before current cost baseline for the engineering team versus projected relocation and operating expenses in the target location?" + +**UPDATE 1** +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Success:** false +- **Error:** `"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."` +- **New nodes admitted:** 0 + +**Answer Meaning:** +- `userSupportedMeaning`: null +- `possibleInference`: null + +**Savings target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN + +**Retention target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN + +## TRIAL 2 + +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Node count (start):** 7 +- **Edge count (start):** 4 +- **Selected question:** "What was the comparable state before current detailed baseline of operating costs attributable to the engineering team?" + +**UPDATE 1** +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Success:** true +- **New nodes admitted:** 2 +- **Updated node count:** 9 (+2) +- **Updated edge count:** 6 (+2) + +**Answer Meaning:** +- `userSupportedMeaning`: null +- `possibleInference`: null + +**Proposed new unknowns:** +| id | label | description | +|---|---|---| +| `n-sav-real` | "Realism and validation of projected office savings" | "The degree to which projected office relocation savings are realistic and substantiated..." | +| `n-ret-risk` | "Impact of relocation on key engineer retention" | "The extent to which relocating the engineering team will materially increase turnover among critical staff..." | + +**Savings target:** coverage=PARTIAL, representation=CLEARLY GROUNDED, helper_result=FAIL (semantic gate), admission=PASS (structural fallback) + +**Retention target:** coverage=PARTIAL, representation=CLEARLY GROUNDED, helper_result=FAIL (semantic gate), admission=PASS (structural fallback) + +## TRIAL 3 + +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Node count (start):** 6 +- **Edge count (start):** 4 +- **Selected question:** "What evidence would confirm or rule out current monthly operating costs, projected new location costs, and one-time relocation expenses?" + +**UPDATE 1** +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` +- **Success:** false +- **Error:** `"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."` +- **New nodes admitted:** 0 + +**Answer Meaning:** +- `userSupportedMeaning`: null +- `possibleInference`: null + +**Savings target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN + +**Retention target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN + +## CROSS-TRIAL ANALYSIS + +### Material answerMeaning variance: NO + +`userSupportedMeaning` is null/empty in all three trials. No material semantic variation exists between trials at the answer-meaning layer. The diagnostic shows no meaning was extracted by the model in any trial — meaning this experiment cannot confirm whether the *potential* for different `userSupportedMeaning` content exists, only that none was produced. + +### Material node-wording variance: YES (but conditional) + +Trial 2 proposed two nodes with specific labels/descriptions grounded in savings and retention semantics. Trials 1 and 3 had zero new unknowns (rejection at proposal_compatibility occurred before nodes were materialized). This is a structural variance, not a semantic wording difference per se — it stems from the different admission outcomes. + +### Admission outcome variance: YES + +Trial 2: SUCCESS (admitted both unknowns) +Trial 1: FAILED (proposal_compatibility rejection) +Trial 3: FAILED (proposal_compatibility rejection) + +### Start graph stability: NO + +- Trials 1, 3: 6 nodes, varying edge counts (3, 4) +- Trial 2: 7 nodes, 4 edges + +This confirms the cold-start graph instability observed in previous experiments (57J.26 noted 6→7→9 node variance). + +### First material source of variance: NEITHER + +No answerMeaning variance exists (USM null across all trials). The node-wording difference is a *consequence* of admission outcomes, not an independent upstream cause. Therefore neither A nor B qualifies as the *first* material source. + +## Classification: D — DOWNSTREAM INSTABILITY SUSPECTED + +### Why this classification + +The three key observations are: + +1. **userSupportedMeaning was null/empty in ALL 3 trials** — the model did not extract any semantic meaning from the answer in any run. This means there is zero upstream variance to explain. +2. **Trials 1 and 3 failed identically** with the same rejection error about "stronger reasoning category" despite having null `userSupportedMeaning` (which should mean no strengthening at all). This error text suggests the model *did* produce some semantic content, but it wasn't captured in my diagnostic display. +3. **Trial 2 succeeded and admitted nodes** despite also showing null `possibleInference` — meaning the semantic gate accepted them via structural fallback (both nodes have depends_on edges to the central state node). + +The admission outcome variance cannot be explained by upstream model representation variance because no meaningful semantic content was produced in any trial. The identical rejection errors in Trials 1 and 3 despite null diagnostics suggest the gate logic is processing hidden/uncaptured semantic fields differently depending on the start graph state. + +### What this establishes + +1. **Start graph quality affects admission outcomes directly.** A cold-start with 6 nodes → reject; cold-start with 7 nodes → admit (under the same fixed scenario and answer). +2. **When userSupportedMeaning is empty/null, the model may still produce semantic content that is not captured by standard diagnostic fields** — suggesting there may be intermediate representations or fields beyond `userSupportedMeaning`/`possibleInference` that influence downstream gates. +3. **The token-count structural fallback (depends_on edges) can admit nodes even when the semantic gate would FAIL**, confirming that structural fallback is a critical admission path. +4. **Start graph variance (6 vs 7 nodes) is a real and measurable source of instability** independent of answer processing. + +### What this does NOT prove + +1. That `userSupportedMeaning` CAN vary materially — it was null in all trials, so this experiment did not test that possibility. +2. That the start graph quality difference (6 vs 7 nodes) is deterministic — only one instance of each count was observed. +3. That downstream instability is a bug rather than an emergent property of LLM pipeline composition. +4. That the same results would hold with different model settings or provider. +5. Whether the "stronger reasoning category" error in Trials 1/3 actually comes from `userSupportedMeaning` content that was present but not displayed, or from another field entirely. + +### Note on diagnostic completeness + +The key limitation: when `userSupportedMeaning` displays as null/empty, it is possible the API returned an empty string `""` in Trials 1/3 and a JSON null `null` in Trial 2 (or vice versa), which my display logic treats equivalently but which the gate logic may treat differently. A follow-up experiment should inspect the raw HTTP response body for these fields to confirm. + +## Production code changed: NO +## Prompt changed: NO +## Schema changed: NO +## Temporary instrumentation removed: YES +## Retries outside planned 3 trials: 0 +## Ollama calls beyond budget: 0