174 lines
9.0 KiB
Markdown
174 lines
9.0 KiB
Markdown
# Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs)
|
|
|
|
**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED**
|
|
|
|
## Objective
|
|
|
|
Determine whether repeated identical live runs produce materially different `userSupportedMeaning`, proposed unknown wording, or both—and whether those differences correlate with admission success/failure across the v0.15 unknown admission path.
|
|
|
|
## Pre-written expectation recorded: YES
|
|
|
|
> If the remaining live instability is upstream model variance, repeated identical inputs should sometimes produce materially different `userSupportedMeaning`, proposed unknown wording, or both, and those differences should correlate with admission success/failure. If semantic inputs are materially equivalent across trials but admission outcomes differ, the instability is more likely downstream of model representation.
|
|
|
|
## Configured apparatus
|
|
|
|
- **Ollama base URL:** `http://192.168.1.111:11434`
|
|
- **Model:** `qwen-claude:latest`
|
|
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
|
|
- **HEAD at experiment start:** `25f56d7` — experiment: capture live node-support semantic inputs
|
|
|
|
## Fixed scenario and answer
|
|
|
|
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
|
|
**Answer (Update 1):** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
|
|
|
## Live-call count
|
|
|
|
```
|
|
start: 3 (one per trial)
|
|
update 1: 3 (one per trial)
|
|
total: 6
|
|
```
|
|
|
|
## TRIAL 1
|
|
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Node count (start):** 6
|
|
- **Edge count (start):** 3
|
|
- **Selected question:** "What was the comparable state before current cost baseline for the engineering team versus projected relocation and operating expenses in the target location?"
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Success:** false
|
|
- **Error:** `"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."`
|
|
- **New nodes admitted:** 0
|
|
|
|
**Answer Meaning:**
|
|
- `userSupportedMeaning`: null
|
|
- `possibleInference`: null
|
|
|
|
**Savings target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN
|
|
|
|
**Retention target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN
|
|
|
|
## TRIAL 2
|
|
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Node count (start):** 7
|
|
- **Edge count (start):** 4
|
|
- **Selected question:** "What was the comparable state before current detailed baseline of operating costs attributable to the engineering team?"
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 200
|
|
- **Stage:** `update_applied`
|
|
- **Success:** true
|
|
- **New nodes admitted:** 2
|
|
- **Updated node count:** 9 (+2)
|
|
- **Updated edge count:** 6 (+2)
|
|
|
|
**Answer Meaning:**
|
|
- `userSupportedMeaning`: null
|
|
- `possibleInference`: null
|
|
|
|
**Proposed new unknowns:**
|
|
| id | label | description |
|
|
|---|---|---|
|
|
| `n-sav-real` | "Realism and validation of projected office savings" | "The degree to which projected office relocation savings are realistic and substantiated..." |
|
|
| `n-ret-risk` | "Impact of relocation on key engineer retention" | "The extent to which relocating the engineering team will materially increase turnover among critical staff..." |
|
|
|
|
**Savings target:** coverage=PARTIAL, representation=CLEARLY GROUNDED, helper_result=FAIL (semantic gate), admission=PASS (structural fallback)
|
|
|
|
**Retention target:** coverage=PARTIAL, representation=CLEARLY GROUNDED, helper_result=FAIL (semantic gate), admission=PASS (structural fallback)
|
|
|
|
## TRIAL 3
|
|
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Node count (start):** 6
|
|
- **Edge count (start):** 4
|
|
- **Selected question:** "What evidence would confirm or rule out current monthly operating costs, projected new location costs, and one-time relocation expenses?"
|
|
|
|
**UPDATE 1**
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility`
|
|
- **Success:** false
|
|
- **Error:** `"answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."`
|
|
- **New nodes admitted:** 0
|
|
|
|
**Answer Meaning:**
|
|
- `userSupportedMeaning`: null
|
|
- `possibleInference`: null
|
|
|
|
**Savings target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN
|
|
|
|
**Retention target:** coverage=PARTIAL (USM empty), representation=UNAVAILABLE, admission=UNPROVEN
|
|
|
|
## CROSS-TRIAL ANALYSIS
|
|
|
|
### Material answerMeaning variance: NO
|
|
|
|
`userSupportedMeaning` is null/empty in all three trials. No material semantic variation exists between trials at the answer-meaning layer. The diagnostic shows no meaning was extracted by the model in any trial — meaning this experiment cannot confirm whether the *potential* for different `userSupportedMeaning` content exists, only that none was produced.
|
|
|
|
### Material node-wording variance: YES (but conditional)
|
|
|
|
Trial 2 proposed two nodes with specific labels/descriptions grounded in savings and retention semantics. Trials 1 and 3 had zero new unknowns (rejection at proposal_compatibility occurred before nodes were materialized). This is a structural variance, not a semantic wording difference per se — it stems from the different admission outcomes.
|
|
|
|
### Admission outcome variance: YES
|
|
|
|
Trial 2: SUCCESS (admitted both unknowns)
|
|
Trial 1: FAILED (proposal_compatibility rejection)
|
|
Trial 3: FAILED (proposal_compatibility rejection)
|
|
|
|
### Start graph stability: NO
|
|
|
|
- Trials 1, 3: 6 nodes, varying edge counts (3, 4)
|
|
- Trial 2: 7 nodes, 4 edges
|
|
|
|
This confirms the cold-start graph instability observed in previous experiments (57J.26 noted 6→7→9 node variance).
|
|
|
|
### First material source of variance: NEITHER
|
|
|
|
No answerMeaning variance exists (USM null across all trials). The node-wording difference is a *consequence* of admission outcomes, not an independent upstream cause. Therefore neither A nor B qualifies as the *first* material source.
|
|
|
|
## Classification: D — DOWNSTREAM INSTABILITY SUSPECTED
|
|
|
|
### Why this classification
|
|
|
|
The three key observations are:
|
|
|
|
1. **userSupportedMeaning was null/empty in ALL 3 trials** — the model did not extract any semantic meaning from the answer in any run. This means there is zero upstream variance to explain.
|
|
2. **Trials 1 and 3 failed identically** with the same rejection error about "stronger reasoning category" despite having null `userSupportedMeaning` (which should mean no strengthening at all). This error text suggests the model *did* produce some semantic content, but it wasn't captured in my diagnostic display.
|
|
3. **Trial 2 succeeded and admitted nodes** despite also showing null `possibleInference` — meaning the semantic gate accepted them via structural fallback (both nodes have depends_on edges to the central state node).
|
|
|
|
The admission outcome variance cannot be explained by upstream model representation variance because no meaningful semantic content was produced in any trial. The identical rejection errors in Trials 1 and 3 despite null diagnostics suggest the gate logic is processing hidden/uncaptured semantic fields differently depending on the start graph state.
|
|
|
|
### What this establishes
|
|
|
|
1. **Start graph quality affects admission outcomes directly.** A cold-start with 6 nodes → reject; cold-start with 7 nodes → admit (under the same fixed scenario and answer).
|
|
2. **When userSupportedMeaning is empty/null, the model may still produce semantic content that is not captured by standard diagnostic fields** — suggesting there may be intermediate representations or fields beyond `userSupportedMeaning`/`possibleInference` that influence downstream gates.
|
|
3. **The token-count structural fallback (depends_on edges) can admit nodes even when the semantic gate would FAIL**, confirming that structural fallback is a critical admission path.
|
|
4. **Start graph variance (6 vs 7 nodes) is a real and measurable source of instability** independent of answer processing.
|
|
|
|
### What this does NOT prove
|
|
|
|
1. That `userSupportedMeaning` CAN vary materially — it was null in all trials, so this experiment did not test that possibility.
|
|
2. That the start graph quality difference (6 vs 7 nodes) is deterministic — only one instance of each count was observed.
|
|
3. That downstream instability is a bug rather than an emergent property of LLM pipeline composition.
|
|
4. That the same results would hold with different model settings or provider.
|
|
5. Whether the "stronger reasoning category" error in Trials 1/3 actually comes from `userSupportedMeaning` content that was present but not displayed, or from another field entirely.
|
|
|
|
### Note on diagnostic completeness
|
|
|
|
The key limitation: when `userSupportedMeaning` displays as null/empty, it is possible the API returned an empty string `""` in Trials 1/3 and a JSON null `null` in Trial 2 (or vice versa), which my display logic treats equivalently but which the gate logic may treat differently. A follow-up experiment should inspect the raw HTTP response body for these fields to confirm.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Schema changed: NO
|
|
## Temporary instrumentation removed: YES
|
|
## Retries outside planned 3 trials: 0
|
|
## Ollama calls beyond budget: 0
|