robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: measure live semantic representation stability

Browse Source
This commit is contained in:
robbond
2026-08-11 07:36:21 +01:00
parent 25f56d75e2
commit 1c15b2b123
2 changed files with 177 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -381,3 +381,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
### Experiment 57J.28 — Live Node-Support Semantic Inputs Capture
**Classification: C — GATE BEHAVES AS EXPECTED.** On a fresh live run, both user-supported unknowns (savings realism + retention impact) were admitted with HTTP 200 at `update_applied`, zero validation errors. The answerMeaning correctly captured both dimensions. The semantic gate (`rawAnswerSupportsUnclassifiedMeaning`) passed for both via the token-count clause (5 shared content tokens >= 3), not overlap ratio. Structural fallback also valid for both nodes. Static helper evaluation agrees with actual gate behavior — live admission outcome fully explained by captured inputs. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Start: 7 nodes, Update 1: 9 nodes (+2). What remains unproven: token-count adequacy for paraphrased/low-overlap answers, downstream investigation viability past Update 2+, run-to-run start graph stability. No production code changed.
### Experiment 57J.29 — Live Semantic Representation Stability (Repeated Identical Runs)
**Classification: D — DOWNSTREAM INSTABILITY SUSPECTED.** Three repeated identical live runs with the fixed scenario ("We are considering relocating the engineering team to reduce operating costs.") and fixed answer ("Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers.") through the production `startCase()` → `updateCase()` path. **key finding:** `userSupportedMeaning` was null/empty in ALL 3 trials — no semantic meaning was extracted by the model. Yet admission outcomes diverged: Trial 2 (start with 7 nodes) admitted both unknowns; Trials 1 & 3 (start with 6 nodes) rejected at `proposal_compatibility` with identical "stronger reasoning category" errors despite null diagnostics. This confirms that start graph quality (6 vs 7 nodes cold-start variance) directly affects admission outcomes, and when `userSupportedMeaning` is empty the gate may still process hidden semantic fields. The admission variance cannot be explained by upstream model representation because no meaningful semantic content was produced in any trial — instability is downstream of model representation. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 6 live calls total. No production code changed.