robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: compare model and deterministic investigation selection

Browse Source
This commit is contained in:
robbond
2026-08-18 07:00:39 +01:00
parent 600b07d820
commit 5d0ce0ddd3
2 changed files with 215 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+13
View File
@@ -104,3 +104,16 @@ The live model again selected a non-financial-comparison target on the product-l
**Update 1 evidence: DISCARDED.** The canonical harness auto-continued with its preconfigured `answers[0]`, so the Update occurred outside the experiment's semantic gate. This was an apparatus defect (60B.98) — the harness did not provide a post-Start stop gate at that time. The HTTP 500 is NOT established as a reasoning defect from 60B.97.
**Apparatus correction:** See section "Canonical harness gated apparatus (60B.99)" above for the fix.
## Experiment 60B.100 result (2026-08-18)
**Classification: B — DETERMINISTIC SELECTOR OVERRIDES MODEL QUESTION**
On a fresh product-launch Start, the LLM reconstruction question targeted one uncertainty ("What is the estimated probability that the large enterprise customer will sign?") while the deterministic graph-backed selector chose another ("What evidence would clarify the exact percentage of total projected revenue attributable to the enterprise customer?"). These are materially different: one asks about deal timing/commitment probability, the other asks about financial proportion/magnitude.
The override was produced by fixed `actor_match` keyword scoring: node n65sgyd's label contained "enterprise customer" which matched the actor dictionary (+10 delta), giving it a decisive score of 10 vs 4 for both competitors. No tie/fallback was involved — the winner was determined entirely by keyword rule weighting.
**What this establishes:** On fresh Start calls, deterministic keyword signals can override model-inferred investigation priority when node labels differ in dictionary-match patterns. The final investigation target is not the model's contextual judgment but the highest-scoring candidate under fixed scoring rules.
**What this does NOT prove:** Whether the deterministic selection is better or worse than the model's suggestion; consistency across scenario types; or downstream investigation quality impact.