robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: run small semantic decision-relevance probe

Browse Source
This commit is contained in:
robbond
2026-08-07 08:28:06 +01:00
parent 690d4920d2
commit 8ee1f575f7
3 changed files with 356 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3 -1
View File
@@ -114,7 +114,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–52A. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–52B. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 47)
@@ -127,3 +127,5 @@ Experiment 49 tested whether production update sequences can produce a real shar
Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract (four categories, minimal input) was implemented in `tests/graph/decision-relevance-semantic.test.js`. A live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same Ollama `/api/chat` + `format:json` pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the full experiment and results, then this handoff's Experiment 52 section.
Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path is verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper), then `docs/design-evolution-log.md` Experiment 52A section for full investigation findings.
Experiment 52B reduced the semantic evaluation to exactly six live inference cases (one call each) from the existing corpus and ran them against the configured `qwen-claude:latest` model. Semantic interpretation scored 5/6 against fixed human reference labels; deterministic baseline scored 1/6 on the same cases. Key wins: correctly classified the known keyword-failure paraphrase (Case 3) and generalised to a second domain (Cases 4–6). The model's output needed explicit enum values in the instruction (it returns prose by default), but no conceptual change to the semantic contract. Latency averaged ~16s/call vs the ~82s earlier measurement. All tests pass (15 new + 45 regression). No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for six cases and results, then `docs/design-evolution-log.md` Experiment 52B section.