robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: separate semantic meaning from relevance labels

Browse Source
This commit is contained in:
robbond
2026-08-07 09:00:46 +01:00
parent 8ee1f575f7
commit b42a1ff244
3 changed files with 628 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+2 -2
View File
@@ -114,7 +114,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–52B. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–52C. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 47)
@@ -128,4 +128,4 @@ Experiment 52 tested whether a small semantic interpretation step can judge deci
Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path is verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper), then `docs/design-evolution-log.md` Experiment 52A section for full investigation findings.
Experiment 52B reduced the semantic evaluation to exactly six live inference cases (one call each) from the existing corpus and ran them against the configured `qwen-claude:latest` model. Semantic interpretation scored 5/6 against fixed human reference labels; deterministic baseline scored 1/6 on the same cases. Key wins: correctly classified the known keyword-failure paraphrase (Case 3) and generalised to a second domain (Cases 4–6). The model's output needed explicit enum values in the instruction (it returns prose by default), but no conceptual change to the semantic contract. Latency averaged ~16s/call vs the ~82s earlier measurement. All tests pass (15 new + 45 regression). No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for six cases and results, then `docs/design-evolution-log.md` Experiment 52B section.
Experiment 52C separated free-language semantic understanding from enum normalisation into two independent calls per case across five decision/question pairs. Meaning mode captured all five intended relationships correctly (5/5). Enum classification matched expected categories on four of five cases (4/5). One meaning-correct / enum-mismatch case occurred: Case 2 (European regulatory compliance) was correctly described as supporting in both modes but classified as `could_change_decision` rather than `supports_decision`. Same Qwen model (`qwen-claude:latest`) and host were retained; no production behaviour changed. What remains uncertain: whether the meaning-enum gap generalises across decision domains, stability over repeated runs, and whether normalisation mechanisms can bridge the gap without altering interpretation. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic-normalisation.test.js` for results, then `docs/design-evolution-log.md` Experiment 52C section.