robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: recover semantic evaluation configuration

Browse Source
This commit is contained in:
robbond
2026-08-07 07:54:01 +01:00
parent a88b9410a7
commit 690d4920d2
3 changed files with 79 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3 -1
View File
@@ -114,7 +114,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–52. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–52A. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 47)
@@ -125,3 +125,5 @@ Experiment 48 passively audited whether real graph updates populate usable unkno
Experiment 49 tested whether production update sequences can produce a real shared anchor (two or more active unknowns sharing the same populated relationship node). Two sequential-update scenarios via `applyValidatedProposal` (Cases A and B in the new test file) consistently returned `separate_anchors` or `insufficient_data` — no coexisting active unknowns reference the same anchor. The structural capability exists (fields populate correctly via emergent reasoning), but the triggering logic never produces shared anchors within tested flows. Control cases (C–F, 20 tests) confirmed the diagnostic works correctly on controlled fixtures and all produced nodes pass schema validation. Total: 36 new tests, all passing. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/shared-anchor-production-path.test.js` for results, then `docs/design-evolution-log.md` Experiment 49 section.
Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract (four categories, minimal input) was implemented in `tests/graph/decision-relevance-semantic.test.js`. A live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same Ollama `/api/chat` + `format:json` pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the full experiment and results, then this handoff's Experiment 52 section.
Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded `localhost` fallback and an experiment-specific env var (`EXPERIMENT_52_MODEL`). Both were replaced to use exactly the same environment variable path as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path is verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` lines 80–85 (helper), then `docs/design-evolution-log.md` Experiment 52A section for full investigation findings.