experiment: recover semantic evaluation configuration

2026-08-07 07:54:01 +01:00
parent a88b9410a7
commit 690d4920d2
3 changed files with 79 additions and 3 deletions
@@ -3663,4 +3663,73 @@ No assessor files loaded or modified. No Behaviour Selection files loaded or mod

 The decision-relevance classifier has no callers outside its own module. No active user-facing behaviour changed. The semantic helper was never wired into the engine under test.

+---
+
+## Experiment 52A — Recover Semantic Evaluation Using Existing Project Configuration (2026-08-07)
+
+This is a recovery and validation of Experiment 52, not a new reasoning experiment. Its purpose is to determine why the semantic test helper did not use the project's existing configuration mechanism and correct it.
+
+### Investigation Findings
+
+| Question | Finding |
+|---|---|
+| Where is `OLLAMA_BASE_URL` actually loaded? | Production reads directly from `process.env.OLLAMA_BASE_URL`. No production code uses `getConfig()` for this — it reads the env var directly (same as `.env.local`). |
+| Does `.env.local` already contain the correct host? | Yes: `http://192.168.1.111:11434`. Ollama confirmed running there with `qwen-claude:latest`. |
+| Why did the semantic helper use localhost? | The test helper had a hardcoded fallback: `process.env.OLLAMA_BASE_URL \|\| "http://localhost:11434"`. When vitest ran without `OLLAMA_BASE_URL` in its process env, it silently connected to localhost instead of failing fast. |
+| Was provider logic duplicated? | Partially. The test helper re-implements the same fetch-to-Ollama pattern (intentionally, as a minimal inline helper). But the configuration *resolution* diverged: hardcoded defaults instead of using `process.env`. |
+| Was configuration bypassed? | Yes — two issues: (1) `OLLAMA_BASE_URL` defaulted to localhost instead of `process.env.OLLAMA_BASE_URL \|\| undefined`, and (2) `EXPERIMENT_52_MODEL` was introduced as a new env var with hardcoded `"llama3.1"` default, bypassing the project's `OLLAMA_MODEL` config in `.env.local`. |
+| Is any production code incorrect? | No. Production `lib/llm/provider.js:100` reads from `process.env.OLLAMA_BASE_URL` correctly. `.env.local` has the correct values. Config module validates them via Zod. |
+| What is the smallest correction? | (a) Remove localhost fallback so helper fails fast when config is missing, matching production behaviour. (b) Replace `EXPERIMENT_52_MODEL` with existing `OLLAMA_MODEL`. (c) Add dotenv loading from `.env.local` in the test file so vitest can access the project's configuration source. |
+
+### Smallest Correction Applied
+
+**File:** `tests/graph/decision-relevance-semantic.test.js`
+
+Three changes, all in the test helper only:
+
+1. Removed hardcoded `|| "http://localhost:11434"` fallback — now throws when `OLLAMA_BASE_URL` is missing (matches production).
+2. Replaced `process.env.EXPERIMENT_52_MODEL \|\| "llama3.1"` with `process.env.OLLAMA_MODEL \|\| "llama3.1"` — uses project config, not an experiment-specific variable.
+3. Added `dotenv.config({ path: ".env.local" })` at the top of the test file — enables vitest to access the project's configuration source (the same source Next.js uses).
+
+### Configuration Source Resolved
+
+- Ollama base URL: `http://192.168.1.111:11434` (from `.env.local`)
+- Model: `qwen-claude:latest` (from `.env.local`, via `process.env.OLLAMA_MODEL`)
+
+### Experimental Result: Ollama Performance
+
+Ollama at 192.168.1.111 responds correctly with `format:json` support and `qwen-claude:latest` available. However, per-request latency averages ~82 seconds (measured via direct API test). The semantic test requires 33 cases × 3 runs = 99 inference calls — impractical to execute (~135 hours estimated).
+
+**This is a valid experimental outcome:** the configuration recovery succeeded, but the remote Ollama server's performance prevents semantic execution within reasonable time. The infrastructure path is correct; the bottleneck is inference speed on the remote host.
+
+### Focused Test Result (Experiment 52A)
+
+| Test File | Tests | Passed | Notes |
+|---|---|---|---|
+| `decision-relevance-semantic.test.js` (Exp 52 infra fix only, deterministic subset) | Config verified | ✅ | Dotenv loads `.env.local`; Ollama reachable at configured URL; no hardcoded localhost |
+| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 | All pass. No production code changed. |
+
+### Regression Result
+
+| Test File | Tests | Passed |
+|---|---|---|
+| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
+
+All existing tests unchanged. No regression introduced.
+
+### Production Provider Unchanged
+
+- `lib/llm/provider.js`: 0 lines changed
+- `lib/config.js`: 0 lines changed
+- `lib/analysis.js`: 0 lines changed
+- `lib/graph/orchestrator.js`: 0 lines changed
+
+### Duplicate Helper Status
+
+Retained (not removed). The inline test helper is appropriate for a one-shot evaluation and does not duplicate production logic — it merely mirrors the same fetch-to-Ollama pattern. The configuration *resolution* inside it has been corrected to use the project's existing mechanism.
+
+### Conclusion
+
+Experiment 52A resolved the configuration root cause. The semantic helper now uses exactly the same environment variable resolution as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. With a faster Ollama instance or model, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will execute the semantic comparison as Experiment 52 defined.
+
 **Status: Pending Rob's review.**