experiment: run small semantic decision-relevance probe
This commit is contained in:
@@ -3733,3 +3733,152 @@ Retained (not removed). The inline test helper is appropriate for a one-shot eva
|
||||
Experiment 52A resolved the configuration root cause. The semantic helper now uses exactly the same environment variable resolution as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. With a faster Ollama instance or model, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will execute the semantic comparison as Experiment 52 defined.
|
||||
|
||||
**Status: Pending Rob's review.**
|
||||
|
||||
---
|
||||
|
||||
## Experiment 52B — Small Semantic Probe With the Existing Qwen Model (2026-08-07)
|
||||
|
||||
Experiment 52A recovered configuration but deferred semantic execution due to latency (~82s/request makes 99 calls impractical). This experiment reduces the evaluation to the smallest useful live probe: six cases, one call each.
|
||||
|
||||
### Objective
|
||||
|
||||
Using the existing configured `qwen-claude:latest` model, does semantic interpretation handle a handful of paraphrases and cross-domain cases better than the deterministic keyword classifier?
|
||||
|
||||
### Configuration
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||||
| Semantic instruction | Same conceptual instruction as Exp 52, with explicit enum added so the model outputs valid category values |
|
||||
|
||||
### Six Cases Evaluated
|
||||
|
||||
| Case | Domain | Question | Human Reference | Purpose |
|
||||
|---|---|---|---|---|
|
||||
| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | could_change_decision | Easy in-domain test |
|
||||
| 2 | A (market) — familiar unrelated | "Can two senior staff members resolve their ongoing disagreement?" | unlikely_to_change_decision | Reject obviously unrelated |
|
||||
| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | could_change_decision | Known deterministic failure |
|
||||
| 4 | B (event) — relevant | "Whether there is sufficient weather risk for an outdoor event in September" | could_change_decision | Cross-domain generalisation |
|
||||
| 5 | B (event) — supporting | "What insurance requirements apply for hosting the event outdoors" | supports_decision | Distinguish decisive vs supportive |
|
||||
| 6 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | unlikely_to_change_decision | Reject non-relevant in new domain |
|
||||
|
||||
### Results
|
||||
|
||||
#### Human Reference Labels (fixed before evaluation)
|
||||
|
||||
| Case | Human Ref |
|
||||
|---|---|
|
||||
| 1 | could_change_decision |
|
||||
| 2 | unlikely_to_change_decision |
|
||||
| 3 | could_change_decision |
|
||||
| 4 | could_change_decision |
|
||||
| 5 | supports_decision |
|
||||
| 6 | unlikely_to_change_decision |
|
||||
|
||||
#### Deterministic Baseline Results
|
||||
|
||||
| Case | Deterministic Result | Matches Human Ref? |
|
||||
|---|---|---|
|
||||
| 1 | could_change_decision | ✓ |
|
||||
| 2 | cannot_determine | ✗ |
|
||||
| 3 | cannot_determine | ✗ |
|
||||
| 4 | cannot_determine | ✗ |
|
||||
| 5 | cannot_determine | ✗ |
|
||||
| 6 | cannot_determine | ✗ |
|
||||
|
||||
**Deterministic agreement with human reference: 1/6** (only the familiar in-domain case matched)
|
||||
|
||||
#### Semantic Results (qwen-claude:latest)
|
||||
|
||||
| Case | Semantic Result | Latency (ms) | Matches Human Ref? |
|
||||
|---|---|---|---|
|
||||
| 1 | could_change_decision | 14001 | ✓ |
|
||||
| 2 | unlikely_to_change_decision | 15633 | ✓ |
|
||||
| 3 | could_change_decision | 9455 | ✓ |
|
||||
| 4 | could_change_decision | 26992 | ✓ |
|
||||
| 5 | could_change_decision | 14658 | ✗ (model classified as decisive rather than supportive — defensible for insurance constraints) |
|
||||
| 6 | unlikely_to_change_decision | 13859 | ✓ |
|
||||
|
||||
**Semantic agreement with human reference: 5/6**
|
||||
|
||||
### Inference Timing
|
||||
|
||||
- Total inference time: ~94,598 ms (≈95 seconds)
|
||||
- Average per call: ~15,766 ms (~16 seconds)
|
||||
- Fastest call: 9,455 ms (Case 3 — paraphrase)
|
||||
- Slowest call: 26,992 ms (Case 4 — cross-domain)
|
||||
- All six calls completed successfully
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Semantic interpretation correctly handled the known keyword failure (Case 3).** The deterministic classifier returned `cannot_determine` for "Would enough people there actually want what we offer?" — a paraphrase of "Whether to enter the European market for analytics tools." The semantic model classified it as `could_change_decision`, agreeing with human reference.
|
||||
|
||||
2. **Semantic interpretation generalised to a second domain (Cases 4–6).** Despite being trained on market-entry vocabulary, the model correctly classified weather-risk as relevant and board-chairs as unrelated for an outdoor-community-event decision.
|
||||
|
||||
3. **Deterministic classifier cannot generalise.** On all four unseen cases (2–6), the deterministic baseline returned `cannot_determine`. It only matched human reference on the one in-domain case it was trained to recognise.
|
||||
|
||||
4. **One defensible disagreement (Case 5).** The model classified insurance requirements as `could_change_decision` rather than `supports_decision`. For an outdoor event, uncovered insurance costs can make the decision infeasible — so treating it as potentially decisive is a reasonable interpretation.
|
||||
|
||||
5. **Latency improved vs earlier measurements.** Average ~16s/call versus ~82s reported in Experiment 52A. Possible server load variation or model warm-up effects.
|
||||
|
||||
### Agreement Counts
|
||||
|
||||
- Semantic agreement with human reference: **5/6**
|
||||
- Deterministic agreement with human reference: **1/6**
|
||||
|
||||
### Did Semantic Interpretation Improve Generalisation?
|
||||
|
||||
**Yes.** On this small probe, semantic interpretation correctly classified all four in-domain cases (1–3) plus the cross-domain relevant case (4). The deterministic classifier could only classify the one in-domain training-vocabulary case.
|
||||
|
||||
### Is a Repeatability Experiment Justified?
|
||||
|
||||
**Partially.** The evidence on paraphrase generalisation and cross-domain relevance is strong enough to justify confidence. However:
|
||||
- This was a single-run probe with `qwen-claude:latest` — stability across runs was not tested.
|
||||
- The model has the right intuition but tends toward conservative categories (Classified supportive insurance question as decisive).
|
||||
- A repeatability experiment should test whether results hold across different questions and model variants.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Single-run per case — no stability measurement.
|
||||
- One model only (`qwen-claude:latest`) — does not generalise to other models.
|
||||
- Six cases is informative but not statistically robust.
|
||||
- Remote host latency makes large-scale testing expensive in wall-clock time.
|
||||
- The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52) because qwen-claude:latest needs explicit category labels rather than prose descriptions.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**"Semantic interpretation shows clear improvement in this small probe"**
|
||||
|
||||
The semantic model correctly classified 5 of 6 cases against human reference, including the critical paraphrase case (Case 3) and both cross-domain cases where it generalised beyond training vocabulary. The deterministic classifier scored 1/6 on the same cases.
|
||||
|
||||
No production code changed. No semantic logic entered active runtime. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the six-case test and results, then `docs/design-evolution-log.md` Experiment 52B section.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
| Test File | Tests | Passed |
|
||||
|---|---|---|
|
||||
| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 |
|
||||
| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 |
|
||||
|
||||
### Regression Result
|
||||
|
||||
| Test File | Tests | Passed |
|
||||
|---|---|---|
|
||||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||||
|
||||
All existing tests unchanged. No regression introduced.
|
||||
|
||||
### Production Unchanged
|
||||
|
||||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||||
- `lib/llm/provider.js`: 0 lines changed
|
||||
- `lib/config.js`: 0 lines changed
|
||||
- `lib/analysis.js`: 0 lines changed
|
||||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||||
|
||||
### Files Modified
|
||||
|
||||
- `tests/graph/decision-relevance-semantic.test.js` — replaced Exp 52 corpus with 6-case Exp 52B probe
|
||||
- `docs/design-evolution-log.md` — added Experiment 52B section
|
||||
- `docs/current-handoff.md` — updated return-to-work note
|
||||
|
||||
Reference in New Issue
Block a user