experiment: test semantic decision relevance
This commit is contained in:
@@ -3522,4 +3522,145 @@ No fixtures were loaded, read, or modified. All unknowns in this test are constr
|
||||
|
||||
The decision-relevance classifier has no callers outside its own module (verified in Exp 28 implementation-verification). No active user-facing behaviour changed.
|
||||
|
||||
### Correction to Experiment 51 Interpretation
|
||||
|
||||
During this session, one labelling interpretation from Experiment 51 was corrected:
|
||||
|
||||
> The item "Whether we should enter the European market for analytics tools" was listed as part of the scattered set (DOMAIN_1_SCATTERED.scattered-demand) in the Exp-51 test file and labelled as a false positive. This is incorrect. That question IS plainly relevant to the stated European-market decision — it is a go/no-go question about entering that market. It must not be counted as a false positive or evidence of classifier error.
|
||||
|
||||
The item's presence in the scattered set was a test-data labelling decision, not a classifier fault. The main Experiment 51 conclusion remains supported entirely by the second-domain and paraphrase failures documented above.
|
||||
|
||||
**Status: Pending Rob's review.**
|
||||
|
||||
## Experiment 52 — Can Semantic Interpretation Generalise Decision Relevance Beyond Keywords? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Test whether a small, passive semantic interpretation step can judge whether an unknown matters to a stated decision more reliably than the existing keyword-based decision-relevance classifier. Specifically: can the same decision-relevance contract work across paraphrases and different domains when the language is interpreted for meaning rather than matched against known phrases?
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A semantic interpreter given only `{ decisionTarget, unknown }` may classify decision relevance more consistently across different wording and domains than the current deterministic keyword rules. The experiment may also show that semantic interpretation is inconsistent, overconfident, or difficult to constrain. Either result would be useful.
|
||||
|
||||
### Semantic Contract
|
||||
|
||||
The semantic interpreter receives:
|
||||
```js
|
||||
{ decisionTarget, unknown }
|
||||
```
|
||||
|
||||
And returns exactly one of the existing four categories:
|
||||
```js
|
||||
{ relevance: "could_change_decision" | "supports_decision" | "unlikely_to_change_decision" | "cannot_determine", reason: "short factual explanation" }
|
||||
```
|
||||
|
||||
No new categories. No chain-of-thought. The `reason` is a single short explanation of the relationship between the unknown and the decision.
|
||||
|
||||
### Interpretation Instruction (domain-neutral, identical for all domains)
|
||||
|
||||
> Given a decision and one unanswered question, classify whether resolving that question could directly change the decision, would provide useful support for the decision, is unlikely to affect the decision, or cannot be determined from the information provided.
|
||||
|
||||
No domain-specific examples, no keyword mentions. The same instruction was used for both Domain A (market entry) and Domain B (community event).
|
||||
|
||||
### Context Pack Used
|
||||
|
||||
Engine Experiment Work pack from `docs/task-context-packs.md`.
|
||||
|
||||
### Additional Documents Loaded and Why
|
||||
|
||||
- `lib/graph/question-decision-relevance.js` — to understand the deterministic baseline classifier being tested
|
||||
- `tests/graph/decision-relative-coherence.test.js` (Exp 51) — to reuse the test cases and confirm regression stability
|
||||
- Experiment 51 entry in `docs/design-evolution-log.md` — to establish what Exp 51 found (language-sensitive keyword matching) and provide test cases for comparison
|
||||
|
||||
### Semantic Infrastructure Used
|
||||
|
||||
The repository has `lib/llm/provider.js` which calls Ollama `/api/chat` with `format: "json"`. For the experiment, a minimal inline helper (10 lines in the test file) was created — it mirrors the same fetch-to-Ollama pattern without introducing production infrastructure. No new module was created.
|
||||
|
||||
### Evaluation Result: Infrastructure Limitation
|
||||
|
||||
**Ollama is not running on this machine.** `OLLAMA_BASE_URL` is unset and no process listens on port 11434. The semantic interpretation cases (18 test cases × 3 runs each) could not be executed against a live model.
|
||||
|
||||
Per the experiment constraint:
|
||||
> "If no existing helper can make this small request without substantial architecture work: document that dependency as the experiment result."
|
||||
|
||||
The helper was created inline in the test file using the same Ollama `/api/chat` + `format: json` pattern as the production provider. The infrastructure exists (same API contract), but is not currently running. **This is a valid experimental outcome, not a test bug.** Model failures during the experiment were recorded as `cannot_determine` with reason `model_failure: <error>` — not silently repaired.
|
||||
|
||||
### Deterministic Baseline Results (Exp 51 classifier, unchanged)
|
||||
|
||||
Against Domain A (European market entry):
|
||||
- Known phrasing ("whether to enter"): classified as `could_change_decision` ✓
|
||||
- Compliance phrasing: classified as `supports_decision` ✓
|
||||
- Paraphrased coherent ("would enough people want it"): classified as `cannot_determine` ✗
|
||||
- Paraphrased unrelated ("which firms set standard"): classified as `cannot_determine` ✓
|
||||
|
||||
The deterministic classifier continues to fail on paraphrases and new domains — exactly as Experiment 51 established. This is the baseline that semantic interpretation is being compared against.
|
||||
|
||||
### Semantic Interpretation Results
|
||||
|
||||
Not obtained — Ollama was not available. The test file (`tests/graph/decision-relevance-semantic.test.js`) contains the complete contract, all fixed human reference labels, three-run stability checks, and cross-domain comparison logic. When Ollama is available on port 11434 with a JSON-capable model (e.g., `llama3.1`), rerunning:
|
||||
|
||||
```bash
|
||||
npx vitest run tests/graph/decision-relevance-semantic.test.js
|
||||
```
|
||||
|
||||
will exercise the semantic interpreter against all test cases.
|
||||
|
||||
### Paraphrase Results
|
||||
|
||||
Not obtained. The semantic contract and paraphrase test cases are in place. Expected outcomes (based on hypothesis):
|
||||
- Coherent paraphrase ("would enough people there actually want what we offer?") → `could_change_decision` (semantic generalisation)
|
||||
- Unrelated paraphrase ("which analytics firms set the industry standard?") → `cannot_determine` or `unlikely_to_change_decision`
|
||||
|
||||
### Could-change versus supports Distinction
|
||||
|
||||
Not evaluated. The semantic interpreter must distinguish between direct decision-changing questions and supporting-evidence questions. This requires model execution against Domain B where coherent cases split between these categories.
|
||||
|
||||
### Repeatability Result
|
||||
|
||||
Not obtained (no model). The test file runs each case exactly three times and classifies stability as `stable` or `unstable`.
|
||||
|
||||
### Questionable or Unsupported Findings
|
||||
|
||||
The core finding here is an infrastructure gap: the semantic interpretation hypothesis cannot be tested without an Ollama instance with JSON-capable model support. This is a testing environment limitation, not a failure of the experimental design.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Experiment could not be completed with existing infrastructure.** The test file documents the complete semantic contract, evaluation set, and human reference labels. When Ollama (`ollama serve`) is available on port 11434, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will complete the comparison against the deterministic baseline.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
| Test File | Tests | Passed | Notes |
|
||||
|---|---|---|---|
|
||||
| `decision-relevance-semantic.test.js` (Exp 52) | 48 | 15 / 33 fail | 15 pass = deterministic guardrails; 33 fail = Ollama not available |
|
||||
|
||||
### Regression / Validation Result
|
||||
|
||||
| Test File | Tests | Passed |
|
||||
|---|---|---|
|
||||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||||
| `question-decision-relevance.test.js` (Exp 21) | 25 | 25 |
|
||||
|
||||
All existing tests unchanged. No regression introduced.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md`: Experiment 51 interpretation corrected; Experiment 52 added
|
||||
- `docs/current-handoff.md`: Return-to-work note updated
|
||||
|
||||
### Confirmation Production Decision-Relevance Classifier Remained Unchanged
|
||||
|
||||
The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass). The test file includes explicit assertions that known patterns return their original classifications unchanged.
|
||||
|
||||
### Confirmation No Semantic Logic Entered Active Runtime
|
||||
|
||||
The semantic helper is defined exclusively within `tests/graph/decision-relevance-semantic.test.js` as a test-level function. It is never imported by production code. No runtime caller was wired.
|
||||
|
||||
### Confirmation Assessor, Behaviour Selection, Graph Construction and Fixtures Remained Unchanged
|
||||
|
||||
No assessor files loaded or modified. No Behaviour Selection files loaded or modified. No graph construction files loaded or modified. No fixtures loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`.
|
||||
|
||||
### Confirmation Active Engine Behaviour Remained Unchanged
|
||||
|
||||
The decision-relevance classifier has no callers outside its own module. No active user-facing behaviour changed. The semantic helper was never wired into the engine under test.
|
||||
|
||||
**Status: Pending Rob's review.**
|
||||
|
||||
Reference in New Issue
Block a user