robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test decision-relevance normalisation

Browse Source
This commit is contained in:
robbond
2026-08-07 09:46:21 +01:00
parent b42a1ff244
commit 34f06f2919
3 changed files with 462 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design-evolution-log.md
+125
View File
@@ -4027,3 +4027,128 @@ All existing tests pass. No regression introduced.
- `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section
- `docs/current-handoff.md` — updated return-to-work note
## Experiment 52D — Can Free-Language Meaning Be Normalised Into the Existing Decision-Relevance Contract? (2026-08-07)
Experiment 52C found that free-language semantic understanding scored 5/5 while enum classification scored 4/5, with the compliance case consistently misclassified as `could_change_decision` instead of `supports_decision`. This experiment isolated the normalisation step: the model receives **only** a correct free-language relationship statement (no decision target, no question) and maps it into the existing four categories.
### Objective
Test whether a separate normalisation step — given an already-correct meaning statement — can reliably map that meaning into the engine's existing enum contract without keyword matching or altering the meaning itself.
> Once the meaning has already been understood correctly, can we reliably translate that meaning into the engine's existing categories?
### Configuration
| Setting | Value |
|---|---|
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
| Model | `qwen-claude:latest` (from `.env.local`) |
| Normalisation instruction | "You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories..." (+ definitions + JSON schema) |
| Input per case | `{"relationship": "<fixed free-language statement>"}` only |
| No input | Original decision target, original unknown question, domain examples, or previous model outputs |
### Domain-Neutral Category Definitions Used
These faithfully reflect the production contract in `lib/graph/question-decision-relevance.js`:
| Category | Definition |
|---|---|
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
### Five Fixed Relationship Statements
| Case | Source | Relationship Statement (verbatim) | Expected Enum |
|------|--------|-----------------------------------|---------------|
| 1 — Demand | Exp 52C Case 1 | "Answering whether genuine customer demand exists could materially determine whether entering the European market is worthwhile." | `could_change_decision` |
| 2 — Compliance | Exp 52C Case 2 | "Knowing whether the product can satisfy European regulatory requirements is an important condition that supports the market-entry decision." | `supports_decision` |
| 3 — Paraphrased demand | Exp 52C Case 3 | "Knowing whether enough people there actually want the product would materially affect whether entering that market is worthwhile." | `could_change_decision` |
| 4 — Weather (cross-domain) | Exp 52C Case 4 | "Knowing the weather risk could materially determine whether holding the community event outdoors is viable." | `could_change_decision` |
| 5 — Unrelated chairs | Exp 52C Case 5 | "Whether the board replaces its meeting-room chairs has no meaningful bearing on whether the community event should be held outdoors." | `unlikely_to_change_decision` |
### Results
| Case | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency |
|------|--------------|---------------|--------|-------------------|---------|
| 1 — Demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that answering could materially determine whether market entry is worthwhile..." | 15,937ms |
| 2 — Compliance | `supports_decision` | `could_change_decision` | ✗ mismatch | "Regulatory compliance is a fundamental viability constraint for market entry, functioning as a go/no-go condition where failure to satisfy it would directly reverse the proposed action." | 28,021ms |
| 3 — Paraphrased demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that the answer would materially affect whether entering the market is worthwhile..." | 13,239ms |
| 4 — Weather (cross-domain) | `could_change_decision` | `could_change_decision` | ✓ match | "The relationship explicitly states that weather risk materially determines the event's viability..." | 8,419ms |
| 5 — Unrelated chairs | `unlikely_to_change_decision` | `unlikely_to_change_decision` | ✓ match | "The statement explicitly notes that answering the question has no meaningful bearing on the decision..." | 8,461ms |
**Enum-match count: 4/5**
### Key Findings
1. **Normalisation matched expected enum on 4/5 cases.** The same four categories normalised cleanly when the meaning was already correct.
2. **The compliance boundary disagreement persisted.** Case 2 (regulatory requirements as a supporting condition) still maps to `could_change_decision`. The model's reason — "Regulatory compliance is a fundamental viability constraint... functioning as a go/no-go condition" — is faithful to the relationship statement itself, not an invented interpretation. Both `supports_decision` and `could_change_decision` are defensible: compliance *supports* the decision by building evidence, but non-compliance would *reverse* it (blocking entry entirely). The model chose the latter reading because the category definition for `could_change_decision` includes "go/no-go condition" which aligns with a regulatory blocker.
3. **The paraphrase-derived meaning normalised identically to the familiar demand meaning.** Cases 1 and 3 both returned `could_change_decision` with matching reasoning ("materially affect/determine whether entering the market is worthwhile"). Meaning preservation through paraphrase held when only normalisation was tested.
4. **Cross-domain generalisation held.** The weather case (Case 4) normalised correctly to `could_change_decision` without any domain-specific tuning. The model applied the category definitions consistently across domains.
5. **The unrelated relationship normalised correctly.** Case 5 mapped cleanly to `unlikely_to_change_decision` with a faithful reason referencing "no meaningful bearing."
6. **The model did not attempt to reinterpret missing context.** All five reasons were grounded in the supplied relationship statement. None fabricated information that was not present in the input.
7. **The four-category contract is sufficiently clear for normalisation** in three of four boundary zones (demand, weather, unrelated all normalised correctly). The remaining ambiguity lies specifically at the `supports_decision` ↔ `could_change_decision` boundary.
8. **Evidence points to category definitions as the remaining problem.** Not semantic understanding (already solved by Exp 52C's meaning mode), not normalisation mechanism (which works for 4/5 cases), but the definition of `could_change_decision` which includes "go/no-go condition" — a phrase that both a compliance blocker and a demand question could satisfy.
### Compliance Boundary Analysis
The persistent disagreement on Case 2 is not a model error or a normalisation failure. It is evidence of genuine ambiguity in the category definitions:
- **Relationship statement (meaning):** "...is an important condition that supports the market-entry decision."
- **Model's reading:** "Regulatory compliance is a fundamental viability constraint... go/no-go condition."
- **Expected:** `supports_decision` — because the relationship says "supports"
- **Actual:** `could_change_decision` — because non-compliance would reverse the action
Both readings are faithful to the same relationship statement. The model applied the category definitions literally: if a condition's negation would reverse the decision, it is a "go/no-go condition" under `could_change_decision`. This interpretation is internally consistent and not an error. **Reference-category boundary appears questionable.**
### Inference Timing
- Total inference time: 74,077 ms (~74 seconds)
- Average per call: ~14,815 ms (~15 seconds)
- Fastest call: 8,419 ms (Case 5 — unrelated chairs)
- Slowest call: 28,021 ms (Case 2 — compliance)
### Focused Test Result
| Test File | Tests | Passed |
|---|---|---|
| `decision-relevance-normalisation.test.js` (Exp 52D) | 20 | 20 |
### Regression Result
Regression tests ran against Exp 52C (`decision-relevance-semantic-normalisation.test.js`) and core classifier (`question-decision-relevance.test.js`) — no regressions introduced.
### Production Unchanged
- `lib/graph/question-decision-relevance.js`: 0 lines changed
- No production files modified
### Files Created
- `tests/graph/decision-relevance-normalisation.test.js` — Exp 52D probe (20 tests, 5 live calls)
### Limitations
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
- Five cases only — sufficient for a diagnostic but not statistically robust.
- Remote host latency (~15s/call) limits scope of repeatability testing.
- The compliance boundary disagreement was not resolved; further analysis is needed on whether the existing definitions can distinguish "supports" from "could change" when both interpretations are faithful to the same relationship statement.
### Conclusion
**"Normalisation works but one category boundary remains ambiguous."**
The model correctly mapped four of five correct meaning statements into the expected enum categories when given only the relationship statement and the category definitions — no original decision context was needed. The single remaining disagreement (Case 2, compliance) is not a normalisation failure or a semantic understanding problem: both `supports_decision` and `could_change_decision` are faithful readings of the same relationship statement under the current definitions. The evidence suggests the remaining problem lies in **category definitions** — specifically, the phrase "go/no-go condition" in `could_change_decision` captures compliance blockers that should arguably be classified as supporting evidence rather than decision-reversing conditions.
### Status
Pending Rob's review. The four-category contract is confirmed clear enough for a separate normalisation step, but the `supports_decision` ↔ `could_change_decision` boundary needs refinement (separate experiment).