robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: separate semantic meaning from relevance labels

Browse Source
This commit is contained in:
robbond
2026-08-07 09:00:46 +01:00
parent 8ee1f575f7
commit b42a1ff244
3 changed files with 628 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design-evolution-log.md
+145
View File
@@ -3882,3 +3882,148 @@ All existing tests unchanged. No regression introduced.
- `tests/graph/decision-relevance-semantic.test.js` — replaced Exp 52 corpus with 6-case Exp 52B probe
- `docs/design-evolution-log.md` — added Experiment 52B section
- `docs/current-handoff.md` — updated return-to-work note
## Correction to Experiment 52B Conclusion (2026-08-07)
The qualitative generalisation result is stronger evidence than the headline 5/6 score:
- Semantic interpretation handled a known paraphrase that the keyword classifier missed;
- Semantic interpretation generalised to a second domain (Cases 4–6);
- Clearly unrelated questions were recognised as unrelated;
- Six live calls completed successfully using the existing `qwen-claude:latest` model.
However, two experimental-control issues were exposed:
1. The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52, but this does influence which category the model selects);
2. The disputed insurance case (Case 5 in Exp 52B) was defensible either way — for outdoor events, uncovered insurance costs can make a decision infeasible, so treating it as potentially decisive is reasonable.
Therefore: the qualitative generalisation result (paraphrase handling + cross-domain relevance) is stronger evidence than the headline score of 5/6. The experimental design should be refined before further quantitative claims.
## Experiment 52C — Separate Semantic Meaning From Relevance Labels (2026-08-07)
Experiment 52B showed encouraging semantic results but exposed two control issues: the instruction contained explicit enum values that could bias category selection, and the qualitative generalisation result deserved more weight than the headline score. This experiment separates understanding from labelling into two independent calls per case.
### Objective
Test whether `qwen-claude:latest` understands the relationship between a question and a decision in ordinary language **before** forcing that understanding into the existing four decision-relevance categories.
> Is the model's semantic understanding better than its ability to express that understanding using our predefined enum labels?
### Configuration
| Setting | Value |
|---|---|
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
| Model | `qwen-claude:latest` (from `.env.local`) |
| Meaning-mode instruction | "Explain in one short sentence how answering this question would or would not matter to the stated decision. Do not classify it, score it, or use predefined category names." (+ JSON schema hint `{relationship: "..."}` for output format) |
| Enum-mode instruction | Same constrained instruction as Exp 52B (four categories) |
### Five Fixed Cases
| Case | Domain | Question | Expected Relationship | Expected Enum |
|------|--------|----------|----------------------|---------------|
| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | Resolving demand could materially change whether market entry is worthwhile. | `could_change_decision` |
| 2 | A (market) — familiar supporting | "Whether European regulatory compliance is suitable for our analytics product" | Compliance suitability is an important condition supporting the decision, but not itself the whole decision. | `supports_decision` |
| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | Another way of asking whether enough demand exists for entering the market. | `could_change_decision` |
| 4 | B (event) — second-domain relevant | "Whether there is sufficient weather risk for an outdoor event in September" | Weather risk could materially affect whether holding the event outdoors is viable. | `could_change_decision` |
| 5 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | Board chairs has no meaningful bearing on outdoor event decision. | `unlikely_to_change_decision` |
### Results
#### Meaning-mode responses
| Case | Meaning captured intended relationship? | Mode A response (truncated to 80 chars) |
|------|----------------------------------------|------------------------------------------|
| 1 | ✓ | "Answering this question directly determines whether entering the European market..." |
| 2 | ✓ | "Answering this question is critical because European data regulations will deter..." |
| 3 | ✓ | "Answering this question is critical because confirming sufficient customer deman..." |
| 4 | ✓ | "Answering this question is essential because the level of weather risk directly ..." |
| 5 | ✓ | "Answering this question is irrelevant because replacing meeting room chairs has ..." |
**Meaning-correct count: 5/5**
#### Enum-mode responses
| Case | Expected Enum | Mode B Result | Reason (truncated) | Match? |
|------|--------------|---------------|-------------------|--------|
| 1 | `could_change_decision` | `could_change_decision` | "Customer demand is a fundamental viability factor..." | ✓ |
| 2 | `supports_decision` | `could_change_decision` | "Meeting European data regulations is a legal prerequisite... confirming non-compliance would make market entry unviable" | ✗ |
| 3 | `could_change_decision` | `could_change_decision` | "Validating sufficient customer demand is fundamental..." | ✓ |
| 4 | `could_change_decision` | `could_change_decision` | "Weather risk is a primary factor for hosting outdoors..." | ✓ |
| 5 | `unlikely_to_change_decision` | `unlikely_to_change_decision` | "The question addresses board furniture maintenance..." | ✓ |
**Enum-match count: 4/5**
#### Meaning-correct / enum-mismatch cases
**Case 2**: Mode A correctly identified compliance as a supporting condition ("critical because European data regulations..."). Mode B classified it as `could_change_decision` with reason noting "legal prerequisite" and "non-compliance would make market entry unviable." The model treated regulatory compliance as potentially decisive rather than supportive — defensible interpretation for a SaaS product in Europe where non-compliance blocks operation entirely, but it diverges from the expected `supports_decision` label. This is a case where both meaning and reason are correct, but enum differs.
### Inference Timing
- Total inference time: ~147,050 ms (≈147 seconds)
- Average per call: ~14,705 ms (~15 seconds)
- Fastest call: ~9,500 ms
- Slowest call: ~27,000 ms
- All 10 calls completed successfully
### Key Findings
1. **Meaning mode scored 5/5 — perfect on this probe.** Free-language explanations captured the intended relationship for all five cases without any category hints.
2. **Enum classification scored 4/5.** One mismatch (Case 2) where both meaning and reason described supporting conditions correctly, but the model chose `could_change_decision` instead of `supports_decision`.
3. **The known paraphrase retained its meaning without enum hints (Case 3).** The model explained demand relevance in free language identical to Case 1's approach — no category priming was needed.
4. **Cross-domain generalisation held without enum hints (Case 4).** Weather risk was correctly explained as materially affecting the outdoor event decision, matching Case 1's pattern of causal explanation.
5. **Unrelated case remained clearly unrelated (Case 5).** Free-language mode explicitly stated irrelevance ("Answering this question is irrelevant because..."), confirming the model does not force false connections when none exist.
6. **Supplying enum names did materially change interpretation.** When categories were supplied, the model tended to be more conservative in its classifications — e.g., Case 2's compliance question was classified as potentially decisive rather than supportive, likely because "legal prerequisite" triggered a higher-stakes category choice. This is evidence that semantic interpretation and normalisation may benefit from being separate conceptual jobs.
### Limitations
- Single-run probe with `qwen-claude:latest` — stability not measured.
- Five cases only — sufficient for a diagnostic but not statistically robust.
- Remote host latency (~15s/call) limits scope of repeatability testing.
- Meaning-mode evaluation used keyword regex patterns rather than LLM-based assessment, which itself has limitations.
- Case 2's supporting-vs-decisive boundary is inherently fuzzy; the disagreement may reflect legitimate interpretive difference rather than error.
### Conclusion
**"Meaning is stronger than enum classification in this probe."**
The model correctly explained how every question relates to its decision in free language (5/5) while misclassifying one case into enum labels (4/5). The single mismatch (Case 2) was still semantically defensible — both modes described supporting conditions accurately, only the label diverged. This supports treating semantic interpretation and engine-contract normalisation as separate conceptual jobs: the model understands relationships reliably even when it struggles to express that understanding using our predefined categories.
### Focused Test Result
| Test File | Tests | Passed |
|---|---|---|
| `decision-relevance-semantic-normalisation.test.js` (Exp 52C) | 29 | 29 |
### Regression Result
| Test File | Tests | Passed |
|---|---|---|
| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 |
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 |
All existing tests pass. No regression introduced.
### Production Unchanged
- `lib/graph/question-decision-relevance.js`: 0 lines changed
- `lib/llm/provider.js`: 0 lines changed
- `lib/config.js`: 0 lines changed
- `lib/analysis.js`: 0 lines changed
- `lib/graph/orchestrator.js`: 0 lines changed
### Files Created
- `tests/graph/decision-relevance-semantic-normalisation.test.js` — Exp 52C probe (29 tests, 10 live calls)
### Files Modified
- `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section
- `docs/current-handoff.md` — updated return-to-work note