70 lines
3.9 KiB
Markdown
70 lines
3.9 KiB
Markdown
# Experiment 57B — Validate Grounded Unclassified Affirmative Answer (Live)
|
||
|
||
**Status:** PASS
|
||
|
||
**Baseline:** `4e4d0fa` (reasoning: stop answer fidelity guard blocking valid unclassified answers)
|
||
|
||
**Branch:** `feature/reasoning-guard-generality-v0.9`
|
||
|
||
## Summary
|
||
|
||
One bounded live semantic probe validates that the configured Qwen model produces user-supported meaning genuinely grounded in a raw affirmative answer — not introducing stronger unsupported semantics. This tests the live semantic premise behind Codex v0.9's grounding approach (semantic token overlap instead of category extension).
|
||
|
||
## Fixed Case
|
||
|
||
**Context / unresolved question:**
|
||
> Whether cost reduction is a genuine reason supporting the relocation decision.
|
||
|
||
**User answer:**
|
||
> We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead.
|
||
|
||
**Pre-written human reference (authoritative):**
|
||
> The answer establishes that cost reduction is a genuine stated reason supporting consideration of the relocation, with approximately £2M annual office-overhead savings cited by the user. It does not by itself establish that relocation is definitely the right decision, that cost is the only consideration, or that all other constraints are satisfied.
|
||
|
||
## Live Result
|
||
|
||
**Configured host/model:** `qwen-claude:latest` at `http://192.168.1.111:11434`
|
||
|
||
**Live-call count:** 1
|
||
|
||
**Duration:** 20,351 ms
|
||
|
||
**Raw structured response:**
|
||
- `userSupportedMeaning`: "The user directly states that cost reduction, specifically approximately £2M in annual office-overhead savings, is a primary stated reason for considering the relocation."
|
||
- `possibleInference`: "This financial emphasis may imply that monetary factors are currently prioritized over other operational or strategic considerations, though this remains unconfirmed."
|
||
|
||
## Classification: PASS
|
||
|
||
**Rationale:**
|
||
|
||
- `userSupportedMeaning` stays within the pre-written reference: cost reduction is genuinely stated as a reason; approximately £2M savings is preserved; no final-decision certainty is added (relocation is framed as "considering" not "decided").
|
||
- No unsupported constraint, preference, approval, or stronger meaning.
|
||
- `possibleInference` correctly placed the financial-prioritization implication beyond stated meaning and flagged it as unconfirmed — appropriate inference separation.
|
||
|
||
## Relationship to v0.9 Codex Premise
|
||
|
||
**Would this live meaning be the kind of grounded unclassified answer v0.9 is intended to allow?** YES
|
||
|
||
The observed `userSupportedMeaning` contains semantic tokens (cost reduction, £2M, annual, office-overhead, savings) that map directly to the raw answer's content. The v0.9 token-overlap grounding mechanism would validate this because it is genuinely derived from the raw answer without strengthening beyond what was stated.
|
||
|
||
## What This Experiment Established
|
||
|
||
- The configured Qwen model can produce grounded user-supported meaning for a legitimate decision-advancing affirmative answer that falls into `other` (unclassified) — exactly the case blocked by the v0.8 guard.
|
||
- The semantic token overlap approach is conceptually sufficient for this fixed case: the model's output stays within the raw answer's semantic range.
|
||
- One live call confirmed the premise on which Codex `4e4d0fa` is based.
|
||
|
||
## What This Experiment Does NOT Prove
|
||
|
||
- Token-overlap threshold (≥ 0.4 ratio or ≥ 3 tokens) adequacy across diverse unclassified answers;
|
||
- Behaviour with weaker, ambiguous, or partially relevant affirmative answers;
|
||
- Behaviour when the model introduces subtle strengthening that still achieves sufficient token overlap (false positive);
|
||
- Deterministic guard integration under production conditions;
|
||
- Stability across repeated runs;
|
||
- Any other regression case (A–F already validated in prior experiments).
|
||
|
||
## Test File
|
||
|
||
`tests/reconstruction/semantic-regression-unclassified-affirmative-answer.test.js`
|
||
|
||
No production code was modified.
|