experiment: probe explicit hard constraint semantic fidelity

This commit is contained in:
2026-08-09 14:13:48 +01:00
parent 23bfe5f756
commit 85ee4bed30
3 changed files with 220 additions and 0 deletions
+10
View File
@@ -201,3 +201,13 @@ Tested whether deterministic derivation refinement from commit `3e78d57` (refine
5. Follow-up question correctly targets the emergent conditional/threshold unknown.
**Key observation**: The LLM does not auto-populate `supportCategory` — it is consistently null in `answerMeaning`. The deterministic derivation layer in `readDiagnostics` (and the inline pipeline) is the sole mechanism by which meaning profile category gets determined. This confirms the design: LLM produces raw meaning; deterministic logic categorizes it. No regression detected. Full results in `docs/experiment-56d.md`. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.
### Experiment 56J Summary — Regression D Explicit Hard Constraint Semantic Probe
Tested whether the configured live Ollama model (`qwen-claude:latest` at `http://192.168.1.111:11434`) preserves explicit hard-constraint meaning from user answer "It's a hard constraint. I don't want any increase in risk." — Regression D from `docs/reasoning-refinement-requirements.md`.
One live Ollama call (19,343 ms) returned `userSupportedMeaning: "Avoiding additional risk is a hard constraint, and no increase in risk is acceptable."` with `possibleInference: null`.
**Classification: PASS.** The model preserved the explicit hard-constraint status without weakening it into preference/trade-off language and did not add unsupported interpretation. `possibleInference` is null, which is appropriate for a direct unambiguous answer.
This experiment does not prove fidelity for other regression cases (E, F), consistency across multiple runs, or behavior in production reasoning paths. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-d-explicit-hard-constraint.test.js` and `docs/experiment-56j.md`. Status pending Rob's review.
+79
View File
@@ -0,0 +1,79 @@
# Experiment 56J — Explicit Hard Constraint Semantic Fidelity (Regression D)
## Purpose
Probe whether the configured live Ollama model preserves the user's explicit hard-constraint meaning without weakening it into a preference/trade-off or adding unsupported meaning.
## Branch / HEAD
- **Branch:** `feature/reasoning-fidelity-v0.8`
- **HEAD:** at time of run, clean working tree on this branch.
## Historical Live-Call Pattern Reused
Experiment 55D — commit `fcb7218407a2921e9197dbb0a65e4e1282459e4c`
File: `tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js`
The established mechanism was reused:
- Vitest ESM test;
- `dotenv` loads `.env.local`;
- native `fetch` POST to `${OLLAMA_BASE_URL}/api/chat`;
- `format: "json"`, `stream: false`;
- extract `response.message.content`;
- strip JSON markdown fences; parse structured JSON.
## Configured Ollama Host / Model
- **Base URL:** `http://192.168.1.111:11434`
- **Model:** `qwen-claude:latest`
## Call Count
**Exactly 1 real Ollama call.** No retries, no voting, no fallback.
## Duration
**19,343 ms** (19.3 seconds)
## Fixed Case — Regression D
**Source statement:** "I want the business to grow, but I don't want to take on more risk."
**Clarification target context:** whether avoiding additional risk is a hard constraint or a preference/trade-off
**Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
**User's answer (verbatim):** "It's a hard constraint. I don't want any increase in risk."
## Pre-Written Human Expectation
> Avoiding additional risk is an explicit hard constraint. The user does not accept any increase in risk.
The answer establishes hard-constraint status. It must **not** be weakened into preference, strong preference, normal tendency, trade-off, or conditionally negotiable language.
## Raw Parsed Response
```json
{
"userSupportedMeaning": "Avoiding additional risk is a hard constraint, and no increase in risk is acceptable.",
"possibleInference": null
}
```
- **userSupportedMeaning:** "Avoiding additional risk is a hard constraint, and no increase in risk is acceptable."
- **possibleInference:** null (correct — explicit answer does not require inferred implication)
## Call Duration
19,343 ms
## Human Semantic Classification: PASS
### Rationale
`userSupportedMeaning` clearly preserves that avoiding additional risk is an explicit hard constraint with no accepted increase in risk. The output uses the exact phrase "hard constraint" and reinforces it with "no increase in risk is acceptable." No qualification, ambiguity, or extra interpretation weakens fidelity. `possibleInference` is null, which is appropriate for a direct, unambiguous answer.
### Specific checks
- **Preserves explicit hard-constraint status:** YES — the words "hard constraint" appear directly, reinforced by "no increase in risk is acceptable."
- **Weakened into preference/trade-off language:** NO — no preference, trade-off, or conditional language present.
- **Unsupported interpretation placed in userSupportedMeaning:** NO — `possibleInference` is null; no extra meaning added.
## What This Experiment Established
For Regression D, the configured live Ollama model (`qwen-claude:latest`) preserves explicit hard-constraint meaning without weakening it. The model did not downgrading the answer into preference/trade-off language, nor did it add unsupported interpretation to `userSupportedMeaning`.
## What This Experiment Does NOT Prove
- Semantic fidelity for other regression cases (E, F, or others).
- Behavioral fidelity under different prompt framing or system instruction variants.
- Consistency across multiple calls (single-call probe only).
- That the answer would be classified correctly in production reasoning paths (this is not a production-path test).
- That other models or model versions would behave identically.